KeyError: 'train_dataloader'

[frankstorming]

I prepare a rctw format dataset,then convert it into OCRDataset format.
I wrote the config file,then prepare to train the dbnet model.
However, the program raise KeyError, but I can't fix it.

code in rctw.py

`rctw_textdet_data_root = 'data/rctw'

rctw_textdet_train = dict(
    type='OCRDataset',
    data_root = rctw_textdet_data_root,
    ann_file='instances_training.json',
    data_prefix=dict(img_path='imgs/'),
    filter_cfg=dict(filter_empty_gt=True, min_size=32),
    pipeline=None)

rctw_textdet_test = dict(
    type='OCRDataset',
    data_root=rctw_textdet_data_root,
    ann_file='instances_test.json',
    data_prefix=dict(img_path='imgs/'),
    test_mode=True,
    pipeline=None)`

code in dbnet_resnet50-oclip_1200e_rctw.py

`_base_ = [
    '_base_dbnet_resnet50-dcnv2_fpnc.py',
    '../_base_/datasets/rctw.py',  # import the dataset config
    '../_base_/default_runtime.py',
    '../_base_/schedules/schedule_sgd_1200e.py',
]

rctw_textdet_train = _base_.rctw_textdet_train            # specify the training set
rctw_textdet_train.pipeline = _base_.train_pipeline   # specify the training pipeline
rctw_textdet_test = _base_.rctw_textdet_test              # specify the testing set
rctw_textdet_test.pipeline = _base_.test_pipeline     # specify the testing pipeline

load_from = 'https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth'  # noqa

train_dataloader = dict(
    batch_size=16,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=rctw_textdet_train)    # specify the dataset in train_dataloader

val_dataloader = dict(
    batch_size=1,
    num_workers=4,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=rctw_textdet_test)    # specify the dataset in val_dataloader

test_dataloader = val_dataloader

auto_scale_lr = dict(base_batch_size=16)

load_from = None

_base_.model.backbone = dict(
    type='CLIPResNet',
    init_cfg=dict(
        type='Pretrained',
        checkpoint='https://download.openmmlab.com/'
        'mmocr/backbone/resnet50-oclip-7ba0c533.pth'))

_base_.train_dataloader.num_workers = 12
_base_.optim_wrapper.optimizer.lr = 0.002

param_scheduler = [
    dict(type='LinearLR', end=100, start_factor=0.001),
    dict(type='PolyLR', power=0.9, eta_min=1e-7, begin=100, end=1200),
]
`

[gaotongxiao]

The config looks alright. What is the exact command you were running and the full error traceback?

[frankstorming]

command:
tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_rctw.py 2

full error traceback:
WARNING:torch.distributed.run:

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

Traceback (most recent call last):
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 51, in getattr
value = super().getattr(name)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/addict/addict.py", line 67, in getattr
return self.getitem(item)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 47, in missing
raise KeyError(name)
KeyError: 'train_dataloader'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/kzx22/ocr/mmocr/tools/train.py", line 111, in
main()
File "/home/kzx22/ocr/mmocr/tools/train.py", line 56, in main
cfg = Config.fromfile(args.config)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 177, in fromfile
cfg_dict, cfg_text, env_variables = Config._file2dict(
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 513, in _file2dict
eval(codeobj, global_locals_var, global_locals_var)
File "", line 42, in
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 53, in getattr
raise AttributeError(f"'{self.class.name}' object has no "
AttributeError: 'ConfigDict' object has no attribute 'train_dataloader'
Traceback (most recent call last):
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 51, in getattr
value = super().getattr(name)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/addict/addict.py", line 67, in getattr
return self.getitem(item)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 47, in missing
raise KeyError(name)
KeyError: 'train_dataloader'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/kzx22/ocr/mmocr/tools/train.py", line 111, in
main()
File "/home/kzx22/ocr/mmocr/tools/train.py", line 56, in main
cfg = Config.fromfile(args.config)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 177, in fromfile
cfg_dict, cfg_text, env_variables = Config._file2dict(
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 513, in _file2dict
eval(codeobj, global_locals_var, global_locals_var)
File "", line 42, in
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/mmengine/config/config.py", line 53, in getattr
raise AttributeError(f"'{self.class.name}' object has no "
AttributeError: 'ConfigDict' object has no attribute 'train_dataloader'
Exception ignored in: <function _TemporaryFileCloser.del at 0x7f40b1225040>
Traceback (most recent call last):
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/tempfile.py", line 445, in del
self.close()
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/tempfile.py", line 441, in close
unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmptz42t1xu/tmpq3eh1v1h.py'
Exception ignored in: <function _TemporaryFileCloser.del at 0x7fbd63e86040>
Traceback (most recent call last):
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/tempfile.py", line 445, in del
self.close()
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/tempfile.py", line 441, in close
unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpcg8e1kr5/tmpx7vz_waw.py'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 28192) of binary: /home/kzx22/anaconda3/envs/mmocr/bin/python
Traceback (most recent call last):
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/torch/distributed/run.py", line 752, in run
elastic_launch(
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/kzx22/anaconda3/envs/mmocr/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

tools/train.py FAILED

[gaotongxiao]

I see. You have overwritten train_dataloader first and then tried to access the one in the _base_, which no longer existed. Removing the later line and only modifying the train_dataloader variable should be fine.

[frankstorming]

Thank you so much!!!
Why didn't I try ctrl f!