Skip to content

Commit 4bca517

Browse files
horchamhorcham
horcham
and
horcham
authored
tools for model exporting and converting (#639)
* Add tools bash for model exporting and converting * convert test * convert test * convert test * produce python tool for export and convert --------- Co-authored-by: horcham <zhanghongquan15@huawei.com>
1 parent 0516cca commit 4bca517

File tree

7 files changed

+1350
-34
lines changed

7 files changed

+1350
-34
lines changed

‎docs/cn/tutorials/frequently_asked_questions.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
- [推理相关问题](#q8-推理时相关问题)
1010
- [DBNet训练速率不及预期](#q9-DBNet训练速率不及预期)
1111
- [`libgomp-d22c30c5.so.1.0.0`相关错误](#q10-libgomp-d22c30c5so100相关错误)
12+
- [当在lmdb dataset上训练abinet报数据管道错误](#q11-当在lmdb-dataset上训练abinet报数据管道错误)
1213

1314
### Q1 未定义符号
1415

@@ -694,3 +695,50 @@ ImportError: /root/mindocr_env/lib/python3.8/site-packages/sklearn/__check_build
694695
```bash
695696
export LD_PRELOAD=/root/mindocr_env/lib/python3.8/site-packages/scikit_learn.libs/libgomp-d22c30c5.so.1.0.0:$LD_PRELOAD
696697
```
698+
699+
### Q11 当在lmdb dataset上训练abinet报数据管道错误
700+
当在lmdb dataset上训练abinet报以下数据管道错误
701+
```bash
702+
mindocr.data.rec_lmdb_dataset WARNING - Error occurred during preprocess.
703+
Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'.
704+
705+
------------------------------------------------------------------
706+
- Dataset Pipeline Error Message:
707+
------------------------------------------------------------------
708+
[ERROR] No cast for the specified DataType was found.
709+
710+
------------------------------------------------------------------
711+
- C++ Call Stack: (For framework developers)
712+
------------------------------------------------------------------
713+
mindspore/ccsrc/minddata/dataset/kernels/py_func_op.cc(143).
714+
```
715+
可以尝试用如下步骤修复
716+
- 找到mindspore的包路径
717+
- 打开文件: `mindspore/dataset/transforms/transform.py`
718+
- 跳转到93行，可以得到如下内容:
719+
```bash
720+
93 if key in EXECUTORS_LIST:
721+
94 # get the executor by process id and thread id
722+
95 executor = EXECUTORS_LIST[key]
723+
96 # remove the old transform which in executor and update the new transform
724+
97 executor.UpdateOperation(self.parse())
725+
98 else:
726+
99 # create a new executor by process id and thread_id
727+
100 executor = cde.Execute(self.parse())
728+
101 # add the executor the global EXECUTORS_LIST
729+
102 EXECUTORS_LIST[key] = executor
730+
```
731+
- 使用`executor = cde.Execute(self.parse())`替换97行, 得到如下内容:
732+
```bash
733+
93 if key in EXECUTORS_LIST:
734+
94 # get the executor by process id and thread id
735+
95 executor = EXECUTORS_LIST[key]
736+
96 # remove the old transform which in executor and update the new transform
737+
97 executor = cde.Execute(self.parse())
738+
98 else:
739+
99 # create a new executor by process id and thread_id
740+
100 executor = cde.Execute(self.parse())
741+
101 # add the executor the global EXECUTORS_LIST
742+
102 EXECUTORS_LIST[key] = executor
743+
```
744+
- 保存后再次尝试训练即可

‎docs/en/tutorials/frequently_asked_questions.md

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@
88
- [Problems related to model converting](#q7-problems-related-to-model-converting)
99
- [Problems related to inference](#q8-problems-related-to-inference)
1010
- [Training speed of DBNet not as fast as expexted](#q9-training-speed-of-dbnet-not-as-fast-as-expexted)
11-
- [Error about `libgomp-d22c30c5.so.1.0.0`](#q9-error-about-libgomp-d22c30c5so100)
11+
- [Error about `libgomp-d22c30c5.so.1.0.0`](#q10-error-about-libgomp-d22c30c5so100)
12+
- [Dataset Pipeline Error when training abinet on lmdb dataset](#q11-dataset-pipeline-error-when-training-abinet-on-lmdb-dataset)
1213

1314
### Q1 Undefined symbol
1415

@@ -683,3 +684,50 @@ You can try the following steps to fix it:
683684
```bash
684685
export LD_PRELOAD=/root/mindocr_env/lib/python3.8/site-packages/scikit_learn.libs/libgomp-d22c30c5.so.1.0.0:$LD_PRELOAD
685686
```
687+
688+
### Q11 Dataset Pipeline Error when training abinet on lmdb dataset
689+
The following error may occur when training abinet on lmdb dataset
690+
```bash
691+
mindocr.data.rec_lmdb_dataset WARNING - Error occurred during preprocess.
692+
Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'.
693+
694+
------------------------------------------------------------------
695+
- Dataset Pipeline Error Message:
696+
------------------------------------------------------------------
697+
[ERROR] No cast for the specified DataType was found.
698+
699+
------------------------------------------------------------------
700+
- C++ Call Stack: (For framework developers)
701+
------------------------------------------------------------------
702+
mindspore/ccsrc/minddata/dataset/kernels/py_func_op.cc(143).
703+
```
704+
You can try the following steps to fix it:
705+
- find the folder of mindspore package
706+
- open file: `mindspore/dataset/transforms/transform.py`
707+
- switch to line 93:
708+
```bash
709+
93 if key in EXECUTORS_LIST:
710+
94 # get the executor by process id and thread id
711+
95 executor = EXECUTORS_LIST[key]
712+
96 # remove the old transform which in executor and update the new transform
713+
97 executor.UpdateOperation(self.parse())
714+
98 else:
715+
99 # create a new executor by process id and thread_id
716+
100 executor = cde.Execute(self.parse())
717+
101 # add the executor the global EXECUTORS_LIST
718+
102 EXECUTORS_LIST[key] = executor
719+
```
720+
- replace line 97 with `executor = cde.Execute(self.parse())`, and get
721+
```bash
722+
93 if key in EXECUTORS_LIST:
723+
94 # get the executor by process id and thread id
724+
95 executor = EXECUTORS_LIST[key]
725+
96 # remove the old transform which in executor and update the new transform
726+
97 executor = cde.Execute(self.parse())
727+
98 else:
728+
99 # create a new executor by process id and thread_id
729+
100 executor = cde.Execute(self.parse())
730+
101 # add the executor the global EXECUTORS_LIST
731+
102 EXECUTORS_LIST[key] = executor
732+
```
733+
- save the file, and try to train the model.

‎mindocr/models/det_dbnet.py

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,12 @@ def _cfg(url='', **kwargs):
1919
'dbnet_resnet18': _cfg(
2020
url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnet_resnet18-0c0c4cfa.ckpt'),
2121
'dbnet_resnet50': _cfg(
22+
url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnet_resnet50-c3a4aa24.ckpt'),
23+
'dbnet_resnet50_ch_en': _cfg(
2224
url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnet_resnet50_ch_en_general-a5dbb141.ckpt'),
2325
'dbnetpp_resnet50': _cfg(
26+
url='https://download-mindspore.osinfra.cn/toolkits/mindocr/dbnet/dbnetpp_resnet50-068166c2.ckpt'),
27+
'dbnetpp_resnet50_ch_en': _cfg(
2428
url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnetpp_resnet50_ch_en_general-884ba5b9.ckpt'),
2529
'dbnet_ppocrv3': _cfg(
2630
url='https://download-mindspore.osinfra.cn/toolkits/mindocr/dbnet/dbnet_mobilenetv3_ppocrv3-70d6018f.ckpt')
@@ -125,6 +129,35 @@ def dbnet_resnet50(pretrained=False, pretrained_backbone=True, **kwargs):
125129
return model
126130

127131

132+
@register_model
133+
def dbnet_resnet50_ch_en(pretrained=False, pretrained_backbone=True, **kwargs):
134+
model_config = {
135+
"backbone": {
136+
'name': 'det_resnet50',
137+
'pretrained': pretrained_backbone
138+
},
139+
"neck": {
140+
"name": 'DBFPN',
141+
"out_channels": 256,
142+
"bias": False,
143+
},
144+
"head": {
145+
"name": 'DBHead',
146+
"k": 50,
147+
"bias": False,
148+
"adaptive": True
149+
}
150+
}
151+
model = DBNet(model_config)
152+
153+
# load pretrained weights
154+
if pretrained:
155+
default_cfg = default_cfgs['dbnet_resnet50_ch_en']
156+
load_pretrained(model, default_cfg)
157+
158+
return model
159+
160+
128161
@register_model
129162
def dbnetpp_resnet50(pretrained=False, pretrained_backbone=True, **kwargs):
130163
model_config = {
@@ -156,6 +189,37 @@ def dbnetpp_resnet50(pretrained=False, pretrained_backbone=True, **kwargs):
156189
return model
157190

158191

192+
@register_model
193+
def dbnetpp_resnet50_ch_en(pretrained=False, pretrained_backbone=True, **kwargs):
194+
model_config = {
195+
"backbone": {
196+
'name': 'det_resnet50',
197+
'pretrained': pretrained_backbone
198+
},
199+
"neck": {
200+
"name": 'DBFPN',
201+
"out_channels": 256,
202+
"bias": False,
203+
"use_asf": True,
204+
"channel_attention": True
205+
},
206+
"head": {
207+
"name": 'DBHead',
208+
"k": 50,
209+
"bias": False,
210+
"adaptive": True
211+
}
212+
}
213+
model = DBNet(model_config)
214+
215+
# load pretrained weights
216+
if pretrained:
217+
default_cfg = default_cfgs['dbnetpp_resnet50_ch_en']
218+
load_pretrained(model, default_cfg)
219+
220+
return model
221+
222+
159223
@register_model
160224
def dbnet_ppocrv3(pretrained=False, **kwargs):
161225
model_config = {

‎mindocr/models/rec_abinet.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ def _cfg(url="", **kwargs):
1212
default_cfgs = {
1313
# 'abinet':
1414
"abinet": _cfg(
15-
url="https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_pretrain_en-821ca20b.ckpt"
15+
url="https://download-mindspore.osinfra.cn/toolkits/mindocr/abinet/abinet_resnet45_en-7efa1184.ckpt"
1616
),
1717
}
1818

0 commit comments

Comments
 (0)