Fine-tuning issue of text detection in Paddle

[Adhitya1996]

Dear Paddle Community,

I am currently performing fine-tuning on text detection for paddle OCR on customized dataset. I have shared my YML file for your perusal.

Model I used: Currently I am performing fine-tuning with en_PP-OCRv3_det from the link: https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/doc/doc_en/models_list_en.md to perform fine-tuning on text detection. I am using trained model and ch_PP-OCRv3_det_cml.yml as config file during training.

Relevant Information: After training the Paddle OCR in sample dataset, I got the following on the images displayed as the results.

Task: I wanted to convert the best_accuracy into a inference model so that I can use it for text detection tasks. I used tools/export_model.py and gave the following command: -c C:/Users/Adhi/paddle_fine_tuning/OCRv3_fine-tuned/config.yml -o Global.pretrained_model=C:/Adhi/I011786/paddle_fine_tuning/OCRv3_fine-tuned/best_accuracy Global.save_inference_dir=C:/Adhi/I011786/paddle_fine_tuning/det_inference

Config file: I used a config file which I used during training, generated another config file, which I used for inference model.

Problem: Now I got three folders of Inference model and I am confused which one to use now?

All three folders, consists of three sets of inference.pdparams, .pdparams.info, .pdmodel

Please guide me on which inference file to use, thanks!

Best Regards,
Adhitya

[Adhitya1996]

Config file I gave during training:

Global:
debug: false
use_gpu: true
epoch_num: 200
log_smooth_window: 20
print_batch_step: 10
save_model_dir: C:/Users/Adhi/paddle_fine_tuning/OCRv3_fine-tuned
save_epoch_step: 50
eval_batch_step:

• 0
• 400
  cal_metric_during_train: false
  pretrained_model: C:/Users/Adhi/paddle_fine_tuning/en_PP-OCRv3_det_distill_train/student.pdparams
  checkpoints: null
  save_inference_dir: null
  use_visualdl: false
  #infer_img: doc/imgs_en/img_10.jpg
  save_res_path: C:/Users/Adhi/paddle_fine_tuning/OCRv3_fine-tuned/OCRv3.txt
  distributed: true
  d2s_train_image_shape: [3, -1, -1]
  amp_dtype: bfloat16

Architecture:
name: DistillationModel
algorithm: Distillation
model_type: det
Models:
Student:
pretrained:
model_type: det
algorithm: DB
Transform: null
Backbone:
name: MobileNetV3
scale: 0.5
model_name: large
disable_se: true
Neck:
name: RSEFPN
out_channels: 96
shortcut: True
Head:
name: DBHead
k: 50
Student2:
pretrained:
model_type: det
algorithm: DB
Transform: null
Backbone:
name: MobileNetV3
scale: 0.5
model_name: large
disable_se: true
Neck:
name: RSEFPN
out_channels: 96
shortcut: True
Head:
name: DBHead
k: 50
Teacher:
freeze_params: true
return_all_feats: false
model_type: det
algorithm: DB
Backbone:
name: ResNet_vd
in_channels: 3
layers: 50
Neck:
name: LKPAN
out_channels: 256
Head:
name: DBHead
kernel_list: [7,2,2]
k: 50

Loss:
name: CombinedLoss
loss_config_list:

• DistillationDilaDBLoss:
  weight: 1.0
  model_name_pairs:
  • ["Student", "Teacher"]
  • ["Student2", "Teacher"]
    key: maps
    balance_loss: true
    main_loss_type: DiceLoss
    alpha: 5
    beta: 10
    ohem_ratio: 3
• DistillationDMLLoss:
  model_name_pairs:
  • ["Student", "Student2"]
    maps_name: "thrink_maps"
    weight: 1.0
    model_name_pairs: ["Student", "Student2"]
    key: maps
• DistillationDBLoss:
  weight: 1.0
  model_name_list: ["Student", "Student2"]
  balance_loss: true
  main_loss_type: DiceLoss
  alpha: 5
  beta: 10
  ohem_ratio: 3

Optimizer:
name: Adam
beta1: 0.9
beta2: 0.999
lr:
name: Cosine
learning_rate: 0.001
warmup_epoch: 2
regularizer:
name: L2
factor: 5.0e-05

PostProcess:
name: DistillationDBPostProcess
model_name: ["Student"]
key: head_out
thresh: 0.3
box_thresh: 0.6
max_candidates: 1000
unclip_ratio: 1.5

Metric:
name: DistillationMetric
base_metric_name: DetMetric
main_indicator: hmean
key: "Student"

Train:
dataset:
name: SimpleDataSet
data_dir: C:/Users/Adhi/paddle_fine_tuning/det_images
label_file_list: [C:/Users/Adhi/paddle_fine_tuning/det_images/train/train_det.txt]
ratio_list: [1.0]
transforms:
- DecodeImage:
img_mode: BGR
channel_first: false
- DetLabelEncode: null
- CopyPaste:
- IaaAugment:
augmenter_args:
- type: Fliplr
args:
p: 0.5
- type: Affine
args:
rotate:
- -10
- 10
- type: Resize
args:
size:
- 0.5
- 3
- EastRandomCropData:
size:
- 960
- 960
max_tries: 50
keep_ratio: true
- MakeBorderMap:
shrink_ratio: 0.4
thresh_min: 0.3
thresh_max: 0.7
- MakeShrinkMap:
shrink_ratio: 0.4
min_text_size: 8
- NormalizeImage:
scale: 1./255.
mean:
- 0.485
- 0.456
- 0.406
std:
- 0.229
- 0.224
- 0.225
order: hwc
- ToCHWImage: null
- KeepKeys:
keep_keys:
- image
- threshold_map
- threshold_mask
- shrink_map
- shrink_mask
loader:
shuffle: true
drop_last: false
batch_size_per_card: 1
num_workers: 4

Eval:
dataset:
name: SimpleDataSet
data_dir: C:/Users/Adhi/paddle_fine_tuning/det_images
label_file_list: [C:/Users/Adhi/paddle_fine_tuning/det_images/test/test_det.txt]
transforms:
- DecodeImage: # load image
img_mode: BGR
channel_first: False
- DetLabelEncode: # Class handling label
- DetResizeForTest:
- NormalizeImage:
scale: 1./255.
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
order: 'hwc'
- ToCHWImage:
- KeepKeys:
keep_keys: ['image', 'shape', 'polys', 'ignore_tags']
loader:
shuffle: False
drop_last: False
batch_size_per_card: 1 # must be 1
num_workers: 2

Config file that I used for converting to Inference model (I got this during fine-tuning of the model):
Global:
debug: false
use_gpu: true
epoch_num: 200
log_smooth_window: 20
print_batch_step: 10
save_model_dir: C:/Users/Adhi/paddle_fine_tuning/OCRv3_fine-tuned
save_epoch_step: 50
eval_batch_step:

• 0
• 400
  cal_metric_during_train: false
  pretrained_model: C:/Users/Adhi/paddle_fine_tuning/en_PP-OCRv3_det_distill_train/student.pdparams
  checkpoints: null
  save_inference_dir: null
  use_visualdl: false
  save_res_path: C:/Users/Adhi/paddle_fine_tuning/OCRv3_fine-tuned/OCRv3.txt
  distributed: true
  d2s_train_image_shape:
• 3
• -1
• -1
  amp_dtype: bfloat16
  Architecture:
  name: DistillationModel
  algorithm: Distillation
  model_type: det
  Models:
  Student:
  pretrained: null
  model_type: det
  algorithm: DB
  Transform: null
  Backbone:
  name: MobileNetV3
  scale: 0.5
  model_name: large
  disable_se: true
  Neck:
  name: RSEFPN
  out_channels: 96
  shortcut: true
  Head:
  name: DBHead
  k: 50
  Student2:
  pretrained: null
  model_type: det
  algorithm: DB
  Transform: null
  Backbone:
  name: MobileNetV3
  scale: 0.5
  model_name: large
  disable_se: true
  Neck:
  name: RSEFPN
  out_channels: 96
  shortcut: true
  Head:
  name: DBHead
  k: 50
  Teacher:
  freeze_params: true
  return_all_feats: false
  model_type: det
  algorithm: DB
  Backbone:
  name: ResNet_vd
  in_channels: 3
  layers: 50
  Neck:
  name: LKPAN
  out_channels: 256
  Head:
  name: DBHead
  kernel_list:
  - 7
  - 2
  - 2
  k: 50
  Loss:
  name: CombinedLoss
  loss_config_list:
• DistillationDilaDBLoss:
  weight: 1.0
  model_name_pairs:
  • • Student
    • Teacher
  • • Student2
    • Teacher
      key: maps
      balance_loss: true
      main_loss_type: DiceLoss
      alpha: 5
      beta: 10
      ohem_ratio: 3
• DistillationDMLLoss:
  model_name_pairs:
  • Student
  • Student2
    maps_name: thrink_maps
    weight: 1.0
    key: maps
• DistillationDBLoss:
  weight: 1.0
  model_name_list:
  • Student
  • Student2
    balance_loss: true
    main_loss_type: DiceLoss
    alpha: 5
    beta: 10
    ohem_ratio: 3
    Optimizer:
    name: Adam
    beta1: 0.9
    beta2: 0.999
    lr:
    name: Cosine
    learning_rate: 0.001
    warmup_epoch: 2
    regularizer:
    name: L2
    factor: 5.0e-05
    PostProcess:
    name: DistillationDBPostProcess
    model_name:
• Student
  key: head_out
  thresh: 0.3
  box_thresh: 0.6
  max_candidates: 1000
  unclip_ratio: 1.5
  Metric:
  name: DistillationMetric
  base_metric_name: DetMetric
  main_indicator: hmean
  key: Student
  Train:
  dataset:
  name: SimpleDataSet
  data_dir: C:/Users/Adhi/paddle_fine_tuning/det_images
  label_file_list:
  • C:/Users/Adhi/paddle_fine_tuning/det_images/train/train_det.txt
    ratio_list:
  • 1.0
    transforms:
  • DecodeImage:
    img_mode: BGR
    channel_first: false
  • DetLabelEncode: null
  • CopyPaste: null
  • IaaAugment:
    augmenter_args:
    • type: Fliplr
      args:
      p: 0.5
    • type: Affine
      args:
      rotate:
      • -10
      • 10
    • type: Resize
      args:
      size:
      • 0.5
      • 3
  • EastRandomCropData:
    size:
    • 960
    • 960
      max_tries: 50
      keep_ratio: true
  • MakeBorderMap:
    shrink_ratio: 0.4
    thresh_min: 0.3
    thresh_max: 0.7
  • MakeShrinkMap:
    shrink_ratio: 0.4
    min_text_size: 8
  • NormalizeImage:
    scale: 1./255.
    mean:
    • 0.485
    • 0.456
    • 0.406
      std:
    • 0.229
    • 0.224
    • 0.225
      order: hwc
  • ToCHWImage: null
  • KeepKeys:
    keep_keys:
    • image
    • threshold_map
    • threshold_mask
    • shrink_map
    • shrink_mask
      loader:
      shuffle: true
      drop_last: false
      batch_size_per_card: 1
      num_workers: 4
      Eval:
      dataset:
      name: SimpleDataSet
      data_dir: C:/Users/Adhi/paddle_fine_tuning/det_images
      label_file_list:
  • C:/Users/Adhi/paddle_fine_tuning/det_images/test/test_det.txt
    transforms:
  • DecodeImage:
    img_mode: BGR
    channel_first: false
  • DetLabelEncode: null
  • DetResizeForTest: null
  • NormalizeImage:
    scale: 1./255.
    mean:
    • 0.485
    • 0.456
    • 0.406
      std:
    • 0.229
    • 0.224
    • 0.225
      order: hwc
  • ToCHWImage: null
  • KeepKeys:
    keep_keys:
    • image
    • shape
    • polys
    • ignore_tags
      loader:
      shuffle: false
      drop_last: false
      batch_size_per_card: 1
      num_workers: 2
      profiler_options: null

[UserWangZz]

For the pursuit of accuracy, choose the teacher model; for the pursuit of speed, choose the student model.

[Adhitya1996]

Thank so much for your answer. What is the difference of using Distillation model against simple one such as en_PP-OCRv3_det with ch_pp-ocrv3_det_student.yml (https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_student.yml)?

You might know that PaddleOCR recommends fine-tuning only with ch_pp-ocrv3_det_student.yml from the fine_tune document, hence my query.

What difference does it make when we use ch_PP-OCRv3_det_cml.yml?

Thanks in advance for answering!

Best Regards,
Adhitya