Skip to content

polish weights #2376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 22, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'cascade_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNeXt101_vd_64x4d_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNeXt101_vd_64x4d_pretrained.pdparams
weights: output/cascade_rcnn_dcn_x101_vd_64x4d_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_vd_pretrained.pdparams
weights: output/faster_rcnn_dcn_r101_vd_fpn_1x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/dcn/faster_rcnn_dcn_r50_vd_fpn_1x_coco.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/faster_rcnn_dcn_r50_vd_fpn_2x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/dcn/faster_rcnn_dcn_r50_vd_fpn_2x_coco.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/faster_rcnn_dcn_r50_vd_fpn_2x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNeXt101_vd_64x4d_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNeXt101_vd_64x4d_pretrained.pdparams
weights: output/faster_rcnn_dcn_x101_vd_64x4d_fpn_1x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/dcn/mask_rcnn_dcn_r101_vd_fpn_1x_coco.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'mask_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_vd_pretrained.pdparams
weights: output/mask_rcnn_dcn_r101_vd_fpn_1x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/dcn/mask_rcnn_dcn_r50_vd_fpn_2x_coco.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'mask_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/mask_rcnn_dcn_r50_vd_fpn_2x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'mask_rcnn_dcn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNeXt101_vd_64x4d_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNeXt101_vd_64x4d_pretrained.pdparams
weights: output/mask_rcnn_dcn_x101_vd_64x4d_fpn_1x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/faster_rcnn/faster_rcnn_r101_1x_coco.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'faster_rcnn_r50_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_pretrained.pdparams
weights: output/faster_rcnn_r101_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_pretrained.pdparams
weights: output/faster_rcnn_r101_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_pretrained.pdparams
weights: output/faster_rcnn_r101_fpn_2x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_vd_pretrained.pdparams
weights: output/faster_rcnn_r101_vd_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_vd_pretrained.pdparams
weights: output/faster_rcnn_r101_vd_fpn_2x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet34_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet34_pretrained.pdparams
weights: output/faster_rcnn_r34_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet34_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet34_vd_pretrained.pdparams
weights: output/faster_rcnn_r34_vd_fpn_1x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/faster_rcnn/faster_rcnn_r50_vd_1x_coco.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_r50_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/faster_rcnn_r50_vd_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/faster_rcnn_r50_vd_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/faster_rcnn_r50_vd_fpn_2x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNeXt101_vd_64x4d_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNeXt101_vd_64x4d_pretrained.pdparams
weights: output/faster_rcnn_x101_vd_64x4d_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'faster_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNeXt101_vd_64x4d_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNeXt101_vd_64x4d_pretrained.pdparams
weights: output/faster_rcnn_x101_vd_64x4d_fpn_2x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/mask_rcnn/mask_rcnn_r101_fpn_1x_coco.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'mask_rcnn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_pretrained.pdparams
weights: output/mask_rcnn_r101_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
_BASE_: [
'mask_rcnn_r50_fpn_1x_coco.yml',
]
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet101_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet101_vd_pretrained.pdparams
weights: output/mask_rcnn_r101_vd_fpn_1x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/mask_rcnn/mask_rcnn_r50_vd_fpn_1x_coco.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'mask_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/mask_rcnn_r50_vd_fpn_1x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/mask_rcnn/mask_rcnn_r50_vd_fpn_2x_coco.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'mask_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNet50_vd_pretrained.pdparams
weights: output/mask_rcnn_r50_vd_fpn_2x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'mask_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNeXt101_vd_64x4d_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNeXt101_vd_64x4d_pretrained.pdparams
weights: output/mask_rcnn_x101_vd_64x4d_fpn_1x_coco/model_final

ResNet:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ _BASE_: [
'mask_rcnn_r50_fpn_1x_coco.yml',
]

pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNeXt101_vd_64x4d_pretrained.tar
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/ResNeXt101_vd_64x4d_pretrained.pdparams
weights: output/mask_rcnn_x101_vd_64x4d_fpn_2x_coco/model_final

ResNet:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/slim/prune/yolov3_prune_fpgm.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Weights of yolov3_mobilenet_v1_voc
pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/dygraph/yolov3_mobilenet_v1_270e_voc.pdparams
pretrain_weights: https://paddledet.bj.bcebos.com/models/yolov3_mobilenet_v1_270e_voc.pdparams
weight_type: resume
slim: Pruner

Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/slim/prune/yolov3_prune_l1_norm.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Weights of yolov3_mobilenet_v1_voc
pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/dygraph/yolov3_mobilenet_v1_270e_voc.pdparams
pretrain_weights: https://paddledet.bj.bcebos.com/models/yolov3_mobilenet_v1_270e_voc.pdparams
weight_type: resume
slim: Pruner

Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/slim/quant/yolov3_mobilenet_v1_qat.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Weights of yolov3_mobilenet_v1_coco
pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/dygraph/yolov3_mobilenet_v1_270e_coco.pdparams
pretrain_weights: https://paddledet.bj.bcebos.com/models/yolov3_mobilenet_v1_270e_coco.pdparams
weight_type: resume
slim: QAT

Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/slim/quant/yolov3_mobilenet_v3_qat.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Weights of yolov3_mobilenet_v3_coco
pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/dygraph/yolov3_mobilenet_v3_large_270e_coco.pdparams
pretrain_weights: https://paddledet.bj.bcebos.com/models/yolov3_mobilenet_v3_large_270e_coco.pdparams
weight_type: resume
slim: QAT

Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/ssd/_base_/ssd_vgg16_300.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
architecture: SSD
pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/dygraph/VGG16_caffe_pretrained.pdparams
pretrain_weights: https://paddledet.bj.bcebos.com/models/pretrained/VGG16_caffe_pretrained.pdparams

# Model Achitecture
SSD:
Expand Down
2 changes: 1 addition & 1 deletion dygraph/configs/yolov3/yolov3_mobilenet_v1_roadsign.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ _BASE_: [
'_base_/yolov3_mobilenet_v1.yml',
'_base_/yolov3_reader.yml',
]
pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/dygraph/yolov3_mobilenet_v1_270e_coco.pdparams
pretrain_weights: https://paddledet.bj.bcebos.com/models/yolov3_mobilenet_v1_270e_coco.pdparams
norm_type: sync_bn
weights: output/yolov3_mobilenet_v1_roadsign/model_final
metric: VOC
Expand Down
21 changes: 9 additions & 12 deletions dygraph/ppdet/engine/trainer.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ def __init__(self, cfg, mode='train'):
# model slim build
if 'slim' in cfg and cfg.slim:
if self.mode == 'train':
self.load_weights(cfg.pretrain_weights, cfg.weight_type)
self.load_weights(cfg.pretrain_weights)
self.slim = create(cfg.slim)
self.slim(self.model)

Expand Down Expand Up @@ -171,17 +171,14 @@ def register_metrics(self, metrics):
"metrics shoule be instances of subclass of Metric"
self._metrics.extend(metrics)

def load_weights(self, weights, weight_type='pretrain'):
assert weight_type in ['pretrain', 'resume', 'finetune'], \
"weight_type can only be 'pretrain', 'resume', 'finetune'"
if weight_type == 'resume':
self.start_epoch = load_weight(self.model, weights, self.optimizer)
logger.debug("Resume weights of epoch {}".format(self.start_epoch))
else:
self.start_epoch = 0
load_pretrain_weight(self.model, weights, weight_type)
logger.debug("Load {} weights {} to start training".format(
weight_type, weights))
def load_weights(self, weights):
self.start_epoch = 0
load_pretrain_weight(self.model, weights)
logger.debug("Load weights {} to start training".format(weights))

def resume_weights(self, weights):
self.start_epoch = load_weight(self.model, weights, self.optimizer)
logger.debug("Resume weights of epoch {}".format(self.start_epoch))

def train(self, validate=False):
assert self.mode == 'train', "Model not in 'train' mode"
Expand Down
46 changes: 23 additions & 23 deletions dygraph/ppdet/utils/checkpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,7 @@ def load_weight(model, weight, optimizer=None):
assert incorrect_keys == 0, "Load weight {} incorrectly, \
{} keys unmatched, please check again.".format(weight,
incorrect_keys)
logger.info('Finish loading model weight parameter: {}'.format(
pdparam_path))
logger.info('Finish resuming model weights: {}'.format(pdparam_path))

model.set_dict(model_weight)

Expand All @@ -142,9 +141,7 @@ def load_weight(model, weight, optimizer=None):
return last_epoch


def load_pretrain_weight(model, pretrain_weight, weight_type='pretrain'):

assert weight_type in ['pretrain', 'finetune']
def load_pretrain_weight(model, pretrain_weight):
if is_url(pretrain_weight):
pretrain_weight = get_weights_path_dist(pretrain_weight)

Expand All @@ -158,24 +155,27 @@ def load_pretrain_weight(model, pretrain_weight, weight_type='pretrain'):

model_dict = model.state_dict()

param_state_dict = paddle.load(path + '.pdparams')
if weight_type == 'pretrain':
model.backbone.set_dict(param_state_dict)
else:
ignore_set = set()
for name, weight in model_dict.items():
if name in param_state_dict.keys():
if weight.shape != list(param_state_dict[name].shape):
logger.info(
'{} not used, shape {} unmatched with {} in model.'.
format(name,
list(param_state_dict[name].shape),
weight.shape))
param_state_dict.pop(name, None)
else:
logger.info('Lack weight: {}'.format(name))
model.set_dict(param_state_dict)
return
weights_path = path + '.pdparams'
param_state_dict = paddle.load(weights_path)
ignore_set = set()
lack_modules = set()
for name, weight in model_dict.items():
if name in param_state_dict.keys():
if weight.shape != list(param_state_dict[name].shape):
logger.info(
'{} not used, shape {} unmatched with {} in model.'.format(
name, list(param_state_dict[name].shape), weight.shape))
param_state_dict.pop(name, None)
else:
lack_modules.add(name.split('.')[0])
logger.debug('Lack weights: {}'.format(name))

if len(lack_modules) > 0:
logger.info('Lack weights of modules: {}'.format(', '.join(
list(lack_modules))))

model.set_dict(param_state_dict)
logger.info('Finish loading model weights: {}'.format(weights_path))


def save_model(model, optimizer, save_dir, save_name, last_epoch):
Expand Down
2 changes: 1 addition & 1 deletion dygraph/ppdet/utils/download.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@

DOWNLOAD_RETRY_LIMIT = 3

PPDET_WEIGHTS_DOWNLOAD_URL_PREFIX = 'https://paddlemodels.bj.bcebos.com/object_detection/'
PPDET_WEIGHTS_DOWNLOAD_URL_PREFIX = 'https://paddledet.bj.bcebos.com/'


def parse_url(url):
Expand Down
2 changes: 1 addition & 1 deletion dygraph/tools/eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def run(FLAGS, cfg):
trainer = Trainer(cfg, mode='eval')

# load weights
trainer.load_weights(cfg.weights, 'resume')
trainer.load_weights(cfg.weights)

# training
trainer.evaluate()
Expand Down
Loading