Skip to content

Commit d84bfbb

Browse files
wangna11BDwangna11BD
authored
Add vis code (#318)
* add LapStyle Model
1 parent b2d44b4 commit d84bfbb

File tree

12 files changed

+719
-17
lines changed

12 files changed

+719
-17
lines changed

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ GAN-Generative Adversarial Network, was praised by "the Father of Convolutional
3737

3838
* [Pixel2Pixel](./docs/en_US/tutorials/pix2pix_cyclegan.md)
3939
* [CycleGAN](./docs/en_US/tutorials/pix2pix_cyclegan.md)
40-
* [LapStyle(coming soon)](./docs/en_US/tutorials/lap_style.md)
40+
* [LapStyle](./docs/en_US/tutorials/lap_style.md)
4141
* [PSGAN](./docs/en_US/tutorials/psgan.md)
4242
* [First Order Motion Model](./docs/en_US/tutorials/motion_driving.md)
4343
* [FaceParsing](./docs/en_US/tutorials/face_parse.md)

‎README_cn.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ GAN--生成对抗网络，被“卷积网络之父”**Yann LeCun（杨立昆）
7070

7171
* [Pixel2Pixel](./docs/zh_CN/tutorials/pix2pix_cyclegan.md)
7272
* [CycleGAN](./docs/zh_CN/tutorials/pix2pix_cyclegan.md)
73-
* [LapStyle(coming soon)](./docs/zh_CN/tutorials/lap_style.md)
73+
* [LapStyle](./docs/zh_CN/tutorials/lap_style.md)
7474
* [PSGAN](./docs/zh_CN/tutorials/psgan.md)
7575
* [First Order Motion Model](./docs/zh_CN/tutorials/motion_driving.md)
7676
* [FaceParsing](./docs/zh_CN/tutorials/face_parse.md)

‎configs/lapstyle_draft.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ min_max:
55
(0., 1.)
66

77
model:
8-
name: LapStyleModel
8+
name: LapStyleDraModel
99
generator_encode:
1010
name: Encoder
1111
generator_decode:
@@ -37,7 +37,7 @@ dataset:
3737
name: LapStyleDataset
3838
content_root: data/coco/test2017/
3939
style_root: data/starrynew.png
40-
load_size: 136
40+
load_size: 128
4141
crop_size: 128
4242
num_workers: 0
4343
batch_size: 1
@@ -56,12 +56,12 @@ optimizer:
5656
beta2: 0.999
5757

5858
validate:
59-
interval: 5000
59+
interval: 500
6060
save_img: false
6161

6262
log_config:
6363
interval: 10
64-
visiual_interval: 5000
64+
visiual_interval: 500
6565

6666
snapshot_config:
6767
interval: 5000

‎configs/lapstyle_rev_first.yaml

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
total_iters: 30000
2+
output_dir: output_dir
3+
checkpoints_dir: checkpoints
4+
min_max:
5+
(0., 1.)
6+
7+
model:
8+
name: LapStyleRevFirstModel
9+
revnet_generator:
10+
name: RevisionNet
11+
revnet_discriminator:
12+
name: LapStyleDiscriminator
13+
draftnet_encode:
14+
name: Encoder
15+
draftnet_decode:
16+
name: DecoderNet
17+
calc_style_emd_loss:
18+
name: CalcStyleEmdLoss
19+
calc_content_relt_loss:
20+
name: CalcContentReltLoss
21+
calc_content_loss:
22+
name: CalcContentLoss
23+
calc_style_loss:
24+
name: CalcStyleLoss
25+
gan_criterion:
26+
name: GANLoss
27+
gan_mode: vanilla
28+
content_layers: ['r11', 'r21', 'r31', 'r41', 'r51']
29+
style_layers: ['r11', 'r21', 'r31', 'r41', 'r51']
30+
content_weight: 1.0
31+
style_weight: 3.0
32+
33+
dataset:
34+
train:
35+
name: LapStyleDataset
36+
content_root: data/coco/train2017/
37+
style_root: data/starrynew.png
38+
load_size: 280
39+
crop_size: 256
40+
num_workers: 16
41+
batch_size: 5
42+
test:
43+
name: LapStyleDataset
44+
content_root: data/coco/test2017/
45+
style_root: data/starrynew.png
46+
load_size: 256
47+
crop_size: 256
48+
num_workers: 0
49+
batch_size: 1
50+
51+
lr_scheduler:
52+
name: NonLinearDecay
53+
learning_rate: 1e-4
54+
lr_decay: 5e-5
55+
56+
optimizer:
57+
optimG:
58+
name: Adam
59+
net_names:
60+
- net_rev
61+
beta1: 0.9
62+
beta2: 0.999
63+
optimD:
64+
name: Adam
65+
net_names:
66+
- netD
67+
beta1: 0.9
68+
beta2: 0.999
69+
70+
validate:
71+
interval: 500
72+
save_img: false
73+
74+
log_config:
75+
interval: 10
76+
visiual_interval: 500
77+
78+
snapshot_config:
79+
interval: 5000

‎configs/lapstyle_rev_second.yaml

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
total_iters: 30000
2+
output_dir: output_dir
3+
checkpoints_dir: checkpoints
4+
min_max:
5+
(0., 1.)
6+
7+
model:
8+
name: LapStyleRevSecondModel
9+
revnet_generator:
10+
name: RevisionNet
11+
revnet_discriminator:
12+
name: LapStyleDiscriminator
13+
draftnet_encode:
14+
name: Encoder
15+
draftnet_decode:
16+
name: DecoderNet
17+
calc_style_emd_loss:
18+
name: CalcStyleEmdLoss
19+
calc_content_relt_loss:
20+
name: CalcContentReltLoss
21+
calc_content_loss:
22+
name: CalcContentLoss
23+
calc_style_loss:
24+
name: CalcStyleLoss
25+
gan_criterion:
26+
name: GANLoss
27+
gan_mode: vanilla
28+
content_layers: ['r11', 'r21', 'r31', 'r41', 'r51']
29+
style_layers: ['r11', 'r21', 'r31', 'r41', 'r51']
30+
content_weight: 1.0
31+
style_weight: 3.0
32+
33+
dataset:
34+
train:
35+
name: LapStyleDataset
36+
content_root: data/coco/train2017/
37+
style_root: data/starrynew.png
38+
load_size: 540
39+
crop_size: 512
40+
num_workers: 16
41+
batch_size: 2
42+
test:
43+
name: LapStyleDataset
44+
content_root: data/coco/test2017/
45+
style_root: data/starrynew.png
46+
load_size: 512
47+
crop_size: 512
48+
num_workers: 0
49+
batch_size: 1
50+
51+
lr_scheduler:
52+
name: NonLinearDecay
53+
learning_rate: 1e-4
54+
lr_decay: 5e-5
55+
56+
optimizer:
57+
optimG:
58+
name: Adam
59+
net_names:
60+
- net_rev_2
61+
beta1: 0.9
62+
beta2: 0.999
63+
optimD:
64+
name: Adam
65+
net_names:
66+
- netD
67+
beta1: 0.9
68+
beta2: 0.999
69+
70+
validate:
71+
interval: 500
72+
save_img: false
73+
74+
log_config:
75+
interval: 10
76+
visiual_interval: 500
77+
78+
snapshot_config:
79+
interval: 5000

‎docs/en_US/tutorials/lap_style.md

Lines changed: 82 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,82 @@
1-
Coming soon.
1+
2+
3+
# LapStyle
4+
5+
6+
This repo holds the official codes of paper: "Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer", which is accepted in CVPR 2021.
7+
8+
## 1 Paper Introduction
9+
10+
11+
Artistic style transfer aims at migrating the style from an example image to a content image. Currently, optimization- based methods have achieved great stylization quality, but expensive time cost restricts their practical applications. Meanwhile, feed-forward methods still fail to synthesize complex style, especially when holistic global and local patterns exist. Inspired by the common painting process ofdrawing a draft and revising the details, [this paper](https://arxiv.org/pdf/2104.05376.pdf) introduce a novel feed- forward method Laplacian Pyramid Network (LapStyle). LapStyle first transfers global style pattern in low-resolution via a Drafting Network. It then revises the local details in high-resolution via a Revision Network, which hallucinates a residual image according to the draft and the image textures extracted by Laplacian filtering. Higher resolution details can be easily generated by stacking Revision Networks with multiple Laplacian pyramid levels. The final stylized image is obtained by aggregating outputs ofall pyramid levels. We also introduce a patch discriminator to better learn local pattern adversarially. Experiments demonstrate that our method can synthesize high quality stylized images in real time, where holistic style patterns are properly transferred.
12+
13+
![lapstyle_overview](https://user-images.githubusercontent.com/79366697/118654987-b24dc100-b81b-11eb-9430-d84630f80511.png)
14+
15+
16+
## 2 How to use
17+
18+
### 2.1 Prepare Datasets
19+
20+
To train LapStyle, we use the COCO dataset as content set. And you can choose any style image you like. Before training or testing, remember modify the data path of style image in the config file.
21+
22+
23+
### 2.2 Train
24+
25+
Datasets used in example is COCO, you can also change it to your own dataset in the config file.
26+
27+
(1) Train the Draft Network of LapStyle under 128*128 resolution:
28+
```
29+
python -u tools/main.py --config-file configs/lapstyle_draft.yaml
30+
```
31+
32+
(2) Then, train the Revision Network of LapStyle under 256*256 resolution:
33+
```
34+
python -u tools/main.py --config-file configs/lapstyle_rev_first.yaml --load ${PATH_OF_LAST_STAGE_WEIGHT}
35+
```
36+
37+
(3) Further, you can train the second Revision Network under 512*512 resolution:
38+
39+
```
40+
python -u tools/main.py --config-file configs/lapstyle_rev_second.yaml --load ${PATH_OF_LAST_STAGE_WEIGHT}
41+
```
42+
43+
### 2.4 Test
44+
45+
To test the trained model, you can directly test the "lapstyle_rev_second", since it also contains the trained weight of previous stages:
46+
```
47+
python tools/main.py --config-file configs/lapstyle_rev_second.yaml --evaluate-only --load ${PATH_OF_WEIGHT}
48+
```
49+
50+
## 3 Results
51+
52+
| Style | Stylized Results |
53+
| --- | --- |
54+
| ![starrynew](https://user-images.githubusercontent.com/79366697/118655415-1ec8c000-b81c-11eb-8002-90bf8d477860.png) | ![chicago_stylized_starrynew](https://user-images.githubusercontent.com/79366697/118655671-59325d00-b81c-11eb-93a3-4fcc24680124.png)|
55+
| ![ocean](https://user-images.githubusercontent.com/79366697/118655407-1c666600-b81c-11eb-83a6-300ee1952415.png) | ![chicago_ocean_512](https://user-images.githubusercontent.com/79366697/118655625-4cae0480-b81c-11eb-83ec-30936ed3df65.png)|
56+
| ![stars](https://user-images.githubusercontent.com/79366697/118655423-20928380-b81c-11eb-92bd-0deeb320ff14.png) | ![chicago_stylized_stars_512](https://user-images.githubusercontent.com/79366697/118655638-50da2200-b81c-11eb-9223-58d5df022fa5.png)|
57+
| ![circuit](https://user-images.githubusercontent.com/79366697/118655399-196b7580-b81c-11eb-8bc5-d5ece80c18ba.jpg) | ![chicago_stylized_circuit](https://user-images.githubusercontent.com/79366697/118655660-56376c80-b81c-11eb-87f2-64ae5a82375c.png)|
58+
59+
## 4 Pre-trained models
60+
61+
We also provide several trained models.
62+
63+
| model | style | path |
64+
|---|---|---|
65+
| lapstyle_circuit | circuit | [lapstyle_circuit](https://paddlegan.bj.bcebos.com/models/lapstyle_circuit.pdparams)
66+
| lapstyle_ocean | ocean | [lapstyle_ocean](https://paddlegan.bj.bcebos.com/models/lapstyle_ocean.pdparams)
67+
| lapstyle_starrynew | starrynew | [lapstyle_starrynew](https://paddlegan.bj.bcebos.com/models/lapstyle_starrynew.pdparams)
68+
| lapstyle_stars | stars | [lapstyle_stars](https://paddlegan.bj.bcebos.com/models/lapstyle_stars.pdparams)
69+
70+
71+
# References
72+
73+
74+
75+
```
76+
@article{lin2021drafting,
77+
title={Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer},
78+
author={Lin, Tianwei and Ma, Zhuoqi and Li, Fu and He, Dongliang and Li, Xin and Ding, Errui and Wang, Nannan and Li, Jie and Gao, Xinbo},
79+
booktitle={Computer Vision and Pattern Recognition (CVPR)},
80+
year={2021}
81+
}
82+
```

‎docs/zh_CN/tutorials/lap_style.md

Lines changed: 78 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,78 @@
1-
即将开源。
1+
2+
# LapStyle
3+
4+
这个repo提供CVPR2021论文"Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer"的官方代码。
5+
6+
## 1 论文介绍
7+
8+
艺术风格迁移的目的是将一个实例图像的艺术风格迁移到一个内容图像。目前，基于优化的方法已经取得了很好的合成质量，但昂贵的时间成本限制了其实际应用。
9+
同时，前馈方法仍然不能合成复杂风格，特别是存在全局和局部模式时。受绘制草图和修改细节这一常见绘画过程的启发，[论文](https://arxiv.org/pdf/2104.05376.pdf) 提出了一种新的前馈方法拉普拉斯金字塔网络（LapStyle）。
10+
LapStyle首先通过绘图网络（Drafting Network）传输低分辨率的全局风格模式。然后通过修正网络（Revision Network）对局部细节进行高分辨率的修正，它根据拉普拉斯滤波提取的图像纹理和草图产生图像残差。通过叠加具有多个拉普拉斯金字塔级别的修订网络，可以很容易地生成更高分辨率的细节。最终的样式化图像是通过聚合所有金字塔级别的输出得到的。论文还引入了一个补丁鉴别器，以更好地对抗的学习局部风格。实验表明，该方法能实时合成高质量的风格化图像，并能正确生成整体风格模式。
11+
12+
![lapstyle_overview](https://user-images.githubusercontent.com/79366697/118654987-b24dc100-b81b-11eb-9430-d84630f80511.png)
13+
14+
15+
## 2 如何使用
16+
17+
### 2.1 数据准备
18+
19+
为了训练LapStyle，我们使用COCO数据集作为内容数据集。您可以任意选择您喜欢的风格图片。在开始训练与测试之前，记得修改配置文件的数据路径。
20+
21+
### 2.2 训练
22+
23+
示例以COCO数据为例。如果您想使用自己的数据集，可以在配置文件中修改数据集为您自己的数据集。
24+
25+
(1) 首先在128*128像素下训练LapStyle的绘图网络（Drafting Network）:
26+
```
27+
python -u tools/main.py --config-file configs/lapstyle_draft.yaml
28+
```
29+
30+
(2) 然后，在256*256像素下训练LapStyle的修正网络（Revision Network）:
31+
```
32+
python -u tools/main.py --config-file configs/lapstyle_rev_first.yaml --load ${PATH_OF_LAST_STAGE_WEIGHT}
33+
```
34+
35+
(3) 最后，在512*512像素下再次训练LapStyle的修正网络（Revision Network）:
36+
```
37+
python -u tools/main.py --config-file configs/lapstyle_rev_second.yaml --load ${PATH_OF_LAST_STAGE_WEIGHT}
38+
```
39+
40+
### 2.4 测试
41+
42+
测试训练好的模型，您可以直接测试 "lapstyle_rev_second"，因为它包含了之前步骤里的训练权重：
43+
```
44+
python tools/main.py --config-file configs/lapstyle_rev_second.yaml --evaluate-only --load ${PATH_OF_WEIGHT}
45+
```
46+
47+
## 3 结果展示
48+
49+
| Style | Stylized Results |
50+
| --- | --- |
51+
| ![starrynew](https://user-images.githubusercontent.com/79366697/118655415-1ec8c000-b81c-11eb-8002-90bf8d477860.png) | ![chicago_stylized_starrynew](https://user-images.githubusercontent.com/79366697/118655671-59325d00-b81c-11eb-93a3-4fcc24680124.png)|
52+
| ![ocean](https://user-images.githubusercontent.com/79366697/118655407-1c666600-b81c-11eb-83a6-300ee1952415.png) | ![chicago_ocean_512](https://user-images.githubusercontent.com/79366697/118655625-4cae0480-b81c-11eb-83ec-30936ed3df65.png)|
53+
| ![stars](https://user-images.githubusercontent.com/79366697/118655423-20928380-b81c-11eb-92bd-0deeb320ff14.png) | ![chicago_stylized_stars_512](https://user-images.githubusercontent.com/79366697/118655638-50da2200-b81c-11eb-9223-58d5df022fa5.png)|
54+
| ![circuit](https://user-images.githubusercontent.com/79366697/118655399-196b7580-b81c-11eb-8bc5-d5ece80c18ba.jpg) | ![chicago_stylized_circuit](https://user-images.githubusercontent.com/79366697/118655660-56376c80-b81c-11eb-87f2-64ae5a82375c.png)|
55+
56+
57+
## 4 模型下载
58+
59+
我们提供几个训练好的权重。
60+
61+
| 模型 | 风格 | 下载地址 |
62+
|---|---|---|
63+
| lapstyle_circuit | circuit | [lapstyle_circuit](https://paddlegan.bj.bcebos.com/models/lapstyle_circuit.pdparams)
64+
| lapstyle_ocean | ocean | [lapstyle_ocean](https://paddlegan.bj.bcebos.com/models/lapstyle_ocean.pdparams)
65+
| lapstyle_starrynew | starrynew | [lapstyle_starrynew](https://paddlegan.bj.bcebos.com/models/lapstyle_starrynew.pdparams)
66+
| lapstyle_stars | stars | [lapstyle_stars](https://paddlegan.bj.bcebos.com/models/lapstyle_stars.pdparams)
67+
68+
69+
# References
70+
71+
```
72+
@article{lin2021drafting,
73+
title={Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer},
74+
author={Lin, Tianwei and Ma, Zhuoqi and Li, Fu and He, Dongliang and Li, Xin and Ding, Errui and Wang, Nannan and Li, Jie and Gao, Xinbo},
75+
booktitle={Computer Vision and Pattern Recognition (CVPR)},
76+
year={2021}
77+
}
78+
```

0 commit comments

Comments
 (0)