Skip to content

Using Smart pointer to optimizer memory usage of dyGraph #17768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

JiabinYang
Copy link
Contributor

@JiabinYang JiabinYang commented May 31, 2019

This PR use smart pointer to hold grads and release the fwd vars. In this way, we can save lots of memory:

Test on Resnet, with Tesla K40m(Mem 11439MiB):

Without smart pointer : max batch size -> 56
With smart pointer: max batch size -> 99

And We test the convergence of several models under models/dygraph the result is as follow:

Mnist:
Mnist
ptbRNN:
ptb
reinforcement learning:
rl
transformer:
transformer

@JiabinYang JiabinYang requested a review from sneaxiy May 31, 2019 09:14
@JiabinYang JiabinYang changed the title for debug Using Smart pointer to optimizer memory usage of dyGraph Jun 2, 2019
@JiabinYang
Copy link
Contributor Author

JiabinYang commented Jun 3, 2019

This Commit will enhance the performance by 15-20%:

The test result is based on Tesla K40m

PTB_model:
before: 82.2777 s per 1/10 of all data
after: 62.12738 s per 1/10 of all data

Resnet. Batch size 50
before: 1.49s per batch
after: 1.26s per batch

Copy link
Collaborator

@sneaxiy sneaxiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@JiabinYang JiabinYang merged commit 3b70f87 into PaddlePaddle:develop Jun 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants