-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Time based validation support #21071
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Time based validation support #21071
Conversation
for more information, see https://pre-commit.ci
…ting logic with time based validation.
…m/Sohaib-Ahmed21/pytorch-lightning into feature/13324_validation-interval
…of time based validation.
9af43d7
to
64eb3c7
Compare
for more information, see https://pre-commit.ci
…m/Sohaib-Ahmed21/pytorch-lightning into feature/13324_validation-interval
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the extension and I think it valuable use-case for continues training, but leaving the decision to @lantiga
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the contribution @Sohaib-Ahmed21
Goes without saying, runs become non reproducible, as in:
- it's not possible to re-run and have validation happen at the same intervals
- interrupting a run and resuming will not lead to the same validation curves
I would make sure users understand this, so in case they use validation loss as a criterion for scheduling or stopping they will be fully aware.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
What does this PR do?
val_check_interval
(string,timedelta
, or dict).check_val_every_n_epoch
, and resets after each validation.save_on_train_epoch_end
is set toTrue
.Fixes #13324
Before submitting
yes
Yes
No breaking changes introduced
PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:
Reviewer checklist
📚 Documentation preview 📚: https://pytorch-lightning--21071.org.readthedocs.build/en/21071/