[not ready] Saves checkpoints at specified steps #12847
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -41,6 +41,7 @@ | |||
|
|
|||
| from nemo.collections.common.callbacks import EMA | |||
| from nemo.constants import NEMO_ENV_VARNAME_TESTING, NEMO_ENV_VARNAME_VERSION | |||
| from nemo.lightning.ckpt_utils import ckpt_to_context_subdir | |||
Check notice
Code scanning / CodeQL
Cyclic import Note
Copilot Autofix
AI 11 days ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
| @@ -265,6 +266,88 @@ | |||
| log_tflops_per_sec_per_gpu: Optional[bool] = True | |||
|
|
|||
|
|
|||
| try: | |||
| from nemo.lightning.io.pl import TrainerContext | |||
Check notice
Code scanning / CodeQL
Cyclic import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 11 days ago
To fix the cyclic import issue, we need to break the cycle by removing the import of TrainerContext from nemo.lightning.io.pl in the nemo/utils/exp_manager.py file. Since the TrainerContext is only used conditionally, we can move the import statement inside the try-except block where it is used. This way, the import will only be executed when needed, and it will not create a cyclic dependency during the initial module loading.
| @@ -269,4 +269,5 @@ | ||
| try: | ||
| from nemo.lightning.io.pl import TrainerContext | ||
| import nemo.lightning.io.pl as pl_module | ||
|
|
||
| TrainerContext = pl_module.TrainerContext | ||
| HAVE_TRAINER_CONTEXT = True |
jomitchellnv
force-pushed
the
jm/save-checkpoint-n-steps
branch
from
3807129 to
19e5324
Compare
Signed-off-by: jomitchellnv <[email protected]>
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information