Create rule S6981: Gradients should be scaled when using mixed precision #3966
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
ghislainpiot
force-pushed
the
rule/add-RSPEC-S6981
branch
from
90dd671 to
4cfbb88
Compare
ghislainpiot
force-pushed
the
rule/add-RSPEC-S6981
branch
from
4cfbb88 to
0660c9d
Compare
rules/S6981/python/rule.adoc
Outdated
|
|
||
| If the gradients underflow, the model might not learn properly and the training might be unstable. | ||
|
|
||
| == How to fix it in Pytorch |
There was a problem hiding this comment.
This should be PyTorch, this is in the list of potential frameworks. But I think as long as we don't have the rule specified for multiple frameworks we should just leave it with a simple How to fix it
rules/S6981/python/rule.adoc
Outdated
| with torch.autocast(device_type="cuda"): | ||
| output = model(x) | ||
| loss = torch.nn.functional.cross_entropy(output, y) | ||
| loss.backward() |
There was a problem hiding this comment.
Here maybe we could just add the Noncompliant comment
|
|
ghislainpiot
changed the title
ghislainpiot
changed the title
ghislainpiot
force-pushed
the
rule/add-RSPEC-S6981
branch
from
1b9243b to
36a7291
Compare
|
You can preview this rule here (updated a few minutes after each push).
Review
A dedicated reviewer checked the rule description successfully for: