-
Notifications
You must be signed in to change notification settings - Fork 90
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
In both ntk_weighting.py and self_adaptive_weighting.py, the per-condition weights are computed from the gradient of the loss corresponding to one condition at a time.
However, gradients are currently obtained via .backward(), which accumulates into the leaf .grad fields. As a result, after the first condition, all subsequent gradients are contaminated by previously accumulated gradients and therefore incorrect.
PyTorch docs confirm this behavior: gradients from .backward() are added to existing .grad values unless they are explicitly cleared. See: https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html
dario-coscia
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working