Skip to content

Wrong gradient of losses in weighting #718

@GiovanniCanali

Description

@GiovanniCanali

Describe the bug
In both ntk_weighting.py and self_adaptive_weighting.py, the per-condition weights are computed from the gradient of the loss corresponding to one condition at a time.

However, gradients are currently obtained via .backward(), which accumulates into the leaf .grad fields. As a result, after the first condition, all subsequent gradients are contaminated by previously accumulated gradients and therefore incorrect.

PyTorch docs confirm this behavior: gradients from .backward() are added to existing .grad values unless they are explicitly cleared. See: https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html

Metadata

Metadata

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions