Previous PRs introduced a bug on Accumulated Gradients Losses

[w32zhong]

System Info

• transformers version: 4.54.1
• Platform: Linux-5.15.0-131-generic-x86_64-with-glibc2.39
• Python version: 3.12.3
• Huggingface_hub version: 0.34.3
• Safetensors version: 0.5.3
• Accelerate version: 1.10.0
• Accelerate config: not found
• DeepSpeed version: 0.17.4
• PyTorch version (accelerator?): 2.8.0a0+5228986c39.nv25.06 (CUDA)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA H100 80GB HBM3

Who can help?

Previous PRs from: #35207 and #34511

It makes the backward() called after rescaling. This creates a double rescaling both here and in Accelerate:

https://github.com/huggingface/accelerate/blob/23cf4ef8a3b58f016f63eeb158b4aa2c3e79fe6f/src/accelerate/accelerator.py#L2724

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When:

• gradient_accumulation_steps > 1
• not using deepspeed
• num_items_in_batch is None and self.compute_loss_func is None (i.e., when user ignores the GA loss bug)

The final loss is rescaled twice:

loss = loss / gradient_accumulation_steps

Expected behavior

It should be rescaled only once.

[w32zhong]

okay, maybe #34511 states this repo will no longer rely on Accelerate API for GA anymore? So.. just ignore this issue is what we expect?

[qgallouedec]

Gradient accumulation in Trainer and Accelerate are independent. Trainer doesn’t use Accelerate’s gradient accumulation setting, so as long as you haven’t set gradient accumulation steps in both, it should be fine

[qgallouedec]

as long as you haven’t set gradient accumulation steps in both, it should be fine

And even in this case there is a sanity check anyway, if I'm not mistaken:

transformers/src/transformers/trainer.py

Lines 5210 to 5216 in f4d57f2

	if "num_steps" in grad_acc_kwargs:
	if self.args.gradient_accumulation_steps > 1:
	# raise because we do not know which setting is intended.
	raise ValueError(
	"The `AcceleratorConfig`'s `num_steps` is set but `gradient_accumulation_steps` is greater than 1 in the passed `TrainingArguments`"
	"If using the passed `AcceleratorConfig` is desired, do not set the `TrainingArguments` `gradient_accumulation_steps`."
	)

[w32zhong]

@qgallouedec Thanks for your response. It is indeed confusing, but you are correct, looks like the self.trainer.accelerator.gradient_accumulation_steps is set agnostic to the upper Trainer layer now.

It wasn't the case a few months ago, things change fast here 🥲. Anyway, thank you for your help!