New GA fix causes training loss multiple times higher across the board (5x to 10x higher)

[JianbangZ]

System Info

8xH100

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

After updating to the latest master branch of transformer, the training loss is mutiple times higher than before (5x-10x). I tried both SFT and DPO (paired with latest trl master), all having the same problems.
SFT after GA fix

SFT before GA fix

Expected behavior

training loss value should be aligned with old values, or should be expected lower.

[KeonwooChoi]

I also experienced a reduction in loss while training, but the value loss and grad_norm became very large. According to the blog (https://huggingface.co/blog/gradient_accumulation), it seems the code has changed to 'sum' the loss instead of using 'mean', and then divide it by 'num_items'. I'm wondering if this 'sum' value is being output. If I'm mistaken, please let me know.

[benjamin-marie]

Can you share more info?
It would be useful to know at least the training arguments and model used.

[danielhanchen]

Actually would it be possible if you both @JianbangZ and @KeonwooChoi try it via Unsloth - use the same dataset if possible - thanks!

[paulcx]

here is my experiment can confirm the fix still needs to be fixed.

[muellerzr]

Will take a look since #34198 is really what fixed it on the trainer

[muellerzr]

Just an FYI for a true test it would be better to compare this against using no gradient accumulation. I wouldn't be surprised if LR hyper parameters are different now, but looking at this today/right now

[JianbangZ]

@muellerzr Does transformer need to update all the loss function in the corresponding modeling files to give the "num_items_in_batch" as argument?
Such as https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L1212C94-L1212C118
Fix is mentioned here (though in Chinese)
hiyouga/LlamaFactory#5747 (comment)

[muellerzr]

Yeah. Will do so today

[muellerzr]

@JianbangZ you can build off of pip install git+https://github.com/huggingface/transformers@fixup-loss_fn_issues. It has the rest of the models + the full fix for trainer

[JianbangZ]

pip install git+https://github.com/huggingface/transformers@fixup-loss_fn_issues

do you have to give num_items_in_batch as a trainer argument now?

[muellerzr]

No, a user never has to keep track of that

[JianbangZ]

No, a user never has to keep track of that

Tried, loss value still very big. Seems num_items_in_batch is not in effect

[muellerzr]

Can you give me a reproducer script? I tested this with SFT. Also the loss could change, I would recommend testing with a larger bs and no grad accum if possible as a baseline