disable rope scaling for training, add yarn during export

[h-guo18]

What does this PR do?

Type of change: ?

Overview: ?

Usage

# Add a code snippet demonstrating how to use this

Testing

Before your PR is "Ready for review"

• Make sure you read and follow Contributor guidelines and your commits are signed.
• Is this change backward compatible?: Yes/No
• Did you write any new necessary tests?: Yes/No
• Did you add or update any necessary documentation?: Yes/No
• Did you update Changelog?: Yes/No

Additional Information

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch haoguo/eagle-rope

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[benchislett]

I'm not sure these are the best choices for rope theta and factor. I think these might depend on how long max_position_embeddings actually is.

Some testing may be required. Gpt Oss uses rope theta 150k, for example. This may be some tradeoff between short-context and long-context accuracy

[h-guo18]

theta=10k is the default from HF: ref

Actually my guess is that it should match the theta used in training.

factor should be a tradeoff I think.

[benchislett]

Pretty sure rope theta goes on the main config and not the rope scaling, and should be set the same for training/inference. Where did this template come from?

[h-guo18]

This is also a version difference. In transformers 4.x it stays outside, while in transformers 5 it has to be in "rope_scaling" field.

Ref:
transforemr 4.55: https://github.com/huggingface/transformers/blob/v4.55-release/src/transformers/modeling_rope_utils.py#L110

5.x:
https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_rope_utils.py#L634
https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L111

[benchislett]

I think you can go further and actually just set rope scaling to null. Not sure if there's a difference in HF

[h-guo18]

Setting it to None triggers an error. We are using Llama definition from transformers 5.0 and it requires a rope type. "rope_type":"default" here will use the traditional rope without scaling.

ref: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L85

[h-guo18]

This is a transforemr verfsion difference. In transforemr 4.x it's ok to leave it None. We are using transformer 5 here.