Skip to content

Save O_PROJ on Fuji 70B-v2 for TRN2#1147

Closed
apoorvtintin wants to merge 5 commits intoapple:mainfrom
apoorvtintin:neuron_save_o_proj
Closed

Save O_PROJ on Fuji 70B-v2 for TRN2#1147
apoorvtintin wants to merge 5 commits intoapple:mainfrom
apoorvtintin:neuron_save_o_proj

Conversation

@apoorvtintin
Copy link
Contributor

@apoorvtintin apoorvtintin commented May 1, 2025

Saving out-projection improves training throughput while still fitting in the mesh defined by neuron-(trn2|trn2n).48xlarge-64.

@apoorvtintin apoorvtintin requested review from a team, markblee and ruomingp as code owners May 1, 2025 22:43
@apoorvtintin
Copy link
Contributor Author

apoorvtintin commented May 13, 2025

Rebased the PR to fix failing CI

@apoorvtintin
Copy link
Contributor Author

apoorvtintin commented May 15, 2025

I see the CI fails for test TestEvaluateFromFile.test_evaluate_from_eval_set with error
#22 453.1 axlearn/open_api/common.py:440: KeyError

This is unrelated to the changes in this PR, I already rebased the PR to 12th May. Can I please get some guidance on how I can fix this? Thank you!

@markblee
Copy link
Contributor

markblee commented May 15, 2025

I see the CI fails for test TestEvaluateFromFile.test_evaluate_from_eval_set with error #22 453.1 axlearn/open_api/common.py:440: KeyError

This is unrelated to the changes in this PR, I already rebased the PR to 12th May. Can I please get some guidance on how I can fix this? Thank you!

I'm disabling this test here: #1184 cc @gyin94

@apoorvtintin
Copy link
Contributor Author

apoorvtintin commented May 19, 2025

Rebased to disable flaky test

changlan and others added 5 commits July 23, 2025 23:26
@changlan
Merge pull request apple#1161 from Steboss:sbosisio/update_utils_tree
3cdc687 
GitOrigin-RevId: d143490
@changlan
Fix running max calculation when logit sink is present (#1512)
625ce3b 
Make sure sink's contribution is added once. Also added tests.

GitOrigin-RevId: 8de870c
@changlan
Relax pyarrow deps
7f359b2 
GitOrigin-RevId: 56cf7e8
@Ruixuan @changlan
pin nccl version
2e03b12 
* pin nccl version

* empty commit

* add actual pacakge

* trigger new build to address flaky test

* Update pyproject.toml

GitOrigin-RevId: b8653ad
@apoorvtintin @changlan
Save O_PROJ on Fuji 70B-v2 for neuron
13072d9
@changlan changlan force-pushed the neuron_save_o_proj branch from 5966cc5 to 13072d9 Compare July 25, 2025 18:45
@github-actions
Copy link

github-actions bot commented Oct 21, 2025

This pull request has been automatically marked as stale because it has been inactive for 60 days. It will be closed in 7 days if no further activity occurs. If you would like to continue working on this, please remove the stale label or leave a comment.

@github-actions github-actions bot added the stale label Oct 21, 2025
@github-actions github-actions bot removed the stale label Nov 1, 2025
@github-actions
Copy link

github-actions bot commented Dec 31, 2025

This pull request has been automatically marked as stale because it has been inactive for 60 days. It will be closed in 7 days if no further activity occurs. If you would like to continue working on this, please remove the stale label or leave a comment.

@github-actions github-actions bot added the stale label Dec 31, 2025
@github-actions
Copy link

github-actions bot commented Jan 8, 2026

This pull request was closed because it has been inactive for more than 7 days since being marked as stale. Please feel free to reopen it if you would like to continue.

@github-actions github-actions bot closed this Jan 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@markblee markblee Awaiting requested review from markblee

1 more reviewer

@ruomingp ruomingp ruomingp approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@apoorvtintin @markblee @ruomingp @changlan @Ruixuan