Skip to content

Disable one splash attention test #8970

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 15, 2025
Merged

Disable one splash attention test #8970

merged 4 commits into from
Apr 15, 2025

Conversation

zpcore
Copy link
Collaborator

@zpcore zpcore commented Apr 14, 2025

Disable test_splash_attention_segment_id to unblock others PR. The failure only happens in github CI randomly. I am unable to reproduce it locally. Will bring it back once I figure it out.

@zpcore zpcore changed the title Disable splash test Disable one splash attention test Apr 14, 2025
@zpcore zpcore marked this pull request as ready for review April 14, 2025 16:44
@zpcore zpcore requested review from tengyifei and qihqi April 14, 2025 16:44
@zpcore
Copy link
Collaborator Author

zpcore commented Apr 14, 2025

I am still seeing the failure randomly. Actually if we check the error, the error always happens in flash attention sharding during backward:

res = fa_backward_callable(grad_output, q, k, v, o, l, m, q_segment_ids,
. Still checking on the root cause.

@zpcore
Copy link
Collaborator Author

zpcore commented Apr 14, 2025

Disable the whole splash attention test for now to unblock other PRs first. Will bring it back once #8971 is resolved.

@tengyifei
Copy link
Collaborator

Ack. Thanks

@tengyifei tengyifei merged commit ba5cd4c into master Apr 15, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants