Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GPU] reduction numerical correctness issue #18479

Open
IanWood1 opened this issue Sep 10, 2024 · 3 comments
Open

[GPU] reduction numerical correctness issue #18479

IanWood1 opened this issue Sep 10, 2024 · 3 comments
Assignees
Labels
bug 🐞 Something isn't working

Comments

@IanWood1
Copy link
Contributor

IanWood1 commented Sep 10, 2024

What happened?

Some changes during dispatch creation (encountered here #18341) causes f16 SDXL to produce incorrect results under aggressive fusion. This only occurs when the linalg.generic ops are at a higher dimensionality.

Steps to reproduce your issue

  1. Download a repro of the correct dispatch as well as the problematic dispatch
  2. Compile/Run both dispatches to generate output. I used this to generate input and this to compile/run each dispatch (the flags are the same as mi300 sdxl CI flags).
  3. Compare the outputs

Version information

I was using b197555, but current main should also work

cc @bangtianliu You said you might want to take a look + I think the work you're doing might solve this issue

@IanWood1 IanWood1 added the bug 🐞 Something isn't working label Sep 10, 2024
@IanWood1
Copy link
Contributor Author

@bangtianliu bangtianliu self-assigned this Sep 10, 2024
@bangtianliu
Copy link
Contributor

Sure, will take a look after I am done with my current task.

@IanWood1
Copy link
Contributor Author

IanWood1 commented Sep 10, 2024

Okay, great. BTW the "good" version has terrible performance on the sdxl benchmarks. I think it goes down the pipeline you fixed a bit ago for #17948, causing the final generic op to be done sequentially. Hopefully with the new pipeline that won't be an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐞 Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants