Skip to content

[GPU] Support RMS Normalization Fusion without Learnable Affine Parameters #33861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
andrew-k-park wants to merge 11 commits into
base: master
Choose a base branch
from
Open

[GPU] Support RMS Normalization Fusion without Learnable Affine Parameters #33861

andrew-k-park wants to merge 11 commits into from

Conversation

@andrew-k-park
Copy link
Contributor

@andrew-k-park andrew-k-park commented Jan 29, 2026

Details:

  • This PR enhances RMS normalization fusion to support pattern without learnable affine parameter (gamma), enabling optimization of transformer architecture like LTX-Video
  • The existing RMS fusion pass only supported pattern with constant gamma parameter. However, some transformer model (e.g., LTX-Video's attention layers) use RMS normalization followed by dynamic scaling operation where the scale factor is non-constant. These pattern was previously unfused, missing optimization opportunity
  • When elementwise_affine=False (equivalent to Pytorch RMS's attribute), RMS normalization does not include learnable gamma parameters. The gamma is implicitly fixed to ones, reducing the decomposed graph pattern
    from:
    x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x, 1/√) → Multiply(gamma)
    to:
    x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x, 1/√) [NO gamma multiplication]
image-2026-01-26-22-53-05-973

Tickets:

@andrew-k-park andrew-k-park requested review from a team as code owners January 29, 2026 04:20
@github-actions github-actions bot added category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations labels Jan 29, 2026
@andrew-k-park andrew-k-park requested a review from a team as a code owner January 29, 2026 12:46
@github-actions github-actions bot added the category: TEMPLATE OpenVINO Template plugin label Jan 29, 2026
@andrew-k-park andrew-k-park added this to the 2026.1 milestone Jan 29, 2026
e-ddykim
e-ddykim reviewed Jan 29, 2026
View reviewed changes
e-ddykim
e-ddykim approved these changes Jan 30, 2026
View reviewed changes
Copy link
Contributor

@e-ddykim e-ddykim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for the GPU part

@andrew-k-park
Copy link
Contributor Author

andrew-k-park commented Jan 30, 2026

@CuriousPanCake Could you review this PR?

CuriousPanCake
CuriousPanCake requested changes Jan 30, 2026
View reviewed changes
Copy link
Contributor

@CuriousPanCake CuriousPanCake left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, review the comments for the Transformations part

@andrew-k-park andrew-k-park force-pushed the ltx_video_rms branch 3 times, most recently from e0d9a76 to c499127 Compare February 1, 2026 23:15
@andrew-k-park
Copy link
Contributor Author

andrew-k-park commented Feb 2, 2026

@CuriousPanCake Could you please review the updated code that apply your comments again?

@andrew-k-park andrew-k-park force-pushed the ltx_video_rms branch 2 times, most recently from e5ebd49 to e008c30 Compare February 2, 2026 23:46
@pereanub
Copy link
Contributor

pereanub commented Feb 3, 2026

@dmatveev, this PR is breaking some NPUW func test. Please have a look.

@andrew-k-park andrew-k-park force-pushed the ltx_video_rms branch 2 times, most recently from e7d36da to 757801a Compare February 4, 2026 03:47
@andrew-k-park
Copy link
Contributor Author

andrew-k-park commented Feb 4, 2026

@dmatveev, this PR is breaking some NPUW func test. Please have a look.
In my opinion, this NPU test appears to be related to the CPU plugin’s transformation pipeline. In the CPU plugin, I have made a change so that the relevant pattern is no longer matched in RMSFusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@e-ddykim e-ddykim e-ddykim approved these changes

@CuriousPanCake CuriousPanCake Awaiting requested review from CuriousPanCake

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

category: GPU OpenVINO GPU plugin category: TEMPLATE OpenVINO Template plugin category: transformations OpenVINO Runtime library - Transformations

Projects

None yet

Milestone

2026.1

Development

Successfully merging this pull request may close these issues.

4 participants

@andrew-k-park @pereanub @e-ddykim @CuriousPanCake