[LoRA] Support FusedMoE LoRA Triton kernel for mxfp4

[xyang16]

Purpose

#28971 got reverted by #29697 because of breaking tests. This PR redo #28971.

• https://buildkite.com/vllm/ci/builds/41020/steps/canvas?jid=019ac943-dd5c-4673-97e4-736893ab6035
  • Fix test_modular_oai_triton_moe.py by removing big MNK size test to avoid OOM.
• https://buildkite.com/vllm/ci/builds/41020/steps/canvas?sid=019ac943-dd87-4ee9-b343-39cd7b152457
  • test_olmoe_tp.py test failure should be unrelated, because [LoRA] Support FusedMoE LoRA Triton kernel for mxfp4 #28971 only apply to mxfp4, and I can see it has been failing before.

Test Plan

pytest -s -v tests/kernels/moe/test_modular_oai_triton_moe.py

Test Result

Unit test passed

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@jeejeelee @DarkLight1337 Please take a look. Thanks a lot for reviewing!

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[gemini-code-assist]

Code Review

This pull request re-introduces support for FusedMoE LoRA with Triton kernels for mxfp4 quantization, which was previously reverted. The changes are well-structured and mainly involve:

1. Adding an UnfusedOAITritonExperts class to allow for LoRA injection by separating GEMM, activation, and reduction steps.
2. Updating the mxfp4 backend selection logic to enable the Triton backend for LoRA when available.
3. Adding a comprehensive test suite to validate the new unfused Triton kernel against a PyTorch reference implementation.

The changes look solid and align with the goal of modularizing the MoE kernels. I have a couple of suggestions for improving maintainability and robustness.