remove torchao/prototype/moe_quant

[vkuzo]

Summary:

This is not used, removing.

Test Plan: CI

Reviewers:

Subscribers:

Tasks:

Tags:

[vkuzo]

Stack from ghstack (oldest at bottom):

• -> remove torchao/prototype/moe_quant #3554

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3554

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[liangan1]

@vkuzo is there any plan to add the new design for moe quant?

[liangan1]

cc @xiaowangintel

[vkuzo]

@vkuzo is there any plan to add the new design for moe quant?

Yes

1. we don't currently plan to provide any "MoE modules" (hence this PR which deletes "MoE modules")
2. we do plan to improve PyTorch Core MoE support (such as torch._grouped_mm) and support quantization as needed. So, if user writes their MoE with torch._grouped_mm, torchao plans to support quantizing that to torch._scaled_grouped_mm.
3. we have tentative plans to improve the integration of torchao checkpoints into vLLM for MoE models, using vLLM's MoE kernels

[liangan1]

@vkuzo is there any plan to add the new design for moe quant?

Yes

1. we don't currently plan to provide any "MoE modules" (hence this PR which deletes "MoE modules")
2. we do plan to improve PyTorch Core MoE support (such as torch._grouped_mm) and support quantization as needed. So, if user writes their MoE with torch._grouped_mm, torchao plans to support quantizing that to torch._scaled_grouped_mm.
3. we have tentative plans to improve the integration of torchao checkpoints into vLLM for MoE models, using vLLM's MoE kernels

Thanks for your info. But I suppose the item 2 should only works for the fp8/fp4, how about int4 and int8? besides, most of the MoE model definition in the HF/transformers is not based on the grouped_mm, is there any plan to extend the torch._grouped_mm adoption scope in the HF/transformers?

[vkuzo]

But I suppose the item 2 should only works for the fp8/fp4, how about int4 and int8?

int4 and int8 could be supported as well, whether the kernel lives in Core, torchao or somewhere else can be case by case in the short term

most of the MoE model definition in the HF/transformers is not based on the grouped_mm

yes, the MoE authoring story is very fragmented. Long term, we want PyTorch core to have the right primitives to make MoE authoring easy, and in torchao have a story to easily quantize them. Short term, we may have to have case-by-case workarounds. We do plan to work on adoption of grouped_mm.