[MXFp4] Add E2E Example Script for Llama3

[dsikka]

Summary

• Requires: [MXFP4] Add calibration support compressed-tensors#509
• Add script to generate an mxfp4 quantized model
• This feature is currently experimental as support has not landed or tested in vLLM

Testing:

Sample Model:

• nm-testing/Meta-Llama-3-8B-Instruct-MXFP4

Sample Generation (Transformers):

========== SAMPLE GENERATION ==============
<|begin_of_text|>Hello my name is Sophia and I am a 3rd year student at the University of California, Berkeley. I am a double major in Linguistics and Psychology, with a minor in Education. I am very interested in the way that language and culture interact, and I believe that education is the key to creating a more just and equitable society.
I am a native speaker of English, and I have also studied Spanish, French, and Mandarin Chinese. I am very interested in the way that language can be used to bring
==========================================

Sample Config:

"quantization_config": {
    "config_groups": {
        "group_0": {
            "format": "mxfp4-pack-quantized",
            "input_activations": {
                "actorder": null,
                "block_structure": null,
                "dynamic": true,
                "group_size": 32,
                "num_bits": 4,
                "observer": null,
                "observer_kwargs": {},
                "scale_dtype": "torch.uint8",
                "strategy": "group",
                "symmetric": true,
                "type": "float",
                "zp_dtype": null
            },
            "output_activations": null,
            "targets": [
                "Linear"
            ],
            "weights": {
                "actorder": null,
                "block_structure": null,
                "dynamic": false,
                "group_size": 32,
                "num_bits": 4,
                "observer": "minmax",
                "observer_kwargs": {},
                "scale_dtype": "torch.uint8",
                "strategy": "group",
                "symmetric": true,
                "type": "float",
                "zp_dtype": null
                }
        }
    },
    "format": "mxfp4-pack-quantized",
}

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Summary of Changes

Hello @dsikka, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces an experimental example script that demonstrates the process of quantizing the Llama-3-8B-Instruct model to MXFP4 using the llmcompressor library. The addition aims to provide a practical, end-to-end workflow for model compression, while also establishing a dedicated experimental/ directory for features that are still under development or lack full support in related ecosystems.

Highlights

• New MXFP4 Quantization Example: An end-to-end example script (llama3_mxfp4.py) has been added to demonstrate how to quantize the Llama-3-8B-Instruct model using MXFP4 with the llmcompressor library.
• Experimental Feature Designation: This new MXFP4 quantization support for Llama-3 is currently experimental, as full integration and testing within LLM Compressor are still in progress. A new experimental/ directory has been introduced to house such features.
• Model Compression Workflow: The example script showcases the full workflow: loading a pre-trained model, configuring and applying a QuantizationModifier with the oneshot function, performing a sample text generation to verify functionality, and saving the quantized model in a compressed format.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds an end-to-end example script for MXFP4 quantization with Llama-3, along with a README for the new experimental directory. The changes are clear and achieve the stated goal. I've provided a couple of suggestions to improve the clarity of the README and the robustness of the example script.

[brian-dellabetta]

🚀