Support FP8 Quantization and Inference Run on Intel Gaudi (HPU) using INC (Intel Neural Compressor) #12010

nirda7 · 2025-01-13T14:39:53Z

This PR adds support to FP8 Quantization and Inference run on Intel Gaudi (HPU) using INC (Intel Neural Compressor).
Currently, quantization is validated only in Llama models.

Measurement files are required to run quantized models with vLLM on Gaudi accelerators. The FP8 model calibration procedure is described in the vllm-hpu-extention package: https://github.com/HabanaAI/vllm-hpu-extension/tree/main/calibration/README.md
Measurements are device dependent - Don't use measurements collected on Gaudi3 on Gaudi2 accelerators as it might cause accuracy issues.

Running Inference in FP8 with INC:
Specify quantization method "inc" and kv cache dtype "fp8_inc" as parameters to the the LLM object.
It will require to set an environment variable "QUANT_CONFIG" which will point to a 'JSON config file' (https://docs.habana.ai/en/latest/PyTorch/Inference_on_PyTorch/Quantization/Inference_Using_FP8.html#supported-json-config-file-options) in QUANTIZE mode. Make sure there are measurement files/scale files in the folder specified as the "dump_stats_path" in the json config file. (If none exists, scale files are generated during the inference run using the measurement files)
At the end of the run, the model executor's shutdown method must be called.

More information on vLLM quantization using INC will be shown here (added in this PR): https://github.com/vllm-project/vllm/blob/main/docs/source/features/quantization/inc.md

This PR adds a new flag "weights_load_device" which allows uploading the model's (unquantized) weights onto a different device than the device on which the model will run. If not provided the behavior is kept by using the device specified in the device config.

github-actions · 2025-01-13T14:40:10Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

vllm/worker/hpu_model_runner.py

mergify · 2025-02-19T09:02:44Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @nirda7.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mgoin · 2025-06-27T16:54:37Z

docs/features/quantization/README.md

 - [BitBLAS](bitblas.md)
 - [GGUF](gguf.md)
 - [GPTQModel](gptqmodel.md)
+- [Inc](inc.md)


From the doc, seems like it should be INC

Suggested change

- [Inc](inc.md)

- [INC](inc.md)

mgoin · 2025-06-27T16:56:49Z

vllm/config.py


 BlockSize = Literal[1, 8, 16, 32, 64, 128]
-CacheDType = Literal["auto", "fp8", "fp8_e4m3", "fp8_e5m2"]
+CacheDType = Literal["auto", "fp8", "fp8_e4m3", "fp8_e5m2", "fp8_inc"]


Based on the comment here, can we remove the new cache dtype now? #12010 (comment)

Since the HPU worker for v1 is moved to plugin and v0 will be deprecated soon. We want to make the map "fp8_inc to fp8_e4m3" being more visible.

Alternatively, do you think we can update the mapping function above conditional like:

STR_DTYPE_TO_TORCH_DTYPE = { "half": torch.half, "bfloat16": torch.bfloat16, "float": torch.float, "fp8": torch.uint8, "fp8_e4m3": torch.uint8 if not current_platform.is_support_fp8_e4m3() else torch.float8_e4m3fn "fp8_e5m2": torch.uint8, "int8": torch.int8, }

It's okay, let's just keep fp8_inc then

mgoin · 2025-06-27T18:19:19Z

vllm/model_executor/layers/quantization/inc.py

Can you please add a header to this file explaining its purpose? It is rather confusing otherwise and this is a good place to define how this quant method works

mgoin · 2025-06-27T18:20:20Z

vllm/model_executor/model_loader/base_loader.py

                model = initialize_model(vllm_config=vllm_config,
                                         model_config=model_config)
+
+            logger.info("Loading weights on %s ...", load_device)


Make this debug

mgoin · 2025-06-27T18:20:43Z

vllm/model_executor/model_loader/weight_utils.py

    # GGUF doesn't have config file
-    if model_config.quantization == "gguf":
-        return quant_cls.from_config({})
+    if model_config.quantization in ("gguf", "inc"):
+        return quant_cls()


Is this valid for gguf ?

yes as gguf also returns always the cls straightforward
https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/gguf.py#L51

mergify · 2025-07-02T05:27:38Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @nirda7.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Uri Livne <[email protected]>

mergify · 2025-07-14T02:46:26Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @nirda7.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

xuechendi · 2025-07-14T15:23:24Z

@mgoin , please help to review the PR again, we have updated the codes and resolved most of your comments.
there is one comment will need to check your opinion:
#12010 (comment)

mgoin

LGTM, thanks for the iterations. Please resolve the merge conflict

mgoin · 2025-07-14T18:06:02Z

vllm/config.py


 BlockSize = Literal[1, 8, 16, 32, 64, 128]
-CacheDType = Literal["auto", "fp8", "fp8_e4m3", "fp8_e5m2"]
+CacheDType = Literal["auto", "fp8", "fp8_e4m3", "fp8_e5m2", "fp8_inc"]


It's okay, let's just keep fp8_inc then

Signed-off-by: Uri Livne <[email protected]>

xuechendi · 2025-07-15T18:09:10Z

@robertgshaw2-redhat @simon-mo @WoosukKwon , We have got one approval from Michael
May you check the PR as well, and help to add "ready" tag if possible, thanks so much

xuechendi · 2025-07-16T19:28:09Z

@mgoin , thanks for the Review, all CI are passed

… INC (Intel Neural Compressor) (vllm-project#12010) Signed-off-by: Nir David <[email protected]> Signed-off-by: Uri Livne <[email protected]> Co-authored-by: Uri Livne <[email protected]> Signed-off-by: x22x22 <[email protected]>

… INC (Intel Neural Compressor) (vllm-project#12010) Signed-off-by: Nir David <[email protected]> Signed-off-by: Uri Livne <[email protected]> Co-authored-by: Uri Livne <[email protected]>

… INC (Intel Neural Compressor) (vllm-project#12010) Signed-off-by: Nir David <[email protected]> Signed-off-by: Uri Livne <[email protected]> Co-authored-by: Uri Livne <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>

… INC (Intel Neural Compressor) (vllm-project#12010) Signed-off-by: Nir David <[email protected]> Signed-off-by: Uri Livne <[email protected]> Co-authored-by: Uri Livne <[email protected]> Signed-off-by: Paul Pak <[email protected]>

… INC (Intel Neural Compressor) (vllm-project#12010) Signed-off-by: Nir David <[email protected]> Signed-off-by: Uri Livne <[email protected]> Co-authored-by: Uri Livne <[email protected]> Signed-off-by: Diego-Castan <[email protected]>

… INC (Intel Neural Compressor) (vllm-project#12010) Signed-off-by: Nir David <[email protected]> Signed-off-by: Uri Livne <[email protected]> Co-authored-by: Uri Livne <[email protected]>

mergify bot added the documentation Improvements or additions to documentation label Jan 13, 2025

mgoin self-requested a review January 13, 2025 19:12

nirda7 force-pushed the dev/hpu_fp8 branch from f664d3c to 2ca5636 Compare January 14, 2025 01:24

zhouyu5 reviewed Jan 14, 2025

View reviewed changes

vllm/worker/hpu_model_runner.py Outdated Show resolved Hide resolved

nirda7 force-pushed the dev/hpu_fp8 branch from 144d28d to c6813cb Compare January 21, 2025 18:41

nirda7 requested review from alexm-redhat, comaniac, njhill, robertgshaw2-redhat, tlrmchlsmth, youkaichao and zhuohan123 as code owners January 21, 2025 18:41

nirda7 force-pushed the dev/hpu_fp8 branch from 1400336 to f3a310f Compare January 21, 2025 19:03

mergify bot added the ci/build label Jan 22, 2025

nirda7 force-pushed the dev/hpu_fp8 branch 3 times, most recently from 5c04292 to ab1c832 Compare January 27, 2025 00:12

nirda7 force-pushed the dev/hpu_fp8 branch 2 times, most recently from d1662df to 8a2ce5f Compare February 6, 2025 15:13

nirda7 force-pushed the dev/hpu_fp8 branch 2 times, most recently from 58e7d72 to 0c1f134 Compare February 16, 2025 10:09

mergify bot added the needs-rebase label Feb 19, 2025

nirda7 force-pushed the dev/hpu_fp8 branch from 0c1f134 to 1ae7a5d Compare February 19, 2025 09:18

mergify bot removed the needs-rebase label Feb 19, 2025

nirda7 force-pushed the dev/hpu_fp8 branch from 24fd72f to 3516ac5 Compare February 20, 2025 11:13

nirda7 requested a review from zhouyu5 February 24, 2025 14:35

nirda7 force-pushed the dev/hpu_fp8 branch 2 times, most recently from 67cd337 to a6feb86 Compare March 4, 2025 10:31

mgoin reviewed Jun 27, 2025

View reviewed changes

mergify bot added the needs-rebase label Jul 2, 2025

ulivne added 2 commits July 8, 2025 10:00

Merge remote-tracking branch 'upstream/main' into dev/hpu_fp8

a4bc3cc

Signed-off-by: Uri Livne <[email protected]>

Additional fix for CR comment

664707b

Signed-off-by: Uri Livne <[email protected]>

mergify bot removed the needs-rebase label Jul 8, 2025

mergify bot added the needs-rebase label Jul 14, 2025

mgoin approved these changes Jul 14, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into dev/hpu_fp8

bddcef0

Signed-off-by: Uri Livne <[email protected]>

mergify bot removed the needs-rebase label Jul 15, 2025

restore function and pre-commit fixes

f557ab6

Signed-off-by: Uri Livne <[email protected]>

mgoin added quantization ready ONLY add when PR is ready to merge/full CI is needed labels Jul 16, 2025

mgoin merged commit 01513a3 into vllm-project:main Jul 16, 2025
86 checks passed

Uh oh!

Support FP8 Quantization and Inference Run on Intel Gaudi (HPU) using INC (Intel Neural Compressor) #12010

Support FP8 Quantization and Inference Run on Intel Gaudi (HPU) using INC (Intel Neural Compressor) #12010

Uh oh!

Conversation

nirda7 commented Jan 13, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 13, 2025

Uh oh!

Uh oh!

mergify bot commented Feb 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jul 2, 2025

Uh oh!

mergify bot commented Jul 14, 2025

Uh oh!

xuechendi commented Jul 14, 2025

Uh oh!

mgoin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuechendi commented Jul 15, 2025

Uh oh!

xuechendi commented Jul 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

nirda7 commented Jan 13, 2025 •

edited by github-actions bot

Loading

mgoin left a comment •

edited

Loading