how to use Multi-GPU to

[liuy-2019]

There is no calculate_offload_device_map function anymore, CPU offloading is too slow, but I want to speed up the quantization with gpu.

[kylesayrs]

Hi @liuy-2019,

LLM Compressor does use a GPU. This method with just one GPU is just as fast as the previous method with multiple GPUS.

This function will dispatch the model to a GPU if it is available

llm-compressor/src/llmcompressor/pipelines/sequential/helpers.py

Line 517 in 5f9af3c

def dispatch_for_sequential(model: PreTrainedModel) -> PreTrainedModel:

Parallel multi-GPU compression is being explored here

[liuy-2019]

I use the static quantification method of fp8. Each round takes 2 minutes, and there are 65 rounds to be conducted, totaling 130 minutes. Previously, it only took about 15 minutes.

config_groups={
                "group_0": dict(
                    targets=["Linear"],
                    weights=QuantizationArgs(
                        num_bits=8,
                        type=QuantizationType.FLOAT,
                        strategy=QuantizationStrategy.TENSOR,
                        symmetric=True,
                        dynamic=False,
                        ignore=["re:.*lm_head","re:visual.*"],  # 细粒度忽略 lm_head
                        targets=["Linear"]
                    ),
                    input_activations=QuantizationArgs(
                        num_bits=8,
                        type=QuantizationType.FLOAT,
                        strategy=QuantizationStrategy.TENSOR,
                        symmetric=True,
                        dynamic=False,  # 改为静态量化
                        ignore=["re:.*lm_head","re:visual.*"],  # 可选：如果不需要对某些层激活量化
                        targets=["Linear"]
                    )
                )
            }

[kylesayrs]

Hm, I'll have to verify this independently. There may be something else going on.

Can you try running with oneshot(pipeline="basic")? This distributes the model across GPUs, which may be better for the QuantizationModifier

[liuy-2019]

It looks the same with oneshot(pipeline="basic")