Separately control the activation quantization granularity #3524
Conversation
Signed-off-by: Cui, Lily <[email protected]>
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3524
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
torchao/prototype/smoothquant/api.py
Outdated
| else: | ||
| raise ValueError(f"Unexpected step: {step}") | ||
|
|
||
| if isinstance(base_config, Int8StaticActivationInt8WeightConfig): |
There was a problem hiding this comment.
I think smoothquant should not mention specific configs
There was a problem hiding this comment.
Thanks for the comment, updated.
torchao/quantization/quant_api.py
Outdated
|
|
||
| scale: torch.Tensor | ||
| granularity: Granularity = PerRow() | ||
| act_granularity: Granularity = PerTensor() |
There was a problem hiding this comment.
also we just use tuple, like this:
ao/torchao/quantization/quant_api.py
Line 1592 in 428bbcf
There was a problem hiding this comment.
Thanks for the comment, updated.
Signed-off-by: Cui, Lily <[email protected]>
torchao/quantization/quant_api.py
Outdated
| ): | ||
| assert config.granularity in {PerRow(), PerTensor()}, ( | ||
| "Only PerRow and PerTensor is supported currently" | ||
| assert isinstance(config.granularity, (tuple, list)) and len(config.granularity) == 2, ( |
There was a problem hiding this comment.
According to the code above, it can only be a list, right?
There was a problem hiding this comment.
Thanks for the comment, updated.
| if isinstance(base_config, Int8StaticActivationInt8WeightConfig): | ||
| quantize_( | ||
| basic_model, | ||
| Int8DynamicActivationInt8WeightConfig( |
There was a problem hiding this comment.
When testing static, It uses dynamic as the basic model. Otherwise, directly quantizing the static config as the basic config will result in errors due to missing act_scale computed by static_scale.
Signed-off-by: Cui, Lily <[email protected]>
Signed-off-by: Cui, Lily <[email protected]>
3 participants
Based on #3468
Separately control the activation quantization granularity and weight quantization granularity