Fix optimum.quanto quantization call in cache_utils #34606
Conversation
| @@ -813,7 +813,8 @@ def _quantize(self, tensor, axis): | |||
| if is_optimum_quanto_available(): | |||
| from optimum.quanto import quantize_weight | |||
|
|
|||
| qtensor = quantize_weight(tensor, self.qtype, axis, self.q_group_size) | |||
There was a problem hiding this comment.
Can you have a look @zucchini-nlp as you did the change in this PR. Looking at optimum-quanto source code, quantize_weight do require to pass scale.
There was a problem hiding this comment.
Now when I look I think it is related to the version of optimum quanto. I see pre v0.24 had no scale
But after v0.24 we have to pass scale in before group size
@SunMarc if that is correct, prob we need to check the version also. Seems like a lot of checks but since the old quanto should be removed in next v4.47 release, could be a workaround. WDYT?
There was a problem hiding this comment.
Sounds good to me ! Also what was the issue with the prior implementation ? Was it to just simplify a bit the code ?
There was a problem hiding this comment.
You mean the prev PR I merged? It only made the code compatible with optimum-quanto v0.24 but I forgot there could be older versions. For the why optimum-quanto is changing its code, i have no idea. But would be nice if they wouldnt change it drastically anymore 😅
i guess you'll have more info about future maintaining plans in quanto :)
There was a problem hiding this comment.
@w3rew thanks for opening the PR, can you please update with suggested changes?
There was a problem hiding this comment.
Sure! Will look into it shortly.
There was a problem hiding this comment.
@zucchini-nlp optimum-quanto 0.2.5 release was synchronized with the switch from quanto to optimum-quanto in transformers at the beginning of october, and the code in cache_utils.py was correct.
It is your pull-request to align with 0.2.4 (a version that was never supported by transformers) that was actually incorrect.
There was a problem hiding this comment.
@zucchini-nlp So, just to clarify: is the version check necessary? If the code used to support only 0.2.5, the additional check seems to be redundant. More so if 0.2.4 was never supported.
There was a problem hiding this comment.
@w3rew hmm, iirc the first time I triggered tests for quant cache pip installed me the 0.2.4 version by default so that was the reason for modification. In case as you mention, we can leave only the post-0.2.5 logic but would be nice to add a version check near the import-chech for optimum quanto with explicit error message that we need the optimum quanto with v0.2.5 or higher
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| @@ -813,7 +813,8 @@ def _quantize(self, tensor, axis): | |||
| if is_optimum_quanto_available(): | |||
| from optimum.quanto import quantize_weight | |||
|
|
|||
| qtensor = quantize_weight(tensor, self.qtype, axis, self.q_group_size) | |||
| scale, zeropoint = self.optimizer(tensor, self.qtype, axis, self.q_group_size) | |||
| qtensor = quantize_weight(tensor, self.qtype, axis, scale, zeropoint, self.q_group_size) | |||
There was a problem hiding this comment.
Why is adding scale to quantize_weight necessary? What does this fix?
There was a problem hiding this comment.
optimum-quanto expects scale and shift to be passed to quantize_weight, and the current implementation does not, so quantization fails
What does this PR do?
Fixes call to optimum.quanto.quantized_weight in QuantoQuantizedCache, which currently lacks
scaleandshiftparameters and thus fails. This was introduced by cac4a48 when migrating to optimum.quanto I think.Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@SunMarc