Skip to content

[BUG] Issue processing NF4 double quantization #183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ishan-modi opened this issue Apr 22, 2025 · 3 comments · May be fixed by #209
Open

[BUG] Issue processing NF4 double quantization #183

ishan-modi opened this issue Apr 22, 2025 · 3 comments · May be fixed by #209
Assignees
Labels
bug Something isn't working

Comments

@ishan-modi
Copy link
Contributor

ishan-modi commented Apr 22, 2025

Describe the bug

I am trying to use NF4 real quantization and came across an error because of scales not being divisible by block size. We should add padding to the scales so that we can quantize it using block quant to mitigate this issue. This can be achieved by adding reduce_block_padding function to double_quantization function.

Suggested change

scales = reduce_block_padding(
    scales.view(-1), block_sizes={-1: scale_block_size}
)

in the following line of code

Version

nvidia_modelopt == 0.27.1

@meenchen
Copy link
Collaborator

Thanks for the suggestion @ishan-modi ! I will make this update.

@ishan-modi
Copy link
Contributor Author

@meenchen any updates on this ?

@ishan-modi
Copy link
Contributor Author

@kevalmorabia97 @meenchen, when do we plan to add this ?

@ishan-modi ishan-modi linked a pull request Jun 8, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants