Question help

[ganjuzhizihuai]

Hello, sorry to bother you. I encountered two problems when reading the code, and I would like to ask you for help.

1. When BC is updated, def update(self, float_output, quantized_output, post_bias=True), how do you calculate the float_output parameter:
2. The channel_size of each layer is different. How do you add a BC after each layer.

looking forward to your reply. grateful!

[GongCheng1919]

Thank you very much for your interest in our work. Here is the response to your questions:

1. The parameters in def update(self, float_output, quantized_output, post_bias=True):

   float_output: The output of a floating-point layer or block.

   quantized_output: The output of the quantized layer or block.

   post_bias: Indicates whether bias has been enabled in the quantized layer. It is a boolean type, with a default value of True.

   As with previous PTQ calibration, we need to use the original non-quantized floating-point output to update the bias vector to calibrate the quantized output, so the float_output parameter is mandatory. To get float_output, you need to prepare a floating-point layer or block corresponding to the quantized layer, as well as an input.

   Suppose the input tensor is input, the floating-point layer is float_layer, and the quantized layer corresponding to the floating-point layer is quantize_layer, then you can use following codes to obtain the parameters:

   float_output = float_layer(input)
   quantized_output=quantize_layer(input)

   If you prefer not to keep two layers simultaneously, we recommend you try our quantization framework for model quantization.
   Our quantization framework adds three quantization modules to each module to be quantized, which are used for the quantization of input, weight, and output respectively.
   You can then use quantized_act0_pre_module, quantized_weight_module, and quantized_act0_post_module to update your quantization algorithm and test in real time.
   Based on our quantization framework, you can first set quantized_act0_pre_module and quantized_weight_module as Identity modules to obtain float_output, then set them as the real quantization modules, such as fixedpoint_quantization, to get quantized_output.

   q_layer.quantized_act0_pre_module = nn.Identity()
   q_layer.quantized_weight_module = nn.Identity()
   q_layer.quantized_act0_post_module = nn.Identity()
   float_output = q_layer(input)
   
   q_layer.quantized_act0_pre_module = fixedpoint_quantization
   q_layer.quantized_weight_module = fixedpoint_quantization
   q_layer.quantized_act0_post_module = bc
   quantized_output = q_layer(input)
   
   bc.update(float_output,quantized_output)

   You can also use composite modules to implement more complex quantization, such as Sequential module.
   In our implementation, BC is applied to quantized_act0_post_module, that is, the position for output quantization, to compensate for the quantization output error.

   You can refer to the specific examples in the demo we provide here.

2. The channel_size parameter issue:
   Since BC is directly added to the layer output, the channel_size parameter can only be obtained through one forward propagation.
   In the forward propagation, we can cache the shape of layer output, and then set the channel_size parameter of BC with the cached shape data.
   Of course, this forward propagation process is not extra, because the result of this forward propagation can also be used to update the bias_vector of BC module.
   Specific examples can be found in lines 112-133 of the code here, as well as lines 120-138 of the code here.

We hope this response has been helpful to you. If you have any further questions, please feel free to contact us.

[GongCheng1919]

Hello, sorry to bother you. I encountered two problems when reading the code, and I would like to ask you for help.

1. When BC is updated, def update(self, float_output, quantized_output, post_bias=True), how do you calculate the float_output parameter:
2. The channel_size of each layer is different. How do you add a BC after each layer.

looking forward to your reply. grateful!

Please see my reply above. Thank you!