About PTQ

[Jsy0220]

Hi,
I want to evaluate the best performance improvement of one quantized model compared with the source one.

So is it possible to provide random or fake dataset to nncf.quantize just for performance evaluation regardless of accuracy ? In other words, does the dataset affect quantization performance in addition to accuracy？

[alexsu52]

Hello @Jsy0220,

Yes, you can provide random or fake dataset to nncf.quantize for performance evaluation regardless of accuracy. The dataset does not affect the performance of the quantized model.

Anyway If you are going to use fake dataset, I would suggest to use valid inputs where model behavior depends from the inputs.

[Jsy0220]

@alexsu52
Okay, I have another question about quantized model generated by nncf.quantize:
I noticed that only part of ops in .xml file have I8 precision, what inference precision of ops stored in I8/FP32 in runtime for openvino? For all these quantized and non-quantized ops, are they still controlled by config of inference precision via core.compile_model ?

[alexsu52]

Hello @Jsy0220,

Short answer is yes, quantized ops has inference precision INT8, and non-quantized ops original precision. More information about how quantized model transformed during compilation you can find in documentation about OpenVINO™ Low Precision Transformations

The inference precision parameter from the config for ov.compile_model controls non-quantized ops.