How can I tell the converted model precision?

[shaharyuval]

I'd like to convert a Pytorch model to IR representation and would like to ensure similar behavior. I currently see a degradation in Openvino relative to Pytorch. My first thought is to verify the model is still in FP32 precision. How can I do so?

This is my conversion and inference code:

torch.onnx.export(model, torch.randn(1, 3, 244, 244), onnx_model_path)
ov_model = openvino.convert_model(onnx_model_path)
compiled_model = core.compile_model(model=ov_model, device_name='GPU')

#inference
results = compiled_model([ov_input_img])

Is there any attribute controlling the IR model precision? I am unable to find documentation for this function.

I see some sources refer to another conversion function (is it older or deprecated?):

ov_model = openvino.tools.mo.convert_model(input_model=onnx_model_path,
                                    input_shape=[1, 3, 244, 244],
                                    layout='NCHW', ...)

Is it possible to control the precision with this one? Where can I find documentation for the parameters?

[ilya-lavrenov]

Hi @shaharyuval
openvino.convert_model does not change the precision of an original model. Right, openvino.tools.mo.convert_model is deprected one and openvino.convert_model is recommended to use since it's significantly simpler and has less parameters.

I see that you are using GPU, which by default performs computations in FP16. Please, try the following:

• Inference on CPU which performs FP32 computations.
• Try to change inference precision on GPU using core.compile_model(ov_model, 'GPU', {inference_precision: Type.f32})

[akuporos]

am I supposed to invoke the call to compile_model using the inference_precision object or the string 'inference_precision'

Since 2022.1 the recommended way is to use object from openvino.runtime.properties. Since 2023.2 they are available from the openvino.properties namespace.
Which version of OpenVINO do you use?

[shaharyuval]

Hi @akuporos,
Apparently I was using 2023.1. After upgrading to 2023.2, I see that inference_precision appears under openvino.properties.
Also after this upgrade, core.compile_model(ov_model, 'GPU', {inference_precision: Type.f32}) no longer issues an error,

However, I still see a degradation relative to the Openvino CPU model and the Pytorch model (the latter two are in agreement).

Could there be other implicit conversions going on when moving the network to the GPU?

[akuporos]

@maxnick @vladimir-paramuzov could you please share your opinion here?

[vladimir-paramuzov]

@shaharyuval GPU plugin may do few conversions for unsupported integer types (like int64 -> int32) in such case, but with {inference_precision: Type.f32} set, fp16 is not supposed to be used. You can check actual execution precision using runtime model feature:

runtime_model = compiled_model.get_runtime_model()
serialize(runtime_model, "exec_graph.xml")

and then inspect this "exec_graph.xml" as regular IR and check used precision for each layer. But for now it just looks like a bug in GPU plugin, so we would need some info from your side to fix that (IR at least)

[shaharyuval]

Hi @vladimir-paramuzov ,
I serialized the model and the xml file shows FP32 for all layers.
Regards,

Shahar