Numeric failure analysis for ONNX models #950
Description
1> We have few models failing with 99.99% match and the failure is due to mainly sqrt/pow operator. For these model, we should lower the tolerance from .001 to .01
flaubert_Opset17_transformers/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 98303 of 98304 = 99.99898274739584%
model--Bartlarge--Shubham09/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 6433864 of 6433920 = 99.99912961305083%
migx_bench_bert-large-uncased_16_384/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 6307839 of 6307840 = 99.99998414671266%
migx_bench_bert-large-uncased_32_384/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 12615679 of 12615680 = 99.99999207335634%
mobilebert_Opset18_transformers/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 66046 of 66048 = 99.9969718992248%
mt5_Opset16_transformers/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 1703848 of 1703936 = 99.99483548677884%
t5_Opset17_transformers/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 1703898 of 1703936 = 99.99776986929086%
umt5_Opset16_transformers/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 1703882 of 1703936 = 99.99683086688702%
xlmroberta_Opset16_transformers/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 99071 of 99072 = 99.99899063307494%
2> iree-org/iree#20429
fbnetv3_b.ra2_in1k_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 79 of 1000 = 7.9%
fbnetv3_b.ra2_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 116 of 1000 = 11.600000000000001%
fbnetv3_d.ra2_in1k_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 79 of 1000 = 7.9%
fbnetv3_d.ra2_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 74 of 1000 = 7.3999999999999995%
fbnetv3_g.ra2_in1k_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 134 of 1000 = 13.4%
fbnetv3_g.ra2_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 164 of 1000 = 16.400000000000002%
shufflenet-v2-12-qdq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 84 of 1000 = 8.4%
hardcorenas_c_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 209 of 1000 = 20.9%
hardcorenas_d_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 155 of 1000 = 15.5%
hardcorenas_e_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 214 of 1000 = 21.4%
RRDB_ESRGAN_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 101513 of 2408448 = 4.21487198394983%
RRDB_ESRGAN_vaiq_int8/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 336635 of 2408448 = 13.97725838382228%
EfficientNet_b2_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 92 of 1000 = 9.2%
EfficientNet_b0_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 87 of 1000 = 8.7%
EfficientNet_b3_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 282 of 1000 = 28.199999999999996%
EfficientNet_b4_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 270 of 1000 = 27.0%
EfficientNet_b7_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 570 of 1000 = 56.99999999999999%
EfficientNet_v2_l_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 420 of 1000 = 42.0%
EfficientNet_v2_m_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 390 of 1000 = 39.0%
EfficientNet_v2_s_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 344 of 1000 = 34.4%
efficientnet_b0.ra_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 206 of 1000 = 20.599999999999998%
efficientnet_b2.ra_in1k_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 216 of 1000 = 21.6%
efficientnet_b2.ra_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 167 of 1000 = 16.7%
efficientnet_b3.ra2_in1k_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 105 of 1000 = 10.5%
efficientnet_b3.ra2_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 111 of 1000 = 11.1%
efficientnet_b4.ra2_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 63 of 1000 = 6.3%
efficientnetv2_rw_m.agc_in1k_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 247 of 1000 = 24.7%
efficientnetv2_rw_m.agc_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 188 of 1000 = 18.8%
efficientnet-lite4-11-qdq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 997 of 1000 = 99.7%
efficientnet_b5.sw_in12k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 1746 of 11821 = 14.770323999661619%
gc_efficientnetv2_rw_t.agc_in1k_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 377 of 1000 = 37.7%
gc_efficientnetv2_rw_t.agc_in1k_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 330 of 1000 = 33.0%
encoderdecoder_Opset16_transformers/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 8723712 of 8723712 = 100.0%
3> LSTM bidirectional
sequencer2d_l_Opset16_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 3 of 1000 = 0.3%
sequencer2d_m_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 45 of 1000 = 4.5%
sequencer2d_s_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 288 of 1000 = 28.799999999999997%
sequencer2d_l_Opset17_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 3 of 1000 = 0.3%
sequencer2d_m_Opset16_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 4 of 1000 = 0.4%
sequencer2d_m_Opset17_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 4 of 1000 = 0.4%
sequencer2d_s_Opset16_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 0 of 1000 = 0.0%
sequencer2d_s_Opset17_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 0 of 1000 = 0.0%
iree-org/iree#20432
resnet10t_Opset17_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 8 of 1000 = 0.8%
resnet14t_Opset18_timm/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 6 of 1000 = 0.6%
resnet10t_train_vaiq/inference_comparison.log:matching values with (rtol,atol) = [0.001, 0.001]: 141 of 1000 = 14.099999999999998%