[Performance]: QWEN3VL 8B INT4 Model Inference Performance Below Expected on Intel Core Ultra 5 225H

[liaochengliang]

OpenVINO Version

2025.3.0-19807-44526285f24-releases/2025/3

Operating System

Other (Please specify in description)

Device used for inference

iGPU

OpenVINO installation

PyPi

Programming Language

Python

Hardware Architecture

x86 (64 bits)

Model used

QWEN3-VL

Model quantization

Yes

Target Platform

(base) stit10@stit10-XiaoXinPro-16-IAH10:/mnt/workspace/weicheng/openvino_notebooks$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 42 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 14
On-line CPU(s) list: 0-13
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) Ultra 5 225H
CPU family: 6
Model: 197
Thread(s) per core: 1
Core(s) per socket: 14
Socket(s): 1
Stepping: 2
CPU(s) scaling MHz: 45%
CPU max MHz: 4900.0000
CPU min MHz: 400.0000
BogoMIPS: 7372.80
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_per
fmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_
1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept
vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk
avx_vnni lam wbnoinvd dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid bus_lock_detect mov
diri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 448 KiB (12 instances)
L1i: 768 KiB (12 instances)
L2: 22 MiB (7 instances)
L3: 18 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-13
Vulnerabilities:
Gather data sampling: Not affected
Ghostwrite: Not affected
Indirect target selection: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS Not affected; BHI BHI_DIS_S
Srbds: Not affected
Tsa: Not affected
Tsx async abort: Not affected
Vmscape: Mitigation; IBPB before exit to userspace
(base) stit10@stit10-XiaoXinPro-16-IAH10:/mnt/workspace/weicheng/openvino_notebooks$ lscpu -e
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ MHZ
0 0 0 0 0:0:0:0 yes 4900.0000 400.0000 4815.9858
1 0 0 1 4:4:1:0 yes 4900.0000 400.0000 400.0000
2 0 0 2 16:16:4:0 yes 4900.0000 400.0000 3006.8589
3 0 0 3 20:20:5:0 yes 4900.0000 400.0000 400.0000
4 0 0 4 2:0 yes 4400.0000 400.0000 4311.0000
5 0 0 5 18:18:2:0 yes 4400.0000 400.0000 3490.6660
6 0 0 6 2:0 yes 4400.0000 400.0000 4301.2539
7 0 0 7 22:22:2:0 yes 4400.0000 400.0000 400.0000
8 0 0 8 24:24:3:0 yes 4400.0000 400.0000 400.0000
9 0 0 9 26:26:3:0 yes 4400.0000 400.0000 400.0000
10 0 0 10 28:28:3:0 yes 4400.0000 400.0000 400.0000
11 0 0 11 30:30:3:0 yes 4400.0000 400.0000 400.0000
12 0 0 12 64:64:8 yes 2500.0000 400.0000 400.0000
13 0 0 13 66:66:8 yes 2500.0000 400.0000 2500.1201

Performance issue description

Description:
We performed INT4 quantization and model conversion of the QWEN3VL 8B model using the OpenVINO Notebook.

Test Environment:
Device: Intel® Core Ultra 5 225H
Memory: 32GB
Model: QWEN3VL 8B
Quantization: INT4

We modified the OpenVINO Notebook to create a custom performance testing script.

Test Results:
On the above configuration, the measured inference speed is only 8 token/s.

run_qwen3vl_int4.py

Questions / Concerns:
Is this performance expected?
Could there be issues in our current testing program that result in the low throughput?
If the testing program is incorrect, could you provide an official recommended performance testing script for accurate benchmarking?

Attachments:
Current performance testing script
Test result logs

log.txt

Step-by-step reproduction

run_qwen3vl_int4.py

Issue submission checklist

• I'm reporting a performance issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[krgkaushik]

Hi @liaochengliang,

Thank you for the detailed report. Qwen3-VL (assuming Qwen2-VL or a very recent variant) 8B at INT4 on a Lunar Lake (Core Ultra 5 225H) iGPU should generally perform better than 8 tokens/s, although this metric heavily depends on the input prompt (image resolution) and whether the bottleneck is compute or memory bandwidth.

To give you an accurate assessment, I need to see the implementation details.

Action Required
Could you please paste the content of run_qwen3vl_int4.py here? (I cannot see the file contents from the attachment list alone).

Immediate Checks
While you provide the script, please verify the following:

Target Device: Are you explicitly compiling for "GPU"?

The Core Ultra 5 225H has a powerful Arc 140V iGPU. Running on "CPU" or "NPU" (which might fall back or be bandwidth-bound for 8B models) could explain the 8 t/s.

Stateful Export: Did you export the model using stateful=True?

If the model was exported without internal KV caching, Python overhead will severely degrade performance.

Input Resolution: Are you testing with a high-resolution image?

Large images increase the pre-fill time significantly, which averages down the "tokens per second" if you are including the first token latency in your calculation.

Recommended Benchmark Step
To isolate the model performance from Python application overhead, please run the standard OpenVINO benchmark tool on the main language model .xml file:

benchmark_app -m <path_to_qwen_int4_ov_model.xml> -d GPU -hint tput -t 15

If benchmark_app shows >> 8 FPS: Your Python script likely has an overhead or configuration issue.

If benchmark_app shows ~8 FPS: The bottleneck is at the inference level (likely bandwidth saturation or driver version).

Once you share the script, I can pinpoint the exact cause.