qnn_llama_runner on SA8295 outputs repetitive “)” or gibberish with Qwen3-0.6B after ExecuTorch export

[wanglinkun]

🐛 Describe the bug

When running qnn_llama_runner with Qwen3-0.6B converted via ExecuTorch (hybrid QNN .pte) on a Qualcomm SA8295 device, the model generates either a long sequence of “)” or random multilingual gibberish. Changing temperature does not fix it. Generation usually hits the max length instead of stopping at EOS.

Steps to Reproduce

Using llama.py to convert qwen3-0_6b
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -m SA8295 --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model qwen3-0_6b --prompt "what is 1+1" --tasks wikitext --limit 1 --compile_only --artifact /workspace/pteModels/qwen_qnn

Command used (one example):
./qnn_llama_runner --decoder_model_version qwen3 --model_path ./hybrid_llama_qnn.pte --tokenizer_path ./tokenizer.json --seq_len 1024 --kv_updater ShiftPointer --prompt "Tell me a story about the Second World War..." --temperature 0.6

Also reproduced with:

• --temperature 0.0 / 0.7
• --kv_updater SmartMask
• tokenizer_path pointing to a directory vs to tokenizer.json

Actual Behavior

The runner prints formatted_prompt in ChatML style, but the output is:
Either 998–1000 identical “)” characters when temperature=0.0
Or long sequences of random words/garbage when temperature>0
It always generates up to max length and never stops at EOS.

What I Tried

• Adjusted sampling (temperature 0.0/0.6/0.7, also tried lower temperature like 0.1). No fix; behavior changes from repeated “)” to random gibberish but remains incorrect.
• Shortened seq_len to 128; still nonsense, just shorter.
• Switched kv_updater between SmartMask and ShiftPointer.
• Pointed tokenizer_path to a single tokenizer.json vs a directory. BOS/EOS ids changed between runs but output remains nonsense.

Versions

Collecting environment information...
PyTorch version: 2.10.0.dev20251003+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.31.6
Libc version: glibc-2.35

Python version: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.84-004.ali5000.alios7.x86_64-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 40
On-line CPU(s) list: 0-39
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 10
Socket(s): 2
Stepping: 4
CPU max MHz: 3000.0000
CPU min MHz: 800.0000
BogoMIPS: 4400.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d
Virtualization: VT-x
L1d cache: 640 KiB (20 instances)
L1i cache: 640 KiB (20 instances)
L2 cache: 20 MiB (20 instances)
L3 cache: 27.5 MiB (2 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-39
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable

Versions of relevant libraries:
[pip3] executorch==1.1.0a0+519c7ff
[pip3] numpy==2.2.6
[pip3] pytorch_tokenizers==0.1.0
[pip3] torch==2.10.0.dev20251003+cpu
[pip3] torchao==0.14.0+git01849b2b1
[pip3] torchaudio==2.8.0.dev20251003+cpu
[pip3] torchdata==0.11.0+cpu
[pip3] torchsr==1.0.4
[pip3] torchtune==0.7.0+cpu
[pip3] torchvision==0.25.0.dev20251003+cpu
[conda] executorch 1.1.0a0+519c7ff pypi_0 pypi
[conda] numpy 2.2.6 pypi_0 pypi
[conda] pytorch-tokenizers 0.1.0 pypi_0 pypi
[conda] torch 2.10.0.dev20251003+cpu pypi_0 pypi
[conda] torchao 0.14.0+git01849b2b1 pypi_0 pypi
[conda] torchaudio 2.8.0.dev20251003+cpu pypi_0 pypi
[conda] torchdata 0.11.0+cpu pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchtune 0.7.0+cpu pypi_0 pypi
[conda] torchvision 0.25.0.dev20251003+cpu pypi_0 pypi

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin

[haowhsu-quic]

I think the HTP arch of SA8295 is v68 which does not support LPBQ (minimum requirement is v69). I guess you're probably aware of it and change the quantization scheme to per-channel configuration which is hard to get good accuracy.

[wanglinkun]

Yes, I used the default quantization scheme, but the transformation model failed and many operators failed validation. So I modified the quantification scheme and successfully converted the model into PTE format.

So, do you have any suggestions to solve the current problem

[haowhsu-quic]

I think 16a8w probably could work, but it will be a trade-off between accuracy and performance.

[wanglinkun]

Unfortunately, the model conversion with the 16A8W quantization scheme failed.

(executorch) root@f4e04719cd03:/work/pytorch/executorch# python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -m SA8295 --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model qwen3-0_6b --prompt "what is 1+1" --tasks wikitext --limit 1 --compile_only --artifact /work/qwen_qnn I tokenizers:regex.cpp:27] Registering override fallback regex NOTE: Using slow Hadamard transform for SpinQuant. For better performance on GPU, install fast_hadamard_transform: pip install git+https://github.com/Dao-AILab/fast-hadamard-transform.git`
QNN_SDK_ROOT=/work/qairt/2.32.0.250228
INFO:root:*** qwen3-0_6b ***
+-----------------------------+-------------------------------+
| Config | Value |
+=============================+===============================+
| custom_annotation | (partial(annotate_down_proj)) |
+-----------------------------+-------------------------------+
| decoder_model_version | qwen3 |
+-----------------------------+-------------------------------+
| get_kv_io_bit_width | 16 |
+-----------------------------+-------------------------------+
| get_logits_output_bit_width | 16 |
+-----------------------------+-------------------------------+
| group_size | 32 |
+-----------------------------+-------------------------------+
| instruct_model | True |
+-----------------------------+-------------------------------+
| masked_softmax | True |
+-----------------------------+-------------------------------+
| num_sharding | 1 |
+-----------------------------+-------------------------------+
| ptq | QuantDtype.use_16a8w |
+-----------------------------+-------------------------------+
| r1 | False |
+-----------------------------+-------------------------------+
| r2 | False |
+-----------------------------+-------------------------------+
| r3 | False |
+-----------------------------+-------------------------------+
| repo_id | Qwen/Qwen3-0.6B |
+-----------------------------+-------------------------------+
| seq_mse_candidates | 1000 |
+-----------------------------+-------------------------------+
| transform_weight | False |
+-----------------------------+-------------------------------+
WARNING:root:Masked softmax is supported after QNN SDK 2.35. Given sdk version v2.32.0.250228225014_116386 is lower the target version. Disabling the feature.
✔ Using cached converted model: /root/.cache/meta_checkpoints/Qwen_Qwen3-0.6B.pth
INFO:root:Time for loading checkpoint: 0.22562360763549805
INFO:root:Quantizing the model...
WARNING:accelerate.utils.other:Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
INFO:lm-eval:Using device 'cpu'
INFO:lm-eval:Using model type 'default'
INFO:lm-eval:Model parallel was set to False, max memory was not set, and device map was set to {'': 'cpu'}
torch_dtype is deprecated! Use dtype instead!
INFO:lm-eval:Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
INFO:lm-eval:Using pre-initialized model
WARNING:lm-eval:[Task: wikitext] metric word_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity
WARNING:lm-eval:[Task: wikitext] metric word_perplexity is defined, but higher_is_better is not. using default higher_is_better=False
WARNING:lm-eval:[Task: wikitext] metric byte_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity
WARNING:lm-eval:[Task: wikitext] metric byte_perplexity is defined, but higher_is_better is not. using default higher_is_better=False
WARNING:lm-eval:[Task: wikitext] metric bits_per_byte is defined, but aggregation is not. using default aggregation=bits_per_byte
WARNING:lm-eval:[Task: wikitext] metric bits_per_byte is defined, but higher_is_better is not. using default higher_is_better=False
INFO:lm-eval:Building contexts for wikitext on rank 0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1965.47it/s]
INFO:lm-eval:Running loglikelihood_rolling requests
0%| | 0/1 [00:00<?, ?it/sINFO:root:kv inference result:
2008, Daylight Robbery by filmmaker Paris Leonti, and Donkey Punch directed by Olly Blackburn. In May 2008, Boulter made a guest appearance on a two-part episode arc of the television series Waking the Dead, followed by an appearance on the television series Survivors in November 2008. He had a recurring role in ten episodes of the television series Casualty in 2010, as "Kieron Fletcher". Boulter starred in the 2011 film Mercenaries directed by Paris Leonti.

== Career ==

=== 2000 – 2005 ===

In 2000 Boulter had a guest-starring role on the television series The Bill; he portrayed "Scott Parry" in the episode, "In Safe Hands". Boulter starred as "Scott" in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre. A review of Boulter's performance in The Independent on Sunday described him as "horribly menacing" in the role, and he received critical reviews in The Herald, and Evening Standard. He appeared in the television series Judge John Deed in 2002 as "Addem Armitage" in the episode "Political Expediency", and had a role as a different character "Toby Steele" on The Bill.
He had a recurring role in 2003 on two episodes of The Bill, as character "Connor Price". In 2004 Boulter landed a role as "Craig" in the episode "Teddy's Story" of the television series The Long Firm; he starred alongside actors Mark Strong and Derek Jacobi. Boulter starred as "Darren", in the 2005 theatre productions of the Philip Ridley play Mercury Fur. It was performed at the Drum Theatre in Plymouth, and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw, Shane Zaza, Harry Kent, Fraser Ayres, Sophie Stanton and Dominic Hall. Boulter received a favorable review in The Daily Telegraph: "The acting is shatteringly intense, with wired performances from Ben Whishaw (now unrecognisable from his performance as Trevor Nunn's Hamlet), Robert Boulter, Shane Zaza and Fraser Ayres." The Guardian noted, "Ben Whishaw and Robert Boulter offer tenderness amid the savagery."

=== 2006 – present ===

In 2006 Boulter starred in the play Citizenship written by Mark Ravenhill. The play was part of a series which featured different playwrights, titled Burn / Chatroom / Citizenship. In a 2006 interview, fellow actor Ben Whishaw identified Boulter as one of his favorite co-stars: "I loved working with a guy called Robert Boulter, who was in the triple bill of Burn, Chatroom and Citizenship at the National. He played my brother in Mercury Fur." He portrayed "Jason Tyler" on the 2006 episode of the television series, Doctors, titled "Something I Ate". Boulter starred as "William" in the 2007 production of How to Curse directed by Josie Rourke. How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. In a review of the production for The Daily Telegraph, theatre critic Charles Spencer noted, "Robert Boulter brings a touching vulnerability to the stage as William."
Boulter starred in two films in 2008, Daylight Robbery by filmmaker Paris Leonti, and Donkey Punch directed by Olly Blackburn. Boulter portrayed a character named "Sean" in Donkey Punch, who tags along with character "Josh" as the "quiet brother ... who hits it off with Tammi". Boulter guest starred on a two-part episode arc "Wounds" in May 2008 of the television series Waking the Dead as character "Jimmy Dearden". He appeared on the television series Survivors as "Neil" in November 2008. He had a recurring role in ten episodes of the television series Casualty in 2010, as "Kieron Fletcher". He portrayed an emergency physician applying for a medical fellowship. He commented on the inherent difficulties in portraying a physician on television: "Playing a doctor is a strange experience. Pretending you know what you 're talking about when you don't is very bizarre but there are advisers on set who are fantastic at taking you through procedures and giving you the confidence to stand there and look like you know what you 're doing." Boulter starred in the 2011 film Mercenaries directed by Paris Leonti.

== Filmography ==

=== Film ===

=== Television ===

=== Theatre ===

INFO:root:kv inference result:
= Robert Boulter =

Robert Boulter is an English film, television and theatre actor. He had a guest-starring role on the television series The Bill in 2000. This was followed by a starring role in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre. He had a guest role in the television series Judge John Deed in 2002. In 2004 Boulter landed a role as "Craig" in the episode "Teddy's Story" of the television series The Long Firm; he starred alongside actors Mark Strong and Derek Jacobi. He was cast in the 2005 theatre productions of the Philip Ridley play Mercury Fur, which was performed at the Drum Theatre in Plymouth and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw, Shane Zaza, Harry Kent, Fraser Ayres, Sophie Stanton and Dominic Hall.
In 2006, Boulter starred alongside Whishaw in the play Citizenship written by Mark Ravenhill. He appeared on a 2006 episode of the television series, Doctors, followed by a role in the 2007 theatre production of How to Curse directed by Josie Rourke. How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. Boulter starred in two films in 2008, Daylight Robbery by filmmaker Paris Leonti, and Donkey Punch directed by Olly Blackburn. In May 2008, Boulter made a guest appearance on a two-part episode arc of the television series Waking the Dead, followed by an appearance on the television series Survivors in November 2008. He had a recurring role in ten episodes of the television series Casualty in 2010, as "Kieron Fletcher". Boulter starred in the 2011 film Mercenaries directed by Paris Leonti.

== Career ==

=== 2000 – 2005 ===

In 2000 Boulter had a guest-starring role on the television series The Bill; he portrayed "Scott Parry" in the episode, "In Safe Hands". Boulter starred as "Scott" in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre. A review of Boulter's performance in The Independent on Sunday described him as "horribly menacing" in the role, and he received critical reviews in The Herald, and Evening Standard. He appeared in the television series Judge John Deed in 2002 as "Addem Armitage" in the episode "Political Expediency", and had a role as a different character "Toby Steele" on The Bill.
He had a recurring role in 2003 on two episodes of The Bill, as character "Connor Price". In 2004 Boulter landed a role as "Craig" in the episode "Teddy's Story" of the television series The Long Firm; he starred alongside actors Mark Strong and Derek Jacobi. Boulter starred as "Darren", in the 2005 theatre productions of the Philip Ridley play Mercury Fur. It was performed at the Drum Theatre in Plymouth, and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw, Shane Zaza, Harry Kent, Fraser Ayres, Sophie Stanton and Dominic Hall. Boulter received a favorable review in The Daily Telegraph: "The acting is shatteringly intense, with wired performances from Ben Whishaw (now unrecognisable from his performance as Trevor Nunn's Hamlet), Robert Boulter, Shane Zaza and Fraser Ayres." The Guardian noted, "Ben Whishaw and Robert Boulter offer tenderness amid the savagery."

=== 2006 – present ===

In 2006 Boulter starred in the play Citizenship written by Mark Ravenhill. The play was part of a series which featured different playwrights, titled Burn / Chatroom / Citizenship. In a 2006 interview, fellow actor Ben Whishaw identified Boulter as one of his favorite co-stars: "I loved working with a guy called Robert Boulter, who was in the triple bill of Burn, Chatroom and Citizenship at the National. He played my brother in Mercury Fur." He portrayed "Jason Tyler" on the 2006 episode of the television series, Doctors, titled "Something I Ate". Boulter starred as "William" in the 2007 production of How to Curse directed by Josie Rourke. How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. In a review of the production for The
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [40:13<00:00, 2413.90s/it]
INFO:root:Perplexity evaluation summary for prepare_pt2e_tasks
INFO:root:wikitext: {'alias': 'wikitext', 'word_perplexity,none': 16.323304778718818, 'word_perplexity_stderr,none': 'N/A', 'byte_perplexity,none': 1.7136239299514666, 'byte_perplexity_stderr,none': 'N/A', 'bits_per_byte,none': 0.7770505318985513, 'bits_per_byte_stderr,none': 'N/A'}
INFO:root:Prompt after applying template: <|im_start|>user
what is 1+1<|im_end|>
<|im_start|>assistant

INFO:root:kv inference result:
user
what is 1+1
assistant

Okay, the user is asking what 1 plus 1 is. Let me think about how to approach this. First, I need to confirm the basic arithmetic operation here. Adding two 1s together.

In mathematics, 1 + 1 equals 2. That's straightforward. But maybe the user is thinking of something else? Like in a different context, such as a puzzle or a specific problem? I should check if there's any hidden meaning or if they might be referring to something else.

Wait, sometimes in puzzles or riddles, numbers can have different interpretations. For example, in some languages, numbers might be read differently. But in English, the standard interpretation is just adding the two numbers.

Also, considering the user might be a student or someone learning basic math, it's important to explain the process clearly. Maybe they need to know the answer first before diving into the explanation.

I should make sure to present the answer clearly and confirm that the result is indeed 2. Let me structure the response to be simple and direct, avoiding any unnecessary details.

1 + 1 equals 2.
INFO:root:Quantizing the model...
WARNING:accelerate.utils.other:Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
INFO:lm-eval:Using device 'cpu'
INFO:lm-eval:Using model type 'default'
INFO:lm-eval:Model parallel was set to False, max memory was not set, and device map was set to {'': 'cpu'}
INFO:lm-eval:Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
INFO:lm-eval:Using pre-initialized model
WARNING:lm-eval:[Task: wikitext] metric word_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity
WARNING:lm-eval:[Task: wikitext] metric word_perplexity is defined, but higher_is_better is not. using default higher_is_better=False
WARNING:lm-eval:[Task: wikitext] metric byte_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity
WARNING:lm-eval:[Task: wikitext] metric byte_perplexity is defined, but higher_is_better is not. using default higher_is_better=False
WARNING:lm-eval:[Task: wikitext] metric bits_per_byte is defined, but aggregation is not. using default aggregation=bits_per_byte
WARNING:lm-eval:[Task: wikitext] metric bits_per_byte is defined, but higher_is_better is not. using default higher_is_better=False
INFO:lm-eval:Building contexts for wikitext on rank 0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2114.06it/s]
INFO:lm-eval:Running loglikelihood_rolling requests
0%| | 0/1 [00:00<?, ?it/s]INFO:root:kv inference result:
2008, Daylight Robbery by filmmaker Paris Leonti, and Donkey Punch directed by Olly Blackburn. In May 2008, Boulter made a guest appearance on a two-part episode arc of the television series Waking the Dead, followed by an appearance on the television series Survivors in November 2008. He had a recurring role in ten episodes of the television series Casualty in 2010, as "Kieron Fletcher". Boulter starred in the 2011 film Mercenaries directed by Paris Leonti.

== Career ==

=== 2000 – 2005 ===

In 2000 Boulter had a guest-starring role on the television series The Bill; he portrayed "Scott Parry" in the episode, "In Safe Hands". Boulter starred as "Scott" in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre. A review of Boulter's performance in The Independent on Sunday described him as "horribly menacing" in the role, and he received critical reviews in The Herald, and Evening Standard. He appeared in the television series Judge John Deed in 2002 as "Addem Armitage" in the episode "Political Expediency", and had a role as a different character "Toby Steele" on The Bill.
He had a recurring role in 2003 on two episodes of The Bill, as character "Connor Price". In 2004 Boulter landed a role as "Craig" in the episode "Teddy's Story" of the television series The Long Firm; he starred alongside actors Mark Strong and Derek Jacobi. Boulter starred as "Darren", in the 2005 theatre productions of the Philip Ridley play Mercury Fur. It was performed at the Drum Theatre in Plymouth, and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw, Shane Zaza, Harry Kent, Fraser Ayres, Sophie Stanton and Dominic Hall. Boulter received a favorable review in The Daily Telegraph: "The acting is shatteringly intense, with wired performances from Ben Whishaw (now unrecognisable from his performance as Trevor Nunn's Hamlet), Robert Boulter, Shane Zaza and Fraser Ayres." The Guardian noted, "Ben Whishaw and Robert Boulter offer tenderness amid the savagery."

=== 2006 – present ===

In 2006 Boulter starred in the play Citizenship written by Mark Ravenhill. The play was part of a series which featured different playwrights, titled Burn / Chatroom / Citizenship. In a 2006 interview, fellow actor Ben Whishaw identified Boulter as one of his favorite co-stars: "I loved working with a guy called Robert Boulter, who was in the triple bill of Burn, Chatroom and Citizenship at the National. He played my brother in Mercury Fur." He portrayed "Jason Tyler" on the 2006 episode of the television series, Doctors, titled "Something I Ate". Boulter starred as "William" in the 2007 production of How to Curse directed by Josie Rourke. How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. In a review of the production for The Daily Telegraph, theatre critic Charles Spencer noted, "Robert Boulter brings a touching vulnerability to the stage as William."
Boulter starred in two films in 2008, Daylight Robbery by filmmaker Paris Leonti, and Donkey Punch directed by Olly Blackburn. Boulter portrayed a character named "Sean" in Donkey Punch, who tags along with character "Josh" as the "quiet brother ... who hits it off with Tammi". Boulter guest starred on a two-part episode arc "Wounds" in May 2008 of the television series Waking the Dead as character "Jimmy Dearden". He appeared on the television series Survivors as "Neil" in November 2008. He had a recurring role in ten episodes of the television series Casualty in 2010, as "Kieron Fletcher". He portrayed an emergency physician applying for a medical fellowship. He commented on the inherent difficulties in portraying a physician on television: "Playing a doctor is a strange experience. Pretending you know what you 're talking about when you don't is very bizarre but there are advisers on set who are fantastic at taking you through procedures and giving you the confidence to stand there and look like you know what you 're doing." Boulter starred in the 2011 film Mercenaries directed by Paris Leonti.

== Filmography ==

=== Film ===

=== Television ===

=== Theatre ===

INFO:root:kv inference result:
= Robert Boulter =

Robert Boulter is an English film, television and theatre actor. He had a guest-starring role on the television series The Bill in 2000. This was followed by a starring role in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre. He had a guest role in the television series Judge John Deed in 2002. In 2004 Boulter landed a role as "Craig" in the episode "Teddy's Story" of the television series The Long Firm; he starred alongside actors Mark Strong and Derek Jacobi. He was cast in the 2005 theatre productions of the Philip Ridley play Mercury Fur, which was performed at the Drum Theatre in Plymouth and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw, Shane Zaza, Harry Kent, Fraser Ayres, Sophie Stanton and Dominic Hall.
In 2006, Boulter starred alongside Whishaw in the play Citizenship written by Mark Ravenhill. He appeared on a 2006 episode of the television series, Doctors, followed by a role in the 2007 theatre production of How to Curse directed by Josie Rourke. How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. Boulter starred in two films in 2008, Daylight Robbery by filmmaker Paris Leonti, and Donkey Punch directed by Olly Blackburn. In May 2008, Boulter made a guest appearance on a two-part episode arc of the television series Waking the Dead, followed by an appearance on the television series Survivors in November 2008. He had a recurring role in ten episodes of the television series Casualty in 2010, as "Kieron Fletcher". Boulter starred in the 2011 film Mercenaries directed by Paris Leonti.

== Career ==

=== 2000 – 2005 ===

In 2000 Boulter had a guest-starring role on the television series The Bill; he portrayed "Scott Parry" in the episode, "In Safe Hands". Boulter starred as "Scott" in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre. A review of Boulter's performance in The Independent on Sunday described him as "horribly menacing" in the role, and he received critical reviews in The Herald, and Evening Standard. He appeared in the television series Judge John Deed in 2002 as "Addem Armitage" in the episode "Political Expediency", and had a role as a different character "Toby Steele" on The Bill.
He had a recurring role in 2003 on two episodes of The Bill, as character "Connor Price". In 2004 Boulter landed a role as "Craig" in the episode "Teddy's Story" of the television series The Long Firm; he starred alongside actors Mark Strong and Derek Jacobi. Boulter starred as "Darren", in the 2005 theatre productions of the Philip Ridley play Mercury Fur. It was performed at the Drum Theatre in Plymouth, and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw, Shane Zaza, Harry Kent, Fraser Ayres, Sophie Stanton and Dominic Hall. Boulter received a favorable review in The Daily Telegraph: "The acting is shatteringly intense, with wired performances from Ben Whishaw (now unrecognisable from his performance as Trevor Nunn's Hamlet), Robert Boulter, Shane Zaza and Fraser Ayres." The Guardian noted, "Ben Whishaw and Robert Boulter offer tenderness amid the savagery."

=== 2006 – present ===

In 2006 Boulter starred in the play Citizenship written by Mark Ravenhill. The play was part of a series which featured different playwrights, titled Burn / Chatroom / Citizenship. In a 2006 interview, fellow actor Ben Whishaw identified Boulter as one of his favorite co-stars: "I loved working with a guy called Robert Boulter, who was in the triple bill of Burn, Chatroom and Citizenship at the National. He played my brother in Mercury Fur." He portrayed "Jason Tyler" on the 2006 episode of the television series, Doctors, titled "Something I Ate". Boulter starred as "William" in the 2007 production of How to Curse directed by Josie Rourke. How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. In a review of the production for The
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:20<00:00, 20.19s/it]
INFO:root:Perplexity evaluation summary for prepare_pt2e_tasks
INFO:root:wikitext: {'alias': 'wikitext', 'word_perplexity,none': 16.323312070910433, 'word_perplexity_stderr,none': 'N/A', 'byte_perplexity,none': 1.7136240776010998, 'byte_perplexity_stderr,none': 'N/A', 'bits_per_byte,none': 0.7770506562043459, 'bits_per_byte_stderr,none': 'N/A'}
INFO:root:Prompt after applying template: <|im_start|>user
what is 1+1<|im_end|>
<|im_start|>assistant

INFO:root:kv inference result:
user
what is 1+1
assistant

Okay, the user is asking what 1 plus 1 is. Let me think about how to approach this. First, I need to confirm the basic arithmetic operation here. Adding two 1s together.

In mathematics, 1 + 1 equals 2. That's straightforward. But maybe the user is thinking of something else? Like in a different context, such as a puzzle or a specific problem? I should check if there's any hidden meaning or if they might be referring to something else.

Wait, sometimes in puzzles or riddles, numbers can have different interpretations. For example, in some languages, numbers might be read differently. But in English, the standard interpretation is just adding the two numbers.

Also, considering the user might be a student or someone learning basic math, it's important to explain the process clearly. Maybe they need to know the answer first before diving into the explanation.

I should make sure to present the answer clearly and confirm that the result is indeed 2. Let me structure the response to be simple and direct, avoiding any unnecessary details.

1 + 1 equals 2.
INFO:root:Time for quantizing: 3551.399327278137
WARNING:root:Op aten.unbind.int was requested for preservation by partitioner. This request is ignored because it is in a blocklist.
WARNING:root:Op aten.unbind.int was requested for preservation by partitioner. This request is ignored because it is in a blocklist.
WARNING:root:Op aten.unbind.int was requested for preservation by partitioner. This request is ignored because it is in a blocklist.
WARNING:root:Op aten.unbind.int was requested for preservation by partitioner. This request is ignored because it is in a blocklist.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 1
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in SAVE MODE.
[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[QNN Partitioner Op Support]: aten.squeeze_copy.dims | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.convolution.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.unsqueeze_copy.default | True
[QNN Partitioner Op Support]: aten.rms_norm.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.convolution.default | True
[QNN Partitioner Op Support]: aten.mul.Tensor | True
[QNN Partitioner Op Support]: aten.convolution.default | True
[QNN Partitioner Op Support]: aten.mul.Tensor | True
[QNN Partitioner Op Support]: aten.sigmoid.default | True
[QNN Partitioner Op Support]: aten.convolution.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.rms_norm.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.convolution.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.cat.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_895 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_894 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_893 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_892 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_891 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_890 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_889 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_888 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_887 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_886 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_885 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_884 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_883 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_882 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_881 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_880 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_879 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_878 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_877 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_876 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_875 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_874 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_873 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_872 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_871 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_870 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_869 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_868 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_867 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.add.Tensor | True
[QNN Partitioner Op Support]: aten.div.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_866 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[ERROR] [Qnn ExecuTorch]: [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]: QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]: Failed to validate op aten_matmul_default_865 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110

...

[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
WARNING:root:Op aten.unbind.int was requested for preservation by partitioner. This request is ignored because it is in a blocklist.
Traceback (most recent call last):
File "/work/pytorch/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 1355, in main
export_llama(args)
File "/work/pytorch/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 1329, in export_llama
compile(args, decoder_model_config, pte_filename, tokenizer, chat_template)
File "/work/pytorch/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 819, in compile
edge_prog_mgr = to_edge_transform_and_lower_to_qnn(
File "/work/pytorch/executorch/backends/qualcomm/utils/utils.py", line 445, in to_edge_transform_and_lower_to_qnn
return to_edge_transform_and_lower(
File "/work/pytorch/executorch/exir/program/_program.py", line 115, in wrapper
return func(*args, **kwargs)
File "/work/pytorch/executorch/exir/program/_program.py", line 1388, in to_edge_transform_and_lower
_sanity_check_graph_for_non_decomp_ops(
File "/work/pytorch/executorch/exir/program/_program.py", line 1049, in _sanity_check_graph_for_non_decomp_ops
raise RuntimeError(warning_str)
RuntimeError: Node aten_matmul_default with op <EdgeOpOverload: aten.matmul.default>: schema = aten::matmul(Tensor self, Tensor other) -> Tensor was not decomposed or delegated.
This op was registered by the partitioner QnnPartitioner to not be decomposed.
The following ops: {<OpOverload(op='torchao.quantize_affine', overload='default')>, <OpOverload(op='aten.linear', overload='default')>, <OpOverload(op='aten.hardswish', overload='default')>, <OpOverload(op='aten.leaky_relu', overload='default')>, <OpOverload(op='aten.pixel_unshuffle', overload='default')>, <OpOverload(op='aten.rms_norm', overload='default')>, <OpOverload(op='aten.prelu', overload='default')>, <OpOverload(op='torchao.dequantize_affine', overload='default')>, <OpOverload(op='aten.matmul', overload='default')>, <OpOverload(op='aten.instance_norm', overload='default')>, <OpOverload(op='aten._safe_softmax', overload='default')>, <OpOverload(op='aten.stack', overload='default')>, <OpOverload(op='aten.elu', overload='default')>, <OpOverload(op='aten.im2col', overload='default')>, <OpOverload(op='aten.hardsigmoid', overload='default')>, <OpOverload(op='aten.col2im', overload='default')>, <OpOverload(op='aten.adaptive_avg_pool2d', overload='default')>, <OpOverload(op='aten.upsample_bicubic2d', overload='vec')>, <OpOverload(op='aten.floor_divide', overload='default')>, <OpOverload(op='aten.pixel_shuffle', overload='default')>} were specified to not be decomposed in kv_forward.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/work/pytorch/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 1366, in
main()
File "/work/pytorch/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 1361, in main
raise Exception(e)
Exception: Node aten_matmul_default with op <EdgeOpOverload: aten.matmul.default>: schema = aten::matmul(Tensor self, Tensor other) -> Tensor was not decomposed or delegated.
This op was registered by the partitioner QnnPartitioner to not be decomposed.
The following ops: {<OpOverload(op='torchao.quantize_affine', overload='default')>, <OpOverload(op='aten.linear', overload='default')>, <OpOverload(op='aten.hardswish', overload='default')>, <OpOverload(op='aten.leaky_relu', overload='default')>, <OpOverload(op='aten.pixel_unshuffle', overload='default')>, <OpOverload(op='aten.rms_norm', overload='default')>, <OpOverload(op='aten.prelu', overload='default')>, <OpOverload(op='torchao.dequantize_affine', overload='default')>, <OpOverload(op='aten.matmul', overload='default')>, <OpOverload(op='aten.instance_norm', overload='default')>, <OpOverload(op='aten._safe_softmax', overload='default')>, <OpOverload(op='aten.stack', overload='default')>, <OpOverload(op='aten.elu', overload='default')>, <OpOverload(op='aten.im2col', overload='default')>, <OpOverload(op='aten.hardsigmoid', overload='default')>, <OpOverload(op='aten.col2im', overload='default')>, <OpOverload(op='aten.adaptive_avg_pool2d', overload='default')>, <OpOverload(op='aten.upsample_bicubic2d', overload='vec')>, <OpOverload(op='aten.floor_divide', overload='default')>, <OpOverload(op='aten.pixel_shuffle', overload='default')>} were specified to not be decomposed in kv_forward.

`

[shewu-quic]

Based on QNN document, 16bit second input for matmul op require minimum arch V73. I think that you can add annotate_kv_8bit into custom annotation to avoid 16bit input for matmul.

diff --git a/examples/qualcomm/oss_scripts/llama/__init__.py b/examples/qualcomm/oss_scripts/llama/__init__.py
index 80a37361d..eae7d0439 100644
--- a/examples/qualcomm/oss_scripts/llama/__init__.py
+++ b/examples/qualcomm/oss_scripts/llama/__init__.py
@@ -445,8 +445,8 @@ class Qwen3_0_6B(LLMModelConfig):
 
     num_sharding = 1
     # quant config
-    ptq = QuantDtype.use_16a4w_block
-    group_size = 32
+    ptq = QuantDtype.use_16a8w
+    group_size = None
     masked_softmax = True
     seq_mse_candidates = 1000
     r1 = False
@@ -456,9 +456,7 @@ class Qwen3_0_6B(LLMModelConfig):
         torch.uint16, weight_dtype=torch.int8, act_observer=MinMaxObserver
     )
     custom_annotation = (
-        partial(
-            annotate_down_proj, quantization_config=quantization_config_down_proj_16a8w
-        ),
+        annotate_kv_8bit,
     )
 
 

[wanglinkun]

Based on QNN document, 16bit second input for matmul op require minimum arch V73. I think that you can add annotate_kv_8bit into custom annotation to avoid 16bit input for matmul.

diff --git a/examples/qualcomm/oss_scripts/llama/__init__.py b/examples/qualcomm/oss_scripts/llama/__init__.py
index 80a37361d..eae7d0439 100644
--- a/examples/qualcomm/oss_scripts/llama/__init__.py
+++ b/examples/qualcomm/oss_scripts/llama/__init__.py
@@ -445,8 +445,8 @@ class Qwen3_0_6B(LLMModelConfig):
 
     num_sharding = 1
     # quant config
-    ptq = QuantDtype.use_16a4w_block
-    group_size = 32
+    ptq = QuantDtype.use_16a8w
+    group_size = None
     masked_softmax = True
     seq_mse_candidates = 1000
     r1 = False
@@ -456,9 +456,7 @@ class Qwen3_0_6B(LLMModelConfig):
         torch.uint16, weight_dtype=torch.int8, act_observer=MinMaxObserver
     )
     custom_annotation = (
-        partial(
-            annotate_down_proj, quantization_config=quantization_config_down_proj_16a8w
-        ),
+        annotate_kv_8bit,
     )
 
 

Thank you for your support. Now I can successfully convert the model and run inference. The example results are as follows

`I 00:00:09.364317 executorch:runner.cpp:444] RSS after finishing text generation: 1065.562500 MiB (0 if unsupported)
PyTorchObserver {"prompt_tokens":17,"generated_tokens":220,"model_load_start_ms":1761817304613,"model_load_end_ms":1761817306645,"inference_start_ms":1761817306645,"inference_end_ms":1761817313975,"prompt_eval_end_ms":1761817306729,"first_token_ms":1761817306729,"aggregate_sampling_time_ms":80,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
I 00:00:09.364377 executorch:stats.h:108] Prompt Tokens: 17 Generated Tokens: 220
I 00:00:09.364380 executorch:stats.h:114] Model Load Time: 2.032000 (seconds)
I 00:00:09.364390 executorch:stats.h:124] Total inference time: 7.330000 (seconds) Rate: 30.013643 (tokens/second)
I 00:00:09.364395 executorch:stats.h:132] Prompt evaluation: 0.084000 (seconds) Rate: 202.380952 (tokens/second)
I 00:00:09.364414 executorch:stats.h:143] Generated 220 tokens: 7.246000 (seconds) Rate: 30.361579 (tokens/second)
I 00:00:09.364430 executorch:stats.h:151] Time to first generated token: 0.084000 (seconds)
I 00:00:09.364443 executorch:stats.h:158] Sampling time over 237 tokens: 0.080000 (seconds)
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[WARNING] [Qnn ExecuTorch]: QnnDsp Function not called, PrepareLib isn't loaded!

atlas:/data/llama/test # cat outputs.txt
cat outputs.txt
<|im_start|>user
what is 1+1 /nothink<|im_end|>
<|im_start|>assistant::

Okay, so I need to figure out what 1 plus 1 is. Let me think. First, the question is written as 1 + 1, which is a simple addition. Adding 1 and 1 together would be 2. But wait, maybe there's more to it? Sometimes, people might interpret these numbers in a different way, like in a math problem where there are more numbers involved. Let me check the original question again. It says "what is 1+1". So, it's straightforward. 1 plus 1 equals 2. But maybe the user is confused because they wrote 1+1 instead of 1+1? Or perhaps there's a typo? Let me confirm. If it's 1 + 1, then yes, 2. If there's a different context, like in a sequence or something else, but with the given information, 1+1 is 2. So the answer should be 2.

1 + 1 equals 2.

Answer: 2<|im_end|>`

But when I try to use Chinese as prompt input , It seems that tokenizer cannot recognize it properly

[wanglinkun]

Perhaps the shell environment in the system does not support Chinese, but when I copied the output file 'outputs. txt' containing Chinese to my computer, I was able to open it normally and display Chinese. ^_^

<|im_start|>user
can you speek chinest, answer me using chinese<|im_end|>
<|im_start|>assistant:
<think>
好的，用户让我用中文回答，但我的回答是英文。首先，我需要确认用户的需求。用户可能想用中文回答，但我的回复是英文，这可能是个误解或者输入错误。接下来，我应该先礼貌地回应，说明我的回答是英文，然后询问用户是否需要帮助。同时，我需要确保回答符合中文的表达习惯，避免使用任何可能引起误解的术语。最后，保持友好和专业的态度，让用户感到被理解和支持。
</think>

您好，我理解您的要求，但我的回答是英文的。如果您有其他问题，欢迎随时告诉我！祝您学习顺利！<|im_end|>

[shewu-quic]

I’m so glad to hear that. Thanks for sharing! Is there anything else I can help with?

[wanglinkun]

I tried using the same method, switching to 16a4w, and successfully converted the model, but the inference still outputs meaningless characters. I want to know if it is possible to support 16a4w quantization, and if so, what should be done.

@@ -452,8 +452,8 @@ class Qwen3_0_6B(LLMModelConfig):

     num_sharding = 1
     # quant config
-    ptq = QuantDtype.use_16a4w_block
-    group_size = 32
+    ptq = QuantDtype.use_16a4w
+    group_size = None
     masked_softmax = True
     seq_mse_candidates = 1000
     r1 = False
@@ -463,9 +463,7 @@ class Qwen3_0_6B(LLMModelConfig):
         torch.uint16, weight_dtype=torch.int8, act_observer=MinMaxObserver
     )
     custom_annotation = (
-        partial(
-            annotate_down_proj, quantization_config=quantization_config_down_proj_16a8w
-        ),
+        annotate_kv_8bit,
     )

[shewu-quic]

Since the V68 device doesn't support LPBQ, you can only use per-channel configuration, which makes achieving good accuracy challenging. I'm glad to hear that 16a8w is working, though it does involve a trade-off between accuracy and performance. You might consider applying 16a4w to some conv nodes alongside 16a8w to improve performance.

[cccclai]

I think it may needs some experiments because LPBQ is not supported here. There are a few configuration we can try

• spinquant (maybe add r1, r2)
• SeqMSE (I think it's already enabled?)
• Try to do mixed precision, maybe need to use 8bit for the outliers