exporting qwen3-0.6B using the w8a8 quantization format, a segmentation error occurs.

[cxn-selfie]

When the quantization format was changed to w8a8, a segmentation fault occurred during the conversion to PTE.
Quantitative configuration：
class Qwen3_0_6BQuantRecipe(StaticLLMQuantRecipe):
# default_quant_dtype = QuantDtype.use_16a4w
default_quant_dtype = QuantDtype.use_8a8w

def __init__(self, verbose: bool = False):
    super().__init__()

    self.recipe = (
        QuantRecipe(
            self.default_quant_dtype,
            False,
            act_observer=MinMaxObserver,
            granularity=QuantGranularity.PER_TENSOR,
            verbose=verbose,
        )
         .add_node_target(
            {
                torch.ops.aten.conv2d.default,
            },
            self.default_quant_dtype,
            False,
            act_observer=MinMaxObserver,
            granularity=QuantGranularity.PER_CHANNEL,
        )
        .add_regex(
            {
                r"layers\..*\.feed_forward\.w2_conv",
            },
            self.default_quant_dtype,
            False,
            act_observer=MinMaxObserver,
            granularity=QuantGranularity.PER_CHANNEL,
        )
       
    )

fault message：
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_matmul_default_895, aten.matmul.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_cat_default_1147, aten.cat.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_view_copy_default_1931, aten.view_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3332, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_1004, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3333, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_view_copy_default_1932, aten.view_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_add_tensor_1174, aten.add.Tensor
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_rms_norm_default_727, aten.rms_norm.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_view_copy_default_1933, aten.view_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3334, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_1005, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_1006, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_sigmoid_default_27, aten.sigmoid.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_mul_tensor_2743, aten.mul.Tensor
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_mul_tensor_2744, aten.mul.Tensor
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_1007, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3335, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_view_copy_default_1934, aten.view_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_add_tensor_1175, aten.add.Tensor
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_rms_norm_default_728, aten.rms_norm.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_unsqueeze_copy_default, aten.unsqueeze_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3336, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_1008, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3337, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten__to_copy_default_1589, aten._to_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_squeeze_copy_dims, aten.squeeze_copy.dims

====== DDR bandwidth summary ======
spill_bytes=0
fill_bytes=0
write_total_bytes=2447360
read_total_bytes=672777216

[ERROR] [Qnn ExecuTorch]: grdep_clone_op.cc:340::ERROR:failed to clone op(0x18FDB00000AA27)

段错误 (核心已转储)

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin

[cccclai]

Does other quantization format work or is it just 8a8w? Also which qnn version are you using?

[cxn-selfie]

I tried w4a16 and w8a16 and they both worked. The qnn sdk version I'm using is 2.37.0.250724.

[cccclai]

@cxn-selfie can you share your intention for 8a8w? For LLMs, I think 8a8w is the least optimum option, because the activation is hard to quantize and we usually want to have more bits for activation. In the meanwhile, weight is relatively easier and we can use lower bits.

We should fix the segfault though, cc: @shewu-quic @haowhsu-quic @DannyYuyang-quic @winskuo-quic

[cxn-selfie]

I’d like to compare inference speeds under different quantization parameters. Do you have any insights into how these parameters affect NPU performance? On an NPU, do different bit-widths or symmetry settings trigger distinct data paths?

[cccclai]

I think at 0.6B scale, 16a4w is still probably the optimum, because it's still memory bound. But if it's smaller, like SmolLM2 135M, it might be different. Like here

executorch/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py

Line 556 in 6903f28

default_quant_dtype = QuantDtype.use_16a8w

I remember use_16a8w is faster

Anyway, I think it still worth benchmark experiments...

[cxn-selfie]

Thank you for your answer.
Does this mean that for small models (such as SmolLM2 135M), the computational overhead of dequantization (restoring the 4-bit weights to a computational data type) is more significant, while for large models (qwen3-0.6b), memory limitations are the more critical influencing factor?
Additionally, I have a basic question about quantization. I understand that in some schemes (such as the so-called "8a8w," where both activation and weights are 8 bits), the inference process can be performed entirely in the integer domain, thus eliminating the need for dequantization. Is my understanding correct?

[cccclai]

Does this mean that for small models (such as SmolLM2 135M), the computational overhead of dequantization (restoring the 4-bit weights to a computational data type) is more significant, while for large models (qwen3-0.6b), memory limitations are the more critical influencing factor?

I think it's more about the computation cost vs memory limitation, and for computation cost, it's not limited to dequantization.

Additionally, I have a basic question about quantization. I understand that in some schemes (such as the so-called "8a8w," where both activation and weights are 8 bits), the inference process can be performed entirely in the integer domain, thus eliminating the need for dequantization. Is my understanding correct?

I think all inference process for qnn is done in integer domain (if we didn't compile with use_fp16). There can be dequant compute, but I think it's more about from 4bit to higher level bits. From what I know, 16a4w_block is implemented in the software level and hence there can be extra cost for the "dequant", QCOM folks might know more details here.

[Ulthunzy]

Summary
I am using Executorch’s ETDump feature to analyze execution time distribution of a QNN model.
I generated the ETDump file using the following code snippet in examples/qualcomm/oss_scripts/llama/qnn_llama_runner.cpp

 #include <executorch/backends/qualcomm/runtime/QnnExecuTorch.h>
 #include <executorch/examples/qualcomm/oss_scripts/llama/runner/runner.h>
 #include <executorch/extension/llm/runner/irunner.h>
 #include <executorch/runtime/platform/log.h>
 #include <gflags/gflags.h> 
 #include <fstream>
 #include <vector>
 #include <executorch/devtools/etdump/etdump_flatcc.h>

 DEFINE_string(
  etdump_path,
  "outputs/etdump.etdump",
  "If set, dump ETDump to this path.");
  ...
 
  ...
 
 template <typename T>
 void start_runner(
     std::unique_ptr<executorch::extension::Module> module,
     std::vector<std::string>& prompts,
     **executorch::etdump::ETDumpGen* etdump_raw = nullptr**) {

   ......

   if (etdump_raw) {
     auto result = etdump_raw->get_etdump_data();
     if (result.buf != nullptr && result.size > 0) {
       FILE* f = fopen(FLAGS_etdump_path.c_str(), "w+");
       fwrite((uint8_t*)result.buf, 1, result.size, f);
       fclose(f);
       free(result.buf);
     }
   }
 }
 
 int main(int argc, char** argv) {
   std::vector<std::string> prompts = CollectPrompts(argc, argv);
   gflags::ParseCommandLineFlags(&argc, &argv, true);
   if (!gflags::GetCommandLineFlagInfoOrDie("prompt").is_default &&
       !gflags::GetCommandLineFlagInfoOrDie("tokenized_prompt").is_default) {
     ET_CHECK_MSG(false, "Only provide prompt or tokenized_input but not both.");
   }
   if (!gflags::GetCommandLineFlagInfoOrDie("dump_logits_path").is_default &&
       FLAGS_eval_mode != 0) {
     ET_CHECK_MSG(
         false, "Only TokenGenerator(kv) mode is supported to dump all logits.");
   }
   auto etdump = std::make_unique<executorch::etdump::ETDumpGen>(); 
   auto* etdump_ptr = etdump.get();
   std::unique_ptr<executorch::extension::Module> module =
       std::make_unique<executorch::extension::Module>(
           FLAGS_model_path.c_str(),
           executorch::extension::Module::LoadMode::MmapUseMlockIgnoreErrors,
           std::move(etdump));
   // Using 8bit as default since this meta is introduced with 16bit kv io
   // support and older models only have 8bit kv io.
   example::KvBitWidth kv_bitwidth = example::KvBitWidth::kWidth8;
   if (module->method_names()->count("get_kv_io_bit_width") > 0) {
     kv_bitwidth = static_cast<example::KvBitWidth>(
         module->get("get_kv_io_bit_width").get().toScalar().to<int64_t>());
   }
 
   if (kv_bitwidth == example::KvBitWidth::kWidth8) {
     start_runner<uint8_t>(std::move(module), prompts, etdump_ptr);
   } else if (kv_bitwidth == example::KvBitWidth::kWidth16) {
     start_runner<uint16_t>(std::move(module), prompts, etdump_ptr);
   } else {
     ET_CHECK_MSG(
         false,
         "Unsupported kv bitwidth: %ld",
         static_cast<int64_t>(kv_bitwidth));
   }
 
   return 0;
 }

While analyzing the ETDump output, I noticed a large amount of copy and to_copy time.
This raised a few questions regarding where these operations are executed and whether certain ops are unexpectedly falling back to the host.

1. Are the copy / to_copy operations executed on the device or on the host?

The ETDump output does not clearly indicate whether these operations run on the QNN device or on the host side, so it's hard to determine if the overhead comes from QNN runtime or host-side memory movement.

2. Conv op seems to fall back to host execution

Based on the timing breakdown, it appears that some conv operators are not executed on the device backend and instead fall back to host execution.
This is unexpected because convolutions are typically offloaded to the QNN backend.

My current assumption (may be incorrect)

1. Some of the copy operations might correspond to host↔device data transfers.

2. Certain conv ops might not satisfy QNN backend requirements and therefore fall back to host execution.

However, I couldn’t find documentation explaining these behaviors in detail.

Below is a small excerpt of the ETDump trace in table format.
The full trace is available here:

https://github.com/Ulthunzy/qnn-trace-data/blob/master/etdump.xlsx

This bar chart was generated from the ETDump trace data during my analysis.

[shewu-quic]

Hi @Ulthunzy ,

Thanks for your detailed profiling. Let me try to reply your questions first.

1. Are the copy / to_copy operations executed on the device or on the host?

To clarify your terms: executing on the device refers to running on the HTP, while executing on the host refers to running on the CPU, correct?
In the etdump results, you can check the is_delegated_op column to determine whether the operation runs on the HTP or CPU. If is_delegated_op is True, the operation runs on the HTP.

Regarding to_copy operation, it is used for requantization. Since we have applied mix-precision in the graph, this leads to a transformation from x->node (u16) -> node(u8) ->.... As a result, we insert the to_copy operation to cast or convert the data type.
As for the other copy operation, I'm not sure which one you are referring to. Could you please clarify your observation about the copy operation?

2. Conv op seems to fall back to host execution

Based on your results, none of the convolution operations will fall back to the CPU—they should all run on HTP. To confirm, you can review your export PTE stage in AOT and ensure that all convolution operations show [QNN Partitioner Op Support]: aten.convolution.default | True.

For more information on profiling details, please refer to this image or QNN doc

To clarify your results, I have some questions:

1. Can you let me know which decoder model you use and what commands you work with?
2. What device are you using?
3. We have applied TagQuantIO pass on LLM model to prevent Q/DQ from appearing in the graph. As a result, Q/DQ should not be present in the profiling. If they do appear, it may indicate an incorrect profiling record that could be misleading or something wrong.

[Ulthunzy]

Hi @shewu-quic
Thanks a lot for the clarification. I will answer your questions first and provide some additional details based on my ETDump results.

1. About the definition of device vs host

Yes, your understanding matches mine:

1. device = HTP
2. host = CPU

And I also confirmed through the is_delegated_op column in the ETDump that this mapping is correct.

2. About the “other copy” operators

Besides to_copy, I also observed several operators whose names include “copy”, such as:

1. aten_slice_copy_tensor_15:OpId_3483 (cycles)
2. aten_permute_copy_default_34:OpId_3900 (cycles)

These operators show 0 cycles (or extremely close to 0) in ETDump.

So my question is:

Are these “slice_copy” / “permute_copy” ops simply lightweight view/reshape-related ops recorded in ETDump, rather than actual memory copy operations?

Originally, I was suspecting whether to_copy serves as a Q/DQ-like step, so I want to confirm my understanding here.

3. Model and commands I used

The decoder model I used is:
Qwen3-0.6B

The command for exporting the model was:

python examples/qualcomm/oss_scripts/llama/llama.py \
  -a ./llama-output/qwen3-0_6b-m1024-b128-kv-w8a16-256 \
  -b build-android \
  -s 431e3c37 \
  -m SM8750 \
  --temperature 0.3 \
  --model_mode kv \
  --max_seq_len 1024 \
  --prefill_ar_len 128 \
  --decoder_model qwen3-0_6b \
  --prompt "I would like to learn python, could you teach me with a simple example?" \
  --compile_only

The command used for running inference was:

python examples/qualcomm/oss_scripts/llama/llama.py \
  -b build-android \
  -s 431e3c37 \
  -m SM8750 \
  --decoder_model qwen3-0_6b \
  --model_mode kv \
  --prefill_ar_len 128 \
  --max_seq_len 4096 \
  --tasks wikitext \
  --limit 1 \
  --prompt "what is 1+1?" \
  --pre_gen_pte ./llama-output/qwen3-0_6b-m1024-b128-kv-w8a16-256

4. About Q/DQ nodes

In my ETDump results, I did not see explicit Quantize / Dequantize or Q/DQ ops in the event_name column.
So this seems consistent with your description that TagQuantIO should remove these nodes.

Additionally, I observed something even more confusing in the ETDump results:

Some non-trivial compute ops — for example:
aten_sigmoid_default_3:OpId_9157 (cycles)
also show avg(cycles) = 0, and all percentile values (p10/p50/p90) are also 0.
Since sigmoid is a real compute operator and cannot be a no-op or a lightweight view operation, I would expect it to have a measurable execution cost (either on CPU or HTP).
However, ETDump still reports 0 cycles, similar to view/slice/permute ops.

So I would like to understand:

1. Why does ETDump report 0 cycles even for compute-heavy ops like sigmoid?
2. Does this indicate that the op was fused into another operator and not executed independently?
3. Or is the profiler sampling interval too coarse to catch short CPU/HTP ops?
4. Or is ETDump only recording delegate-level blocks and not per-op timing for some operators?

Is this expected behavior for small ops in LLM workloads?

Understanding why even aten_sigmoid_default shows 0 cycles would help me better interpret the profiling results and determine whether I'm missing some timing information.

[shewu-quic]

Besides to_copy, I also observed several operators whose names include “copy”, such as:
aten_slice_copy_tensor_15:OpId_3483 (cycles)
aten_permute_copy_default_34:OpId_3900 (cycles)
Are these “slice_copy” / “permute_copy” ops simply lightweight view/reshape-related ops recorded in ETDump, rather than
actual memory copy operations?

aten_permute_copy_default_34 might be generated from transpose or permute and aten_slice_copy_tensor_15 might be source from tensor slicing. All of them run on HTP.

Why does ETDump report 0 cycles even for compute-heavy ops like sigmoid?
Does this indicate that the op was fused into another operator and not executed independently?
Or is the profiler sampling interval too coarse to catch short CPU/HTP ops?
Or is ETDump only recording delegate-level blocks and not per-op timing for some operators?

In detailed profiling, if the operation cycle is zero, it could indicate either fusion in QNN HTP or parallel execution of operations.

I am glad to help you to clarify profiling results :)
If possible, try using QHAS to profile the LLM for more detailed insights and better readability.
Please refer to README.md and set optrace = True in llama.py for instructions on generating QHAS.

If you encounter any issues, please feel free to let me know.