MultiDeviceModel problem

[Chebart]

environment:

pcie environment
transformers: 4.51.0
torch: 2.6.0
LLM-TPU: f5502b5 2025.03.29
tpu-mlir: 5eba2a0e2 2025.03.20
driver version: release version:0.5.2 release date: 20250411-003300

path:

/workspace/LLM-TPU/template/parallel_demo

operate:

python3 pipeline.py --devid 4,5,6,7 --dir_path /workspace/LLM-TPU/inference_bmodels/Llama-3.2-3B-Instruct-w4-4dev

question:

Hello, I'm using the SC7-224T card with sophosdk version 24.04.01 and libsophon version 0.5.2, and I've encountered an issue with the chat.cpp module located in the template/parallel_demo directory. Previously, models configured to run on 1, 2, 4, and 8 chips all functioned as expected. However, now unexpectedly only the 1-chip and 2-chip models are working. The 4-chip and 8-chip configurations, which used to run without any issues, are now failing to load. You can download the full debug trace using this link. Do you have any idea what might be causing this issue or how I can fix it?

[sgwzy22]

"The 4-chip and 8-chip configurations, which used to run without any issues, are now failing to load."
Are there any changes in the code or environment during this period ？
If you are unsure，try these :

1. Ensure the chips status are active before operation, you can use bm-smi to view all chip status;
2. Run in another terminal and monitor their status, make sure that no other models are running and chip memory is enough;
3. If still failed, try ulimit -HSn 65536

[Chebart]

To eliminate issues related to code or environment changes, I created a new directory, cloned the necessary repositories, changed the repositories to the versions described above, set up a new docker container, built the required modules, and attempted to use models configured for 4 chips. Unfortunately, I encountered the same result. Here's a detailed explanation of what happened:

1. I verified that all chips showed an active status in bm-smi.

2. I attempted to run the QwQ-32B and Llama3.2-3B-Instruct models, both compiled for a 4-chip configuration like this:

python3 pipeline.py --devid 4,5,6,7 --dir_path /workspace/LLM-TPU/inference_bmodels/Llama-3.2-3B-Instruct-w4-4dev
python3 pipeline.py --devid 4,5,6,7 --dir_path /workspace/LLM-TPU/inference_bmodels/QwQ-32B-w4

3. Upon sending the first request, all chips immediately reached 100% utilization, then froze, and eventually entered a fault state.

4. ulimit -HSn 65536 does not work too

[sgwzy22]

The issue seems to be inconsistent with the log provided earlier，The previous log showed an error during loading, but this issue occurred during runtime and could result chip hanging. I will verify this demo later.

QWQ-32B is the same architecture as Qwen2.5，you can try Qwen2.5 multi-dev demo instead.

As for Llama3.2-3B-Instruct，we do not recommend using multi-dev to run such lightweight model，The communication time between chips will be much longer than the inference time of a single chip，you can try the single chip demo instead.