[Bug] InternVL3_5-38B: ValueError: Following weights were not initialized from checkpoint: #1232
Description
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
Describe the bug
vllm 0.11.2
vllm serve
${model_path}
--served-model-name InternVL3_5-38B
--port 40001
--host 127.0.0.1
--dtype bfloat16
--trust-remote-code
--tensor-parallel-size 2
Error:
raise ValueError(
(Worker_TP0 pid=85304) ERROR 11-25 17:57:23 [multiproc_executor.py:743] ValueError: Following weights were not initialized from checkpoint: {'language_model.model.layers.2.mlp.down_proj.weight', 'language_model.model.layers.17.post_attention_layernorm.weight', 'language_model.model.layers.5.self_attn.o_proj.weight', 'language_model.model.layers.22.self_attn.k_norm.weight', 'language_model.model.layers.22.self_attn.qkv_proj.weight', 'language_model.model.layers.20.mlp.gate_up_proj.weight', 'language_model.model.layers.18.mlp.down_proj.weight', 'language_model.model.layers.3.input_layernorm.weight', 'language_model.model.layers.3.mlp.down_proj.weight', 'language_model.model.layers.21.self_attn.qkv_proj.weight', 'language_model.model.layers.21.self_attn.o_proj.weight', 'language_model.model.layers.6.input_layernorm.weight', 'language_model.model.layers.5.mlp.down_proj.weight', 'language_model.model.layers.18.input_layernorm.weight', 'language_model.model.layers.22.self_attn.q_norm.weight', 'language_model.model.layers.19.self_attn.qkv_proj.weight', 'language_model.model.layers.20.post_attention_layernorm.weight', 'language_model.model.layers.7.self_attn.qkv_proj.weight', 'language_model.model.layers.4.input_layernorm.weight', 'language_model.model.layers.18.self_attn.qkv_proj.weight', 'language_model.model.layers.5.self_attn.qkv_proj.weight', 'language_model.model.layers.20.mlp.down_proj.weight', 'language_model.model.layers.3.post_attention_layernorm.weight', 'language_model.model.layers.6.mlp.down_proj.weight', 'language_model.model.layers.20.input_layernorm.weight', 'language_model.model.layers.18.self_attn.k_norm.weight', 'language_model.model.layers.5.input_layernorm.weight', 'language_model.model.layers.19.self_attn.q_norm.weight', 'language_model.model.layers.21.input_layernorm.we
...
Reproduction
vllm 0.11.2
vllm serve
${model_path}
--served-model-name InternVL3_5-38B
--port 40001
--host 127.0.0.1
--dtype bfloat16
--trust-remote-code
--tensor-parallel-size 2
Environment
==============================
PyTorch Info
==============================
PyTorch version : 2.9.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.10.19 | packaged by conda-forge | (main, Oct 22 2025, 22:29:10) [GCC 14.3.0] (64-bit runtime)
Python platform : Linux-5.10.134-010.ali5000.al8.x86_64-x86_64-with-glibc2.32
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.8.61
CUDA_MODULE_LOADING set to :
GPU models and configuration :
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
Nvidia driver version : 550.144.04
cuDNN version : Probably one of the following:
/usr/lib64/libcudnn.so.9.10.2
/usr/lib64/libcudnn_adv.so.9.10.2
/usr/lib64/libcudnn_cnn.so.9.10.2
/usr/lib64/libcudnn_engines_precompiled.so.9.10.2
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.10.2
/usr/lib64/libcudnn_graph.so.9.10.2
/usr/lib64/libcudnn_heuristic.so.9.10.2
/usr/lib64/libcudnn_ops.so.9.10.2
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 58-77,154-173
Off-line CPU(s) list: 0-57,78-153,174-191
Thread(s) per core: 0
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 143
Model name: Intel(R) Xeon(R) Platinum 8469C
Stepping: 8
CPU MHz: 3099.919
CPU max MHz: 3800.0000
CPU min MHz: 800.0000
BogoMIPS: 5185.53
Virtualization: VT-x
L1d cache: 48K
L1i cache: 32K
L2 cache: 2048K
L3 cache: 99840K
NUMA node0 CPU(s): 0-47,96-143
NUMA node1 CPU(s): 48-95,144-191
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm uintr md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.5.2
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.16.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.3.0
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.0
[pip3] torchaudio==2.9.0
[pip3] torchvision==0.24.0
[pip3] transformers==4.57.1
[pip3] triton==3.5.0
[conda] flashinfer-python 0.5.2 pypi_0 pypi
[conda] numpy 2.2.6 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi
[conda] nvidia-cudnn-frontend 1.16.0 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi
[conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi
[conda] nvidia-cutlass-dsl 4.3.0 pypi_0 pypi
[conda] nvidia-ml-py 13.580.82 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.27.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-nvshmem-cu12 3.3.20 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi
[conda] pyzmq 27.1.0 pypi_0 pypi
[conda] torch 2.9.0 pypi_0 pypi
[conda] torchaudio 2.9.0 pypi_0 pypi
[conda] torchvision 0.24.0 pypi_0 pypi
[conda] transformers 4.57.1 pypi_0 pypi
[conda] triton 3.5.0 pypi_0 pypi
==============================
vLLM Info
==============================
ROCM Version : Could not collect
vLLM Version : 0.11.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 GPU1 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 NIC14 NIC15 NIC16 NIC17 NIC18 NIC19 NIC20 NIC21 NIC22 NIC23 NIC24 NIC25 NIC26 NIC27 NIC28 NIC29 NIC30 NIC31 NIC32 NIC33 NIC34 NIC35 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE N/A
GPU1 NV18 X PIX PIX PIX PIX PIX PIX PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE N/A
NIC0 NODE PIX X PIX PIX PIX PIX PIX PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC1 NODE PIX PIX X PIX PIX PIX PIX PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC2 NODE PIX PIX PIX X PIX PIX PIX PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC3 NODE PIX PIX PIX PIX X PIX PIX PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC4 NODE PIX PIX PIX PIX PIX X PIX PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC5 NODE PIX PIX PIX PIX PIX PIX X PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC6 NODE PIX PIX PIX PIX PIX PIX PIX X PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC7 NODE PIX PIX PIX PIX PIX PIX PIX PIX X NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE
NIC8 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE X PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC9 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX X PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC10 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX PIX X PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC11 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX PIX PIX X PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC12 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX PIX PIX PIX X PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC13 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX PIX PIX PIX PIX X PIX PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC14 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX X PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC15 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX X SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX
NIC16 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX PIX PIX PIX PIX PIX PIXNODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC17 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X PIX PIX PIX PIX PIX PIXNODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX X PIX PIX PIX PIX PIXNODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC19 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX X PIX PIX PIX PIXNODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC20 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX X PIX PIX PIXNODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC21 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX X PIX PIXNODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC22 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX X PIXNODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC23 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX PIX X NODE NODE NODE NODE NODE NODE NODE NODE PIX NODE SYS SYS
NIC24 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE X PIX PIX PIX PIX PIX PIX PIX NODE PIX SYS SYS
NIC25 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX X PIX PIX PIX PIX PIX PIX NODE PIX SYS SYS
NIC26 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX PIX X PIX PIX PIX PIX PIX NODE PIX SYS SYS
NIC27 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX PIX PIX X PIX PIX PIX PIX NODE PIX SYS SYS
NIC28 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX PIX PIX PIX X PIX PIX PIX NODE PIX SYS SYS
NIC29 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX PIX PIX PIX PIX X PIX PIX NODE PIX SYS SYS
NIC30 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX PIX PIX PIX PIX PIX X PIX NODE PIX SYS SYS
NIC31 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX PIX PIX PIX PIX PIX PIX X NODE PIX SYS SYS
NIC32 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX PIX PIXNODE NODE NODE NODE NODE NODE NODE NODE X NODE SYS SYS
NIC33 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODEPIX PIX PIX PIX PIX PIX PIX PIX NODE X SYS SYS
NIC34 NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X NODE
NIC35 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
NIC10: mlx5_10
NIC11: mlx5_11
NIC12: mlx5_12
NIC13: mlx5_13
NIC14: mlx5_14
NIC15: mlx5_15
NIC16: mlx5_16
NIC17: mlx5_17
NIC18: mlx5_18
NIC19: mlx5_19
NIC20: mlx5_20
NIC21: mlx5_21
NIC22: mlx5_22
NIC23: mlx5_23
NIC24: mlx5_24
NIC25: mlx5_25
NIC26: mlx5_26
NIC27: mlx5_27
NIC28: mlx5_28
NIC29: mlx5_29
NIC30: mlx5_30
NIC31: mlx5_31
NIC32: mlx5_bond_0
NIC33: mlx5_bond_1
NIC34: mlx5_bond_2
NIC35: mlx5_bond_3
==============================
Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/gcc-5.3.0/lib64:/usr/local/lib:/usr/local/lib64:/usr/local/lib64/boost:/opt/taobao/java/jre/lib/amd64/server:/hadoop_java/java/jdk/jre/lib/amd64/server:/worker/:/worker/lib:/apsara_lib64/:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/:/lib:/opt/rh/gcc-toolset-13/root/usr/lib64:/opt/rh/gcc-toolset-13/root/usr/lib:/usr/local/lib64:/lib64:/usr/local/gcc75/lib:/usr/local/gcc75/lib64::/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/lib64:/pu:/opt/taobao/java/jre/lib/amd64/server:/apsara/alicpp/built/gcc-4.9.2/glog-0.3.4/lib:/apsara/alicpp/built/gcc-4.9.2/gflags-2.1.2/lib:/apsara/alicpp/built/gcc-4.9.2/protobuf-2.4.1.ali/lib:/apsara/alicpp/built/gcc-4.9.2/odps-cryptography-1.0.0/lib:/apsara/alicpp/built/gcc-4.9.2/boost-1.58.0.fix.thread/lib:/apsara/alicpp/built/gcc-4.9.2/openssl-1.0.2a/lib:/apsara/alicpp/built/gcc-4.9.2/mysql-connector-c-6.1.6/lib:/apsara/alicpp/built/gcc-4.9.2/arrow-0.16.0/lib64:/apsara/alicpp/built/gcc-4.9.2/bzip2-1.0.6/lib64:/apsara/alicpp/built/gcc-4.9.2/zstd-1.4.4/lib:/apsara/alicpp/built/gcc-4.9.2/libevent-2.0.22.stable/lib64:/worker:/worker/lib:/opt/conda/envs/python3.10.13/lib
NCCL_IB_GID_INDEX=3
TORCH_EXTENSIONS_DIR=:/usr/local/ninja
NVIDIA_VISIBLE_DEVICES=0,1
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
NVIDIA_VOL_MNT_PATH=/usr/local/nvidia/
NVIDIA_DRIVER_CAPABILITIES=all
NCCL_NET_PLUGIN=none
NCCL_DEBUG=INFO
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1Error traceback
raise ValueError(
(Worker_TP0 pid=85304) ERROR 11-25 17:57:23 [multiproc_executor.py:743] ValueError: Following weights were not initialized from checkpoint: {'language_model.model.layers.2.mlp.down_proj.weight', 'language_model.model.layers.17.post_attention_layernorm.weight', 'language_model.model.layers.5.self_attn.o_proj.weight', 'language_model.model.layers.22.self_attn.k_norm.weight', 'language_model.model.layers.22.self_attn.qkv_proj.weight', 'language_model.model.layers.20.mlp.gate_up_proj.weight', 'language_model.model.layers.18.mlp.down_proj.weight', 'language_model.model.layers.3.input_layernorm.weight', 'language_model.model.layers.3.mlp.down_proj.weight', 'language_model.model.layers.21.self_attn.qkv_proj.weight', 'language_model.model.layers.21.self_attn.o_proj.weight', 'language_model.model.layers.6.input_layernorm.weight', 'language_model.model.layers.5.mlp.down_proj.weight', 'language_model.model.layers.18.input_layernorm.weight', 'language_model.model.layers.22.self_attn.q_norm.weight', 'language_model.model.layers.19.self_attn.qkv_proj.weight', 'language_model.model.layers.20.post_attention_layernorm.weight', 'language_model.model.layers.7.self_attn.qkv_proj.weight', 'language_model.model.layers.4.input_layernorm.weight', 'language_model.model.layers.18.self_attn.qkv_proj.weight', 'language_model.model.layers.5.self_attn.qkv_proj.weight', 'language_model.model.layers.20.mlp.down_proj.weight', 'language_model.model.layers.3.post_attention_layernorm.weight', 'language_model.model.layers.6.mlp.down_proj.weight', 'language_model.model.layers.20.input_layernorm.weight', 'language_model.model.layers.18.self_attn.k_norm.weight', 'language_model.model.layers.5.input_layernorm.weight', 'language_model.model.layers.19.self_attn.q_norm.weight', 'language_model.model.layers.21.input_layernorm.we
...