[Bug] Why does LoRA training save the full InternVL3-14B model instead of adapter weights? #1229
Description
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
Describe the bug
I followed the script here:
https://github.com/OpenGVLab/InternVL/blob/main/internvl_chat/shell/internvl3.0/2nd_finetune/internvl3_14b_dynamic_res_2nd_finetune_full.sh
and converted it into a LoRA version.
My script:
set -x
GPUS=${GPUS:-1}
BATCH_SIZE=${BATCH_SIZE:-4}
PER_DEVICE_BATCH_SIZE=${PER_DEVICE_BATCH_SIZE:-1}
GRADIENT_ACC=$((BATCH_SIZE / PER_DEVICE_BATCH_SIZE / GPUS))
export PYTHONPATH="${PYTHONPATH}:$(pwd)"
export MASTER_PORT=34229
export TF_CPP_MIN_LOG_LEVEL=3
export LAUNCHER=pytorch
OUTPUT_DIR='work_dirs/internvl_chat_v3/internvl3_14b_dynamic_res_2nd_finetune_lora/test'
if [ ! -d "$OUTPUT_DIR" ]; then
mkdir -p "$OUTPUT_DIR"
fi
# number of gpus: 1
# batch size per gpu: 4
# gradient accumulation steps: 4
# total batch size: 128
# epoch: 1
torchrun \
--nnodes=1 \
--node_rank=0 \
--master_addr=127.0.0.1 \
--nproc_per_node=${GPUS} \
--master_port=${MASTER_PORT} \
internvl/train/internvl_chat_finetune.py \
--model_name_or_path "OpenGVLab/InternVL3-14B" \
--conv_style "internvl2_5" \
--use_fast_tokenizer False \
--output_dir ${OUTPUT_DIR} \
--meta_path "./shell/data/custom_dataset_hht.json" \
--overwrite_output_dir True \
--force_image_size 448 \
--max_dynamic_patch 12 \
--down_sample_ratio 0.5 \
--drop_path_rate 0.1 \
--freeze_llm True \
--freeze_mlp True \
--use_llm_lora 8 \
--freeze_backbone True \
--vision_select_layer -1 \
--dataloader_num_workers 4 \
--bf16 True \
--num_train_epochs 1 \
--per_device_train_batch_size ${PER_DEVICE_BATCH_SIZE} \
--gradient_accumulation_steps ${GRADIENT_ACC} \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 50 \
--save_total_limit 1 \
--learning_rate 2e-5 \
--weight_decay 0.05 \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--max_seq_length 16384 \
--do_train True \
--grad_checkpoint True \
--group_by_length True \
--dynamic_image_size True \
--use_thumbnail True \
--ps_version 'v2' \
--deepspeed "zero_stage1_config.json" \
--report_to "tensorboard" \
2>&1 | tee -a "${OUTPUT_DIR}/training_log.txt"
However, the final output is still the full model checkpoint. I don’t see any LoRA adapter files such as adapter_model.safetensors or adapter_config.json.
I also checked this LoRA script:
https://github.com/OpenGVLab/InternVL/blob/main/internvl_chat/shell/internvl2.5/2nd_finetune/internvl2_5_2b_dynamic_res_2nd_finetune_lora_coco.sh
My implementation is very similar, and both scripts eventually run the same training entry:
internvl/train/internvl_chat_finetune.py.
So I’m wondering why my training run produces a full model instead of LoRA-only adapter weights. Is there something I might be missing in the configuration or script?
Thanks a lot for your help and for maintaining this project!
Reproduction
set -x
GPUS=${GPUS:-1}
BATCH_SIZE=${BATCH_SIZE:-4}
PER_DEVICE_BATCH_SIZE=${PER_DEVICE_BATCH_SIZE:-1}
GRADIENT_ACC=$((BATCH_SIZE / PER_DEVICE_BATCH_SIZE / GPUS))
export PYTHONPATH="${PYTHONPATH}:$(pwd)"
export MASTER_PORT=34229
export TF_CPP_MIN_LOG_LEVEL=3
export LAUNCHER=pytorch
OUTPUT_DIR='work_dirs/internvl_chat_v3/internvl3_14b_dynamic_res_2nd_finetune_lora/test'
if [ ! -d "$OUTPUT_DIR" ]; then
mkdir -p "$OUTPUT_DIR"
fi
# number of gpus: 1
# batch size per gpu: 4
# gradient accumulation steps: 4
# total batch size: 128
# epoch: 1
torchrun \
--nnodes=1 \
--node_rank=0 \
--master_addr=127.0.0.1 \
--nproc_per_node=${GPUS} \
--master_port=${MASTER_PORT} \
internvl/train/internvl_chat_finetune.py \
--model_name_or_path "OpenGVLab/InternVL3-14B" \
--conv_style "internvl2_5" \
--use_fast_tokenizer False \
--output_dir ${OUTPUT_DIR} \
--meta_path "./shell/data/custom_dataset_hht.json" \
--overwrite_output_dir True \
--force_image_size 448 \
--max_dynamic_patch 12 \
--down_sample_ratio 0.5 \
--drop_path_rate 0.1 \
--freeze_llm True \
--freeze_mlp True \
--use_llm_lora 8 \
--freeze_backbone True \
--vision_select_layer -1 \
--dataloader_num_workers 4 \
--bf16 True \
--num_train_epochs 1 \
--per_device_train_batch_size ${PER_DEVICE_BATCH_SIZE} \
--gradient_accumulation_steps ${GRADIENT_ACC} \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 50 \
--save_total_limit 1 \
--learning_rate 2e-5 \
--weight_decay 0.05 \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--max_seq_length 16384 \
--do_train True \
--grad_checkpoint True \
--group_by_length True \
--dynamic_image_size True \
--use_thumbnail True \
--ps_version 'v2' \
--deepspeed "zero_stage1_config.json" \
--report_to "tensorboard" \
2>&1 | tee -a "${OUTPUT_DIR}/training_log.txt"
Environment
1.
lmdeploy check_env:
sys.platform: linux
Python: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA H100 80GB HBM3
CUDA_HOME: /usr/local/cuda-12.4
NVCC: Cuda compilation tools, release 12.4, V12.4.99
GCC: x86_64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
PyTorch: 2.8.0+cu128
PyTorch compiling details: PyTorch built with:
- GCC 13.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 12.8
- NVCC architecture flags: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_100,code=sm_100;-gencode;arch=compute_120,code=sm_120
- CuDNN 91.0.2 (built against CUDA 12.9)
- Built with CuDNN 90.8
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=a1cb3cc05d46d198467bebbb6e8fba50a325d4e7, CUDA_VERSION=12.8, CUDNN_VERSION=9.8.0, CXX_COMPILER=/opt/rh/gcc-toolset-13/root/usr/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,
TorchVision: 0.23.0+cu128
LMDeploy: 0.10.2+
transformers: 4.47.0
fastapi: 0.121.0
pydantic: 2.11.10
triton: 3.4.0
NVIDIA Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-25 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
2.
Package Version
----------------------------- -------------
absl-py 2.3.1
accelerate 1.11.0
addict 2.4.0
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.13.2
aiosignal 1.4.0
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-doc 0.0.3
annotated-types 0.7.0
anyio 4.11.0
attrdict 2.0.1
attrs 25.4.0
binpacking 1.5.2
Brotli 1.1.0
certifi 2025.10.5
cffi 2.0.0
charset-normalizer 3.4.4
click 8.2.1
contourpy 1.3.3
cpm-kernels 1.0.11
crcmod 1.7
cryptography 46.0.3
cycler 0.12.1
dacite 1.9.2
datasets 3.6.0
decord 0.6.0
deepspeed 0.18.1
dill 0.3.8
distro 1.9.0
einops 0.8.1
fastapi 0.121.0
ffmpy 0.6.4
filelock 3.20.0
fire 0.7.1
flash_attn 2.8.0.post2
fonttools 4.60.1
frozenlist 1.8.0
fsspec 2025.3.0
future 1.0.0
gradio 5.49.1
gradio_client 1.13.3
groovy 0.1.2
grpcio 1.76.0
h11 0.16.0
hf-xet 1.2.0
hjson 3.1.0
httpcore 1.0.9
httpx 0.28.1
huggingface-hub 0.36.0
idna 3.11
ImageIO 2.37.2
importlib_metadata 8.7.0
jieba 0.42.1
Jinja2 3.1.6
jiter 0.11.1
jmespath 0.10.0
joblib 1.5.2
json_repair 0.52.4
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
kiwisolver 1.4.9
lmdeploy 0.10.2
Markdown 3.10
markdown-it-py 4.0.0
MarkupSafe 3.0.3
matplotlib 3.10.7
mdurl 0.1.2
mmengine-lite 0.10.7
modelscope 1.31.0
mpmath 1.3.0
ms_swift 3.9.3
msgpack 1.1.2
multidict 6.7.0
multiprocess 0.70.16
networkx 3.5
ninja 1.13.0
nltk 3.9.2
numpy 2.2.6
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-ml-py 13.580.82
nvidia-nccl-cu12 2.27.3
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvshmem-cu12 3.3.20
nvidia-nvtx-cu12 12.8.90
openai 2.7.1
openai-harmony 0.0.4
opencv-python 4.12.0.88
orjson 3.11.4
oss2 2.19.1
packaging 25.0
pandas 2.3.3
partial-json-parser 0.2.1.1.post6
peft 0.14.0
pillow 11.3.0
pip 24.0
platformdirs 4.5.0
prometheus_client 0.23.1
propcache 0.4.1
protobuf 6.33.0
psutil 7.1.3
py-cpuinfo 9.0.0
pyarrow 22.0.0
pycparser 2.23
pycryptodome 3.23.0
pydantic 2.11.10
pydantic_core 2.33.2
pydub 0.25.1
Pygments 2.19.2
pynvml 13.0.1
pyparsing 3.2.5
python-dateutil 2.9.0.post0
python-multipart 0.0.20
pytz 2025.2
PyYAML 6.0.3
pyzmq 27.1.0
ray 2.51.1
referencing 0.37.0
regex 2025.11.3
requests 2.32.5
rich 14.2.0
rouge 1.0.1
rpds-py 0.28.0
ruff 0.14.3
safehttpx 0.1.7
safetensors 0.6.2
scipy 1.16.3
semantic-version 2.10.0
sentencepiece 0.2.1
setuptools 80.9.0
shellingham 1.5.4
shortuuid 1.0.13
simplejson 3.20.2
six 1.17.0
sniffio 1.3.1
sortedcontainers 2.4.0
starlette 0.49.3
sympy 1.14.0
tensorboard 2.20.0
tensorboard-data-server 0.7.2
termcolor 3.2.0
tiktoken 0.12.0
timm 1.0.22
tokenizers 0.21.4
tomlkit 0.13.3
torch 2.8.0
torchvision 0.23.0
tqdm 4.67.1
transformers 4.47.0
transformers-stream-generator 0.0.5
triton 3.4.0
trl 0.23.1
typer 0.20.0
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
urllib3 2.5.0
uvicorn 0.38.0
websockets 15.0.1
Werkzeug 3.1.3
xgrammar 0.1.27
xxhash 3.6.0
yapf 0.43.0
yarl 1.22.0
zipp 3.23.0
zstandard 0.25.0