[Usage] Mismatch between mm_projector in saved config and mm_projector used in code. #1884
Description
Describe the issue
Issue:
I am fine-tuning the llava-v1.5 model in a custom setting, using LoRA. The script loads the default value for model_args.mm_projector_type = "linear" (loaded from ModelArguments). However, the model loaded model = LlavaLlamaForCausalLM.from_pretrained reads mm_projector from config.json stored in model_path. This incorrect value is saved in config.json after the model is fully trained.
This not only causes issues during inference of the fine-tuned models but also during editing the model for a custom setting.
Command:
#!/bin/bash
# IMPORTANT: this is the training script for the original LLaVA, NOT FOR LLaVA V1.5!
# Uncomment and set the following variables correspondingly to run this script:
################## VICUNA ##################
PROMPT_VERSION=v1
MODEL_VERSION="models--liuhaotian--llava-v1.5-13b"
ADAPTER_VERSION="models--liuhaotian--llava-v1.5-mlp2x-336px-pretrain-vicuna-13b-v1.5"
################## VICUNA ##################
################## LLaMA-2 ##################
# PROMPT_VERSION="llava_llama_2"
# MODEL_VERSION="llama-2-7b-chat"
################## LLaMA-2 ##################
deepspeed llava/train/train_mem.py \
--deepspeed ./scripts/zero2.json \
--lora_enable True \
--bits 16 \
--model_name_or_path ~/.cache/huggingface/hub/$MODEL_VERSION/snapshots/901a44b9113dea67b976e71f58d4e372cf9de81a \
--version $PROMPT_VERSION \
--data_path ./data/SketchLLM_coco_2.json \
--image_folder ~/aditay/coco/train2017 \
--sketch_folder ~/aditay/coco/sketch_train/combined \
--vision_tower openai/clip-vit-large-patch14-336 \
--pretrain_mm_mlp_adapter ~/.cache/huggingface/hub/$ADAPTER_VERSION/snapshots/e362f049527e677c163c89a4449c780256e4beb8/mm_projector.bin \
--mm_vision_select_layer -2 \
--mm_use_im_start_end False \
--mm_use_im_patch_token False \
--freeze_mm_mlp_adapter True \
--sep_img_sketch_mlp_adapter True \
--bf16 True \
--output_dir ./checkpoints_coco/llava-$MODEL_VERSION-finetune_lora \
--num_train_epochs 1 \
--per_device_train_batch_size 12 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1000 \
--save_total_limit 1 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--tf32 True \
--model_max_length 2048 \
--gradient_checkpointing True \
--lazy_preprocess True \
--dataloader_num_workers 8 \
--report_to wandb
Screenshots:
