QLoRA Finetuning Scripts for Gemma, Mistral, and Qwen

This repository contains scripts for finetuning Gemma, Mistral, and Qwen models using the QLoRA (Quantized Low-Rank Adaptation) technique.

Requirements

Install the required packages:

pip install -r requirements.txt

Data Directory Structure

It is recommended to organize your data into the following directory structure:

/data
  /training
    - train_data_1.json
    - train_data_2.json
  /validation
    - valid_data_1.json
  /images
    - image_1.jpg
    - image_2.png

Dataset Format

The scripts expect JSON files for training and validation datasets.

Text-Based Models

For text-only finetuning, the dataset should be in the following format:

[
  [
    {
      "role": "user",
      "content": "What legal framework governs the regulations for Highly Automated Vehicles in Pennsylvania?"
    },
    {
      "role": "assistant",
      "content": "The regulations for Highly Automated Vehicles (HAV) in Pennsylvania are governed by the 75 Pa.C.S. This legal framework provides the necessary guidelines and standards to ensure the safe operation and integration of HAVs within the state's transportation system."
    }
  ]
]

Vision Models

For vision-capable models, the dataset format is slightly different to accommodate images.

• Gemma: Can take vision input, but it is optional.
• Mistral and Qwen: Vision input is required.

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "image_name.jpg"
          }
        },
        {
          "type": "text",
          "text": "What is in this image?"
        }
      ]
    },
    {
      "role": "assistant",
      "content": "This is an image of a cat."
    }
  ]
}

Usage

The scripts use a .env file to configure the dataset paths, output directory, and other settings. Create or edit the .env file with the following content:

# QLoRA Finetuning Environment Variables
TRAIN_DATASET_PATH=./data/training
VALID_DATASET_PATH=./data/validation
OUTPUT_DIR=./model
MODEL="google/gemma-3-12b-it" # or "unsloth/Mistral-Small-3.2-24B-Instruct-2506" or "Qwen/Qwen2.5-VL-7B-Instruct"
IMAGE_BASE_URL=./data/images # Required for vision models

Then run the desired finetuning script:

• For Gemma: python qlora_finetune_gemma.py
• For Mistral: python qlora_finetune_mistral.py
• For Qwen: python qlora_finetune_qwen.py

Finetuning Scripts

Gemma

• Model: google/gemma-3-12b-it
• Type: Text and Vision (optional)
• Description: This script finetunes the Gemma model. It can handle both text-only and text-with-image datasets.
• Command: python qlora_finetune_gemma.py

Mistral

• Model: unsloth/Mistral-Small-3.2-24B-Instruct-2506
• Type: Vision (required)
• Description: This script finetunes the Mistral vision model using the unsloth library for faster and more memory-efficient training. Image input is required.
• Command: python qlora_finetune_mistral.py

Qwen

• Model: Qwen/Qwen2.5-VL-7B-Instruct (or other Qwen 2.5 VL models)
• Type: Vision (required)
• Description: This script finetunes Qwen 2.5 VL vision models. It uses bitsandbytes for quantization and accelerate for distributed training. Image input is required.
• Command: python qlora_finetune_qwen.py

Inference

The inference.py script is currently configured for the Gemma model. It supports loading both the GGUF format and the adapter-based model.

To run inference with the Gemma model:

python inference.py

For the Mistral and Qwen models, you will need to adapt the inference.py script or use a different method for inference, as the current script is not generalized for vision models.