Reminder

• I have read the above rules and searched the existing issues.

Description

I propose the integration and support within LlamaFactory for two recently introduced, critical features from the PEFT (Parameter-Efficient Fine-Tuning) library, which significantly extend the capabilities and robustness of LoRA fine-tuning, particularly for complex model architectures.

The features in question, based on recent PEFT updates (e.g., PRs #2638 and #2665), are:

Direct LoRA Targeting of nn.Parameter via target_parameters:

Description: LoRA is now capable of targeting any arbitrary nn.Parameter object directly, supplementing the existing target_modules functionality for nn.Module instances. This is configured using the target_parameters list within LoraConfig.

Justification: This functionality is essential for fine-tuning models that utilize custom layers or store weights as standalone parameters, most notably in Mixture of Expert (MoE) models. MoE layers frequently manage expert weights as nn.Parameter objects, which cannot be reliably targeted using target_modules.

Example Use Case (MoE):

Python

config = LoraConfig(
...,
target_modules=[], # To explicitly exclude default module targeting
target_parameters=["feed_forward.experts.down_proj", "feed_forward.experts.gate_up_proj"], # Targets specific expert parameters
)
Adapter Injection Based on state_dict Content:

Description: The PEFT library now supports injecting adapters into a base model by inspecting the keys within the loaded adapter state_dict via the inject_adapter_in_model function.

Justification: This enhancement improves the resilience of adapter loading. When the original PEFT configuration (specifically the target_modules argument) is unavailable or incomplete, the framework can infer the necessary injection points directly from the checkpoint's state_dict. This eliminates the cumbersome and error-prone process of reverse-engineering the LoraConfig.

Example Use Case:

Python

from peft import LoraConfig, inject_adapter_in_model

... load state_dict ...

lora_config = LoraConfig() # Minimal configuration is sufficient
model = inject_adapter_in_model(lora_config, model, state_dict=state_dict)
Proposed Action: We request that LlamaFactory be updated to allow configuration of the target_parameters list through its command-line arguments and configuration files for training. Furthermore, the adapter loading and merging mechanisms should be updated to leverage the state_dict-based injection method for improved compatibility and robustness when handling adapter checkpoints.

💻 System Info
Please provide your system information by running the command llamafactory-cli env and pasting the output below.

[Placeholder for llamafactory-cli env output]

Please insert the actual output here.

📝 Reproduction
Since this is a feature request for new functionality, no error messages or stack traces are available. The following details outline the anticipated usage flow upon implementation:

1. target_parameters Integration (Training):

Entry Arguments: A new, dedicated command-line argument (e.g., –lora_target_parameters) should be introduced to accept a comma-separated list of nn.Parameter names.

Expected Usage (CLI Example):

Bash

python src/train_bash.py
--model_name_or_path path/to/moe_model
--do_train
--lora_target_parameters "feed_forward.experts.down_proj,feed_forward.experts.gate_up_proj"
# ... other training arguments ...
2. state_dict-Based Injection (Loading/Merging):

Expected Behavior: When LlamaFactory loads an adapter checkpoint (for inference, merging, or prediction), if the corresponding PEFT configuration file is missing or incomplete, the adapter loading logic should attempt to utilize the state_dict-based injection logic provided by the updated PEFT library to correctly apply the weights.

Pull Request

No response