Question about multiple LoRAs and FSDP

[l1346792580123]

Hello, I want to train multiple LoRA simultaneously based on a single base model. The base model is too big so I need to use FSDP to save GPU memory. My current implementation as follows

modelA = basemodel
modelB = inject_adapter_in_model(lora_config, basemodel)
modelC = inject_adapter_in_model(lora_config, basemodel)

modelA = FSDP(modelA)
modelB = FSDP(modelB)
modelC = FSDP(modelC)

training_loop

As you can see, the base model is repeated three times. Is there a way to use only one base model and two LoRA adapters to achieve the same functionality ?

[githubnemo]

I'm not sure if I understand the purpose of this process. Why train two adapters at the same time one the same data?

Also in the pseudo-code inject_adapter_in_model modifies the base model, so modelC == modelB == modelA - is that intended?

In any way, to answer your question, yes it is possible to load multiple adapters (of the same type) via set_adapter:

peft_config_1 = LoraConfig(...)
peft_model = get_peft_model(base_model, peft_config_1)

peft_config_2 = LoraConfig(...)
peft_model.add_adapter("adapter2", peft_config_2)

peft_model.set_adapter(["default", "adapter2"])

[... training ...]

[l1346792580123]

Thanks for your reply! Actually I want to implement the distribution matching distillation (DMD) algorithm, which involves three model: a real score model, a fake score model and a generator. The real score model is frozen, and the fake score and generator are updated during the training loop. I am sorry the pseudo-code above caused a misunderstanding. Actually I initialize the base model three times and inject lora to modelB and modelC.

The solution you provided works when the model is small. While I'd like to know how to combine it with Fully Sharded Data Parallel(FSDP). FSDP share parameters in all gpus and gather the full parameters during forward or backward computations. I' like to know how to load/unload LoRa adapter on the FSDP-wrapped basemodel.