How to load a pre-trained LoRA Adapter for GRPO Training for Multi-Stage RL?

[awinml]

I’m implementing a multi-stage reinforcement learning (RL) pipeline for reasoning tasks using GRPO, and I’d like to:

1. Train a model on Dataset A with LoRA.
2. Save the trained LoRA adapter weights.
3. Resume training on Dataset B by continuing to update the same LoRA adapter, rather than initializing a fresh one.

Setup:

• Base model: Qwen3
• Adapter: LoRA
• Rollout engine: vLLM (multi-turn)
• Training algorithm: GRPO

In standard Hugging Face + PEFT workflows, I can load a pre-trained LoRA adapter like this:

from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM

config = PeftConfig.from_pretrained("path_to_trained_lora_adapter")
base_model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path)
lora_model = PeftModel.from_pretrained(base_model, "path_to_trained_lora_adapter")

However, in my current GRPO trainer config, LoRA is initialized from scratch via these parameters:

actor_rollout_ref.model.lora_rank=32 \
actor_rollout_ref.model.lora_alpha=32 \
actor_rollout_ref.model.target_modules=all-linear \

There doesn’t appear to be a config option (e.g., lora_path or pretrained_adapter_name_or_path) to load an existing adapter instead of initializing new LoRA weights.

Question:
How can I configure the GRPO trainer to load and continue training from a pre-trained LoRA adapter? Is this supported, and if so, what’s the correct way to specify the adapter path in the config or code?

[piood]

Now verl can't load and continue training from a pre-trained LoRA adapter, after the pr (#3523) it can work.

[awinml]

Thanks for adding this functionality!