Guide on training that requires both LoRA and base model forward calls ?

[thangld201]

Hi, I'm working on some training variants that require hidden states from the base model and the hidden states produced with LoRA. I'm currently initializing two separate model objects:

        from peft import get_peft_model
        m1=AutoModelForCausalLM.from_pretrained(model_path)
        m2=AutoModelForCausalLM.from_pretrained(model_path)
        lora_config = LoraConfig(....)
        m2 = get_peft_model(m2, lora_config)

Is there already an api to call non-lora forward with m2 object ? I believe it'll be more memory efficient.

[BenjaminBossan]

Yes, you can disable all adapters like this:

with model.disable_adapter():
    output_base_model = model(**kwargs)