Intergrate MonteCLoRA (TMLR 2025 accepted) into PEFT

[victor7246]

We are integrating Monte-Carlo-based robust LoRA (MonteCLoRA), accepted at TMLR 2025 - https://openreview.net/forum?id=2HFmicB8kh. This method uses probabilistic priors and mixture-of-LoRA for robust estimation of low-rank adapters.

• Added monteclora under tuner
• Update tuner_utils with monteclora changes
• Added fine-tuning examples on GLUE classification task with RoBERTa-base model
• All quality checks are passed.

[BenjaminBossan]

Thanks for this PR to add MoneCLoRA to PEFT.

I went through the paper but must confess that I don't understand all of the math in there. I will trust you on the correctness of the implementation when it comes to this.

Regarding the PEFT integration, I think we should take a bit of a different approach. Right now, you inherit from LoraLayer, LoraModel, etc. The reason is that you can reuse most of the existing code. However, this approach can be problematic, as other code may do checks like if isinstance(layer, LoraLayer) assuming it gets a normal LoRA layer.

Checking the actual changes you added, I have thus a different proposal: Let's implement MonteCLoRA as a "LoRA variant". This way, users can use a normal LoRA model but with MonteCLoRA under the hood. Check the existing LoRA variants to get an idea of what is required for this:

https://github.com/huggingface/peft/blob/f2c066889543778ea3eaf43798d041bd9acda5a7/src/peft/tuners/lora/variants.py

From a user's perspective, it would look something like this:

monte_clora_config = MonteCLoraConfig(...)
lora_config = LoraConfig(..., use_monteclora=monta_clora_config)  # by default use_monteclora = False
model = get_peft_model(model, lora_config)  # <= uses normal LoraModel

Please check if that makes sense to you.

Also, while working on this, please ensure that the copyright notice is present in all new files and with the current year. Once you finish, don't forget to call make style.

There are some other areas that need work, but let's get these out of the way first and then we can discuss the rest.

[BenjaminBossan]

It should be fine to add this to the proper PEFT package, so that users don't need to copy & paste this code. A good place would be the helpers.py file. Also, let's define a separate mixin class with compute_loss and MonteCLoRATrainer which inherits from Trainer and this mixin. The reason for that is so that users who use a different Trainer class, say, SFTTrainer, can use the mixin.

[BenjaminBossan]

This is all unnecessary, right? Let's remove this.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

Gentle ping @victor7246

[iamsecretlyflash]

Hi @BenjaminBossan ,
I am a co-author on the paper. Firstly, @victor7246 and I would like to apologise for the delay in communication.
As for the changes, we are okay with all the suggested changes and are working to incorporate them ASAP.