Integration into PEFT

[githubnemo]

Hey!

This method looks quite interesting! Do you by any chance want to integrate it into PEFT?

Next to showing good results Uni-LoRA would also be interesting because currently we don't see as good as results for
low-parameter count results and Uni-LoRA could be a good competitor to LoRA in the benchmarks.

If so, feel free to open an early draft PR and we can get the implementation going!

Let me know if need any guidance :)

[KaiyangLi1992]

Thanks a lot for the kind words and for the encouragement!

Yes , I’ve already opened an early draft PR integrating UniLoRA into PEFT:huggingface/peft#2968

The current PR includes the core UniLoRA tuner implementation, configuration,
basic tests. I’m very happy to iterate on the implementation and adjust anything as needed.