[FEAT] Integrate LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

[mbaddar1]

Feature request

Paper link : https://aclanthology.org/2024.naacl-long.174.pdf
Code link : https://github.com/yifanycc/loretta

Motivation : Tensor-Trains are one of the promising low-rank structures heavily applied for parameter-efficient machine learning, esp. LLMs

LoRETTA takes LoRa to the next level by re-parameterizing the updates for pretrained weights to set of two tensor-trains

Performance : The paper claims that loretta leads to significant parameters reduction wrt LoRa

Also the paper claims LoRETTA achieves comparable results to other BERT family models

Note
I have contacted authors about this FR
yifanycc/loretta#7

Your contribution

If it is OK, I would love to start a PR on that

[BenjaminBossan]

Thank you for your proposal to add LoRETTA to PEFT. The paper looks interesting enough from a methodology point of view, so it could be a good fit. Before you work on the PR, let's give the authors a bit of time to respond to your request on their repo. If, in one week, we don't hear back or they give their thumbs up, you can proceed.

Just a heads up, I skimmed their repo and saw that it's based on a very old version of PEFT, so integrating LoRETTA will require a bit of work that goes beyond copy-pasting their code base.

[mbaddar1]

Thanks @BenjaminBossan. Good idea, let's give authors one week or so

[mbaddar1]

Hi @BenjaminBossan
I have contacted Yifan Yang (Github, Website) via email and he showed a great interest in integrating LoRETA with PEFT. I you would like me to forward the conversation to you let me know. I will start a PR and share a development plan (As it is my first time contributing to PEFT to discuss)
Thanks.

[BenjaminBossan]

@mbaddar1 There is no need to forward the conversation. If you've got the okay from Yifan Yang, just go ahead and start the PR.

I will start a PR and share a development plan (As it is my first time contributing to PEFT to discuss)

That's fine, you can start with a draft PR and we can provide early feedback.