Merging several LoRA

[Rudy34160]

Hi,

I would like to know if it is possible to merge several LoRAs, some of which are identical but with different weights, in order to create a single one.

I have done many tests on a LoRA that I am trying to refine, and I would like to keep the best of each test.

For example, in my prompt, I would like to transform

"lora:ejlor04_dream-000004:0.3 lora:ejlor04_dream-000008:0.2 lora:ejlor04_dream-000009:0.2 lora:ejlor09_dream-000008:0.2 lora:ejlor06_dream:0.1"

into a single LoRA, like "lora:ejlor_final:1".

I saw the possibility to do it in Kohya GUI, but it only works for 2 LoRA, and I can't specify the weights...

Ideas ? 🤔

[ride5k]

supermerger extension should be able to do this, have done so myself with it.

[Rudy34160]

In the same type of configuration as my example?🤔

[kohya-ss]

There are two scripts in this repo to merge: networks/merge_lora.py and svd_merge_lora.py.

There is a document for them (Japanese only). I hope this helps you.

https://github.com/kohya-ss/sd-scripts/blob/main/docs/train_network_README-ja.md#%E3%83%9E%E3%83%BC%E3%82%B8%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%97%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6

[oberon168]

When I merge using Kohya_ss GUI Utilities to merge lora for 4 models with weights. If they are coming from the same subset of a training set, it works. But if they are coming from different training set, the output seems as it forget the keyword and not producing the subject. However if I join them under additional network in automatic1111 gui, it works fine. Any idea?