Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save memory of extract_lora_ups_down by returning a generator instead… #24

Closed
wants to merge 2 commits into from
Closed

save memory of extract_lora_ups_down by returning a generator instead… #24

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b49e0ca
save memory of extract_lora_ups_down by returning a generator instead…
Thomas-MMJ Dec 13, 2022
b6f7b95
Merge branch 'cloneofsimo:master' into patch-1
Thomas-MMJ Dec 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions lora_diffusion/lora.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,18 +78,16 @@ def inject_trainable_lora(


def extract_lora_ups_down(model, target_replace_module=["CrossAttention", "Attention"]):

loras = []

no_injection = True

for _module in model.modules():
if _module.__class__.__name__ in target_replace_module:
for _child_module in _module.modules():
if _child_module.__class__.__name__ == "LoraInjectedLinear":
loras.append((_child_module.lora_up, _child_module.lora_down))
if len(loras) == 0:
no_injection = False
yield (_child_module.lora_up, _child_module.lora_down)
if no_injection:
raise ValueError("No lora injected.")
return loras


def save_lora_weight(
model, path="./lora.pt", target_replace_module=["CrossAttention", "Attention"]
Expand Down