Adam-mini

[sdbds]

Tested batch size 1 can use about 16GB of VRAM.

[kohya-ss]

Thank you, that looks good!

Is the Adam_mini code from some repository? If so, please let me know so I can add the reference source and the license.

[sdbds]

Thank you, that looks good!

Is the Adam_mini code from some repository? If so, please let me know so I can add the reference source and the license.

code from here and i think we should test for different blocks.
Officail says reduce 50% VRAM but i just found reduce 20%...
I changed block checked for lora part.
https://github.com/zyushun/Adam-mini

[FurkanGozukara]

Thank you, that looks good!
Is the Adam_mini code from some repository? If so, please let me know so I can add the reference source and the license.

code from here and i think we should test for different blocks. Officail says reduce 50% VRAM but i just found reduce 20%... I changed block checked for lora part. https://github.com/zyushun/Adam-mini

this sounds amazing half VRAM of Adam for same performance

ah only 20% vram usage :/ did you got good results with it? par with Adam for real?

[sdbds]

Thank you, that looks good!
Is the Adam_mini code from some repository? If so, please let me know so I can add the reference source and the license.

code from here and i think we should test for different blocks. Officail says reduce 50% VRAM but i just found reduce 20%... I changed block checked for lora part. https://github.com/zyushun/Adam-mini

this sounds amazing half VRAM of Adam for same performance

ah only 20% vram usage :/ did you got good results with it? par with Adam for real?

At least on the SDXL I tested it works about the same.

[FurkanGozukara]

@sdbds thanks a lot for answer

do you think adafactor or mini adam would save more vram have you compared?

[Bocchi-Chan2023]

I'm curious about the difference in performance between CAME and adam-mini
#1409

[kohya-ss]

Is this PR includes #1417, or is it better to merge #1417 first?