Templates for training

[elismasilva]

Hi, do you have some template or sample configuration file for training SDXL ? i am starting training and testing hyperparameters, but i am trying to find one configuration that combine low vram consumption and velocity on training. I changed my bucket to 768x768 put grandient to 4 and batch size to 1 with 16 images. My scheduler is 'sine', what params impact more in memory consumption?

Thank you!

[bghira]

likely the most you can 'get away with' is:

• adafactor optimiser
• pytorch latest build
• CUDA 11.8, as 12.x seems to waste VRAM (confirmed by multiple users)
• low batch size
• zero gradient accumulation steps, as this also consumes memory
• lower resolution

even then, with all of the optimisations, SDXL training will consume more than 22G of VRAM without DeepSpeed.

SDXL is a large model. if you aren't re-parameterising the thing (eg. going from epsilon to x-prediction) you are likely better off training a LoRA.

try USE_DORA=true for LoRA training.

if you're really needing to optimise full u-net training at the expense of correctness guarantees, OneTrainer claims to have support for full u-net training on a 3060 Ti. that's not something i'm interested in implementing here, as i honestly believe that's just not a good idea.

[elismasilva]

How did you know i have 3060 Ti ? ahaha i think then i will need test OneTrainer.

[bghira]

yep, 8GB just isn't enough. it will work on a LoRA at a batch size of 1 and resolution of 512px, but you'll just be wasting time with that level of compute.

when using DeepSpeed, it still needed 9GB of VRAM, and almost 80GB of system memory - even at the highest level of optimisation i could achieve.

[elismasilva]

I understand, well I'm not a person who gives up easily, maybe I can solve it and train with 8GB, I like to try haha. But you need to fix that annoying bucket error, I don't know what else to do with it haha.