v0.9.3 - no more autocast

[bghira]

• option renamed, vae_cache_behaviour -> vae_cache_scan_behaviour
• add --encode_during_training to skip pre-processing of vae embeds (only - for now)
• majorly reworked vae embed management code means there may be some issues at first
• debug.log in project dir root now contains the DEBUG level outputs
• add --adam_bfloat16 for sdxl and sd2.x, essentially mandatory now.
• precision level improvements and fixes for SD 2.x and SDXL training
  • fp16 is no longer supported on SD 2.x or SDXL. use bf16.
  • the SD 2.x VAE can still run in fp16 mode, but it's not recommended. it's bf16 by default now.
• pytorch 2.3 requirement on apple mps for access to bf16
• mps: fixes to loading torch files from disk, moving items into correct dtypes
• mps: fix multiprocessing, enable by default while preserving --disable_multiprocessing as a workaround
  • apple python uses "spawn" multiprocessing strategy but linux uses "fork", switched to fork by default
• mps: enable unet attention slicing on SD 2.x to avoid NDArray crash in MPS
• (for large datasets) preserve_data_backend_cache now accepts string values as well as bool, "image" and "text" to preserve just a split of the cache.
• skip certain OS directories on MacOS and Jupyter notebooks

pull requests

• sd2x: num steps remaining fix
• vaecache: exit with problematic data backend id by @bghira in sd2x: num steps remaining fix | vaecache: exit with problematic data backend id #332
• Feature/on demand vae cache by @bghira in Feature/on demand vae cache #336
• stochastic bf16 impl by uptightmoose by @bghira in stochastic bf16 impl by uptightmoose #337
• next by @bghira in next #339

Full Changelog: v0.9.2...v0.9.3

---

This discussion was created from the release v0.9.3 - no more autocast.

[minienglish1]

Curious as to why you made the decision to go full bf16?
I've always wondered if full fp16/bf16for sdxl training would produce high quality results, but never got around to testing.
How do full bf16 results compare to mixed precision fp16 results?

Also, amazing project, you've put a lot into it and it shows.

[bghira]

use the BF16 optimizer code from this repo or the pypi project (https://socket.dev/pypi/package/adamw-bf16)

it's important that stochastic rounding is used with full bf16.

though there is also the AnyPrecision optimizer which uses a different strategy, and no stochastic rounding.

for MPS, there is no 8Bit AdamW. that is reliant on Bits and Bytes, which I also don't think works on ROCm yet.

for what it's worth, the training precision matters a lot for which downstream hardware will be able to run your model. see how SD 2.1-v was trained with Xformers, which caused overflows when not using Xformers as it was trained in bf16, which has more range than fp16.

[minienglish1]

Appreciate the help. I'll take some time to read up and learn about your suggestions and figure out what I want to do. I'm still a novice programmer so I have lots to learn. Luckily, the same hardware will be used for both training and inference, so I don't have to worry about potential differences between end-users. Thanks again!

[bghira]

programming or impostor syndrome, i'm not sure which, have me feeling like a life-long novice. you're in pretty deep for a so-called beginner :P don't get discouraged. remember to take breaks, spend time with people you enjoy annoying in meatspace.

[minienglish1]

Appreciate the support. I think I've been programming in my spare time now for 9 months now, so truly still a novice. But I started with the specific goal of writing a sdxl training script, so that's all I know how to program. Started from zero programming skills, read half of "Automate the Boring Stuff With Python", then dove straight into accelerate.

Your repo is part of the reason why I can program today. Long time ago, I tried to get it to work, but couldn't because I had no idea what I was doing. Figured I need to learn how to program if I want to make any progress.

I'm doing all of this to make teaching materials for children's English classes at a school I own. So I spend every day playing with kids & writing code afterwards. Good times.

Again, seriously appreciate the info. I'll put it to good use.

[bghira]

it can be so satisfying. i am on my 20th year building total garbage and i love it