local gradio issues

[Manni1000]

these packages are missing in the requirements.txt
pip install h5py
pip install opencv-python

in the gradio_demo.py :
from models.lora import replace_linear_with_lora
shuld be
from models_accessory.lora import replace_linear_with_lora

train_args = torch.load(os.path.join(args.ckpt, "model_args.pth"))
shuld be
train_args = torch.load(os.path.join(args.ckpt, "model_args.pth"), weights_only=False)

also its still using above 24gb vram so i could not test it further. quite surprising because way bigger models like flux work on my pc. maby adding a quantized version would help.
i have tired to add model offloading but the main model still used so much vram that i could not runn it.

[Manni1000]

i guess that its not actually running in bf16 but i am not sure.

[RuoyiDu]

Thanks! Considering the code has just been reorganized, there might be some minor issues. Your feedback is very valuable!

May I ask whether the OOM issue occurred during model loading or the inference stage? Due to the presence of conditioning images, when generating a 1024×1024 image, the model actually processes a sequence length roughly equivalent to that of a 1024×2048 image. This high resolution may lead to higher VRAM requirements.

[Manni1000]

the oom issue is happening at the inference stage. i think when computing the attention.

[poluramus]

The inference is starting correctly, but no output after 10 minutes, inference is still running... I have a 3090. Is that normal behavior ?

[yyyyyxie]

I used an 80g a100 to inference, set the inference_step to 1, and it took me 600 seconds