some info regarding Multi GPU

[FurkanGozukara]

@bmaltais can you give some info regarding Multi GPU that is coming with this pull request

https://github.com/bmaltais/kohya_ss/pull/2192/files

Previously, when we were training on a multi gpu platform, it was cloning the training process on each gpu

so lets say training on 1 gpu takes 20 gb. this was cloned on all gpus

thus lets say if i have 2 gpus, it was becoming effective batch size 2

with this new options can you explain difference? how to utilize? any info? thank you so much

[bmaltais]

I wish I could. What I am doing is exposing a way to configure the multi gpu option of the accelerate launch cli… but I personally have no idea and experience with how to best configure multi gpu parameters for kohya_ss training. If someone come up with a multi gpu preset I could add it to the list for others to use.

[FurkanGozukara]

thanks I asked Kohya himself lets see if he knows. thank you for your efforts

[okingjo]

Well, the interesting thing is I used to get multi GPU working, but now it is not after the last updates.
Previously, what I did was simply select "multi-GPU" during the accelerate config, just keep in mind that the epoch is "per GPU" other than total, and it worked perfectly. I tried on a 8x2080ti server and a 6x3090 server, linux, on docker, both were fine.
However, in the latest update, this was no longer the case. I tried everything: select “no distribution training” or "multi GPU", check or uncheck the "multi GPU' box, put "all" or the exact GPU IDs, or nothing in the GPU selection box, or resulting in the same failure "torch.distributed.elastic.multiprocessing.api: [ERROR] failed..."
Single GPU works fine, but as long as I start the multi GPU, it failed.
Interestingly, I tried on my windows laptop and it was the same torch.distributed.elastic.multiprocessing.api error (but I have never successfully run multi GPU on windows either so I was not surprised)