how many partitios will zero stage 3 divide mode

[cccc0der]

Hi!

I have 640 GPUs on 80 machines to do some pretrain work now, I have read the zero tutorials but didn't see any config about number of partitions.

My question is, if I start traing on all 80 machines with zero stage 3(without any model parrallel), will model-parameters be split to 640(dp 640?) parts loaded on each GPU?

Waiting for reply, thank you!

[tjruwase]

@cccc0der, yes, the vanilla zero3 will split each parameter across the 640 GPUs. However, we are integrating zero3 improvements that split across a subset of DP GPUs. One such called miCS is already available in the 0.9.2 release. A second algorithm will be out soon, so be on the lookout.

@samadejacobs, FYI.

[cccc0der]

thank you!

[yunll]

I use miCS config like this

I get this error

[JianqiaoLu]

do you solve this problem ? I also come across this,,,

[tjruwase]

@cccc0der, please open an issue for this.

[cccc0der]

done, please check it. #3485

[samadejacobs]

@cccc0der, calling MiCS Init in your code should fix reported error:
with mics.MiCS_Init(data_parallel_group......)

[learning-chip]

Hit the same bug and found here. Yes the error AttributeError: 'Parameter' object has no attribute 'comm' can be fixed by using with deepspeed.zero.MiCS_Init(config_dict_or_path=deepspeed_config): rather than with deepspeed.zero.Init():. The config_dict_or_path is required in order to know mics_shard_size at init stage.

This behavior is currently not tested in test_mics_config.py. Better add it to unit test?

[JianqiaoLu]

basically, how can i alter my code to run like mics_init instead of zero init? do you mean I need to change deepspeed internal code?