Run training across multiple GPUs and multiple nodes on DAWN

[llewelld]

Run training in two scenarios on DAWN:

1. Across four GPUs on a single node.
2. Across eight GPUs on two nodes.

Compare the timing and GPU usage results to check things are working as expected.

Compare the results with those from #16 on Baskerville to see whether there are any obvious differences.

[rwood-97]

TODO:

• 1 node, 1 GPU

#SBATCH --job-name=example-ddp
#SBATCH --output=one_node_one_gpu
#SBATCH --account=airr-p8-rcpp-dawn-gpu
#SBATCH --partition=pvc9 # Dawn PVC partition
#SBATCH -n 1   # Number of tasks (usually number of MPI ranks)
#SBATCH -c 24  # Number of cores per task
#SBATCH -N 1 # Number as nodes
#SBATCH --gres=gpu:1 # Number of requested GPUs per node
#SBATCH --time 01:00:00

• 1 node, 4 GPUs (full node)

#SBATCH --job-name=example-ddp
#SBATCH --output=one_node_four_gpus
#SBATCH --account=airr-p8-rcpp-dawn-gpu
#SBATCH --partition=pvc9 # Dawn PVC partition
#SBATCH -n 4   # Number of tasks (usually number of MPI ranks)
#SBATCH -c 24  # Number of cores per task
#SBATCH -N 1 # Number as nodes
#SBATCH --gres=gpu:4 # Number of requested GPUs per node
#SBATCH --time 01:00:00

• 2 nodes, 4 GPUs (2 GPUs per node)

#SBATCH --job-name=example-ddp
#SBATCH --output=two_nodes_four_gpus
#SBATCH --account=airr-p8-rcpp-dawn-gpu
#SBATCH --partition=pvc9 # Dawn PVC partition
#SBATCH -n 4   # Number of tasks (usually number of MPI ranks)
#SBATCH -c 24  # Number of cores per task
#SBATCH -N 2 # Number as nodes
#SBATCH --gres=gpu:2 # Number of requested GPUs per node
#SBATCH --time 01:00:00

• 2 nodes, 8 GPUs (full node)

#SBATCH --job-name=example-ddp
#SBATCH --output=two_nodes_eight_gpus
#SBATCH --account=airr-p8-rcpp-dawn-gpu
#SBATCH --partition=pvc9 # Dawn PVC partition
#SBATCH -n 8   # Number of tasks (usually number of MPI ranks)
#SBATCH -c 24  # Number of cores per task
#SBATCH -N 2 # Number as nodes
#SBATCH --gres=gpu:4 # Number of requested GPUs per node
#SBATCH --time 01:00:00

• 4 nodes, 4 GPUs (1 GPU per node)

#SBATCH --job-name=example-ddp
#SBATCH --output=four_nodes_four_gpus
#SBATCH --account=airr-p8-rcpp-dawn-gpu
#SBATCH --partition=pvc9 # Dawn PVC partition
#SBATCH -n 4   # Number of tasks (usually number of MPI ranks)
#SBATCH -c 24  # Number of cores per task
#SBATCH -N 4 # Number as nodes
#SBATCH --gres=gpu:1 # Number of requested GPUs per node
#SBATCH --time 01:00:00

[rwood-97]

I'm unsure if our code is working, we might potentially need to explicitly tell the 4x1 job to use one GPU per node.

1x4 (4 GPUs total)
[0] Average time per epoch (ignoring first): 17.7179012298584 seconds
[0] Effective time for an epoch: 4.4294753074646 seconds
[0] Equivalent training speed: 0.22576037049293518 epochs per seconds
[0] Total time for 32 epochs: 993.3740234375
[0] Total time: 318.66189789772034

4x1 (4 GPUs total)
[0] Average time per epoch (ignoring first): 16.719465255737305 seconds
[0] Effective time for an epoch: 4.179866313934326 seconds
[0] Equivalent training speed: 0.23924210667610168 epochs per seconds
[0] Total time for 32 epochs: 995.736328125
[0] Total time: 314.45791816711426

[rwood-97]

Trying this again...
I am assuming that 4x8 would only work if its being split across GPUs, but still unsure if this is doing 0,1,2,3,4,5,6,7 or 0,1,4,5,8,9,12,13

4x1 (4 GPUs total)
[0] Average time per epoch (ignoring first): 17.721572875976562 seconds
[0] Effective time for an epoch: 4.430393218994141 seconds
[0] Equivalent training speed: 0.2257135957479477 epochs per seconds
[0] Total time for 32 epochs: 1027.2216796875
[0] Total time: 310.24275302886963

4x2 (8 GPUs total)
[0] Average time per epoch (ignoring first): 20.47967529296875 seconds
[0] Effective time for an epoch: 2.5599594116210938 seconds
[0] Equivalent training speed: 0.39063119888305664 epochs per seconds
[0] Total time for 32 epochs: 1232.814453125
[0] Total time: 186.45419454574585

**EDIT: This is now fixed by #29 **

[rwood-97]

Results

1 node, 1 GPU

[0] Average time per epoch (ignoring first): 17.847061157226562 seconds
[0] Effective time for an epoch: 17.847061157226562 seconds
[0] Equivalent training speed: 0.05603163316845894 epochs per seconds
[0] Total time for 30 epochs: 622.86572265625
[0] Total time: 691.4870030879974

• repeat (use this one for reporting)...

[0] Average time per epoch (ignoring first): 17.873273849487305 seconds
[0] Effective time for an epoch: 17.873273849487305 seconds
[0] Equivalent training speed: 0.055949460715055466 epochs per seconds
[0] Total time for 30 epochs: 605.0115356445312
[0] Total time: 650.5900146961212

1 node, 4 GPUs (full node)

[0] Average time per epoch (ignoring first): 17.7179012298584 seconds
[0] Effective time for an epoch: 4.4294753074646 seconds
[0] Equivalent training speed: 0.22576037049293518 epochs per seconds
[0] Total time for 32 epochs: 993.3740234375
[0] Total time: 318.66189789772034

• repeat (use this one for reporting)...

[0] Average time per epoch (ignoring first): 17.708354949951172 seconds
[0] Effective time for an epoch: 4.427088737487793 seconds
[0] Equivalent training speed: 0.22588208317756653 epochs per seconds
[0] Total time for 32 epochs: 897.4592895507812
[0] Total time: 271.16698002815247

2 nodes, 4 GPUs (2 GPUs per node)

[0] Average time per epoch (ignoring first): 19.39745330810547 seconds
[0] Effective time for an epoch: 4.849363327026367 seconds
[0] Equivalent training speed: 0.20621263980865479 epochs per seconds
[0] Total time for 32 epochs: 899.998291015625
[0] Total time: 257.879816532135

2 nodes, 8 GPUs (full node)

[0] Average time per epoch (ignoring first): 19.380022048950195 seconds
[0] Effective time for an epoch: 2.4225027561187744 seconds
[0] Equivalent training speed: 0.4127962291240692 epochs per seconds
[0] Total time for 32 epochs: 1380.4417724609375
[0] Total time: 211.3158929347992

• repeat (use this one for reporting)...

[0] Average time per epoch (ignoring first): 19.383628845214844 seconds
[0] Effective time for an epoch: 2.4229536056518555 seconds
[0] Equivalent training speed: 0.4127194285392761 epochs per seconds
[0] Total time for 32 epochs: 1149.7550048828125
[0] Total time: 180.48243927955627

4 nodes, 4 GPUs (1 GPU per node)

[0] Average time per epoch (ignoring first): 19.57634925842285 seconds
[0] Effective time for an epoch: 4.894087314605713 seconds
[0] Equivalent training speed: 0.20432819426059723 epochs per seconds
[0] Total time for 32 epochs: 889.144287109375
[0] Total time: 264.3371512889862

4 nodes, 8 GPUs (2 GPUs per node)

[0] Average time per epoch (ignoring first): 19.656265258789062 seconds
[0] Effective time for an epoch: 2.457033157348633 seconds
[0] Equivalent training speed: 0.40699490904808044 epochs per seconds
[0] Total time for 32 epochs: 1218.0537109375
[0] Total time: 186.62970972061157

[rwood-97]

Total nodes	Total GPUs	Average time per epoch (seconds)	Effective epoch time (time / processes) (seconds)	Equivalent training speed (processes / epoch time) (epochs per second)
1	1	17.873	17.873	0.0559
1	4	17.708	4.427	0.226
2	4	19.397	4.849	0.206
4	4	19.576	4.894	0.204
2	8	19.380	2.422	0.413

[Iain-S]

Would we expect the average time per epoch to decrease with more GPUs? If an epoch is a complete pass over the whole dataset.

[rwood-97]

I think the times in the first row might actually be per batch not per epoch. As in, I think we time how long it takes to process one batch on each GPU and record that.

Maybe we should update the table headers?
I think effective epoch time is waht should be decreasing (again maybe this should say batch not epoch).