Hpc setup #1004
Hpc setup #1004
Conversation
Optional adds to the toml files for running in environments that uses non-torchrun launchers.
facebook-github-bot
added
the
CLA Signed
|
Aren’t those environment variables set automatically when using |
|
As I mentioned above, this is for non-torchrun launchers like mpirun/mpiexec etc. Torchrun is very specific to torch, HPC environments run many other applications which can not use torchrun. HPC environments are optimized for whatever launcher they were designed for. In those environments, torchrun usually does not work or has sub-optimal performance. |
There was a problem hiding this comment.
Sorry I'm not familiar with the HPC setup you mentioned. I wonder over there could you set the environment variables outside train.py like what we do in run_train.sh?
Even if you have to do it within the python job, with the flexible job config support (we recently added), you can customize it without changing torchtitan, e.g. by using torchtitan Trainer as a submodule.
https://github.com/pytorch/torchtitan/blob/main/docs/extension.md#extending-jobconfig
HPC type setup ( e.g. mpirun , mpiexec) currently requires code change in train.py ( or elsewhere) . Adding couple of optional additional args ( e.g. rank and local rank) to toml file along with a simple function in train.py removes the need to make those changes.