submit slurm script does not allow to continue an experiment

[tjhunter]

Is your feature request related to a problem? Please describe.

The private submit.py script currently expects the passed slurm script to contain all the logic: train, evaluate, etc.

What is better?

• a single slurm script with many options
• one slurm script for each of the train, finetune, continue, evaluate/sample

It seems to me each of these stages has different hardware requirements, so I would be enclined to have one script for each stage, but I am not an HPC expert here.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Organisation

No response

[tjhunter]

@sophie-xhonneux @clessig what are your thoughts?

cc @Jubeku

[sophie-xhonneux]

You can have dependencies between slurm scripts, i.e. that they can only run after another job has finished, which might make several slurm scripts quite convenient as any individual stage can be re-run quite easily.

[tjhunter]

evaluation also has the same issue:

• evaluation has separate script

• keep same script for train + continue

ensure that the slurm config can be overridden from command line

[Jubeku]

The issue with train_continue script is that it should not again copy anything when only run_id is specified, and copy from the submitted run dir to a new run when parameters are changed.