submit slurm script does not allow to continue an experiment

[tjhunter]

Is your feature request related to a problem? Please describe.

The private submit.py script currently expects the passed slurm script to contain all the logic: train, evaluate, etc.

What is better?

• a single slurm script with many options
• one slurm script for each of the train, finetune, continue, evaluate/sample

It seems to me each of these stages has different hardware requirements, so I would be enclined to have one script for each stage, but I am not an HPC expert here.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Organisation

No response

[tjhunter]

@sophie-xhonneux @clessig what are your thoughts?

cc @Jubeku

[sophie-xhonneux]

You can have dependencies between slurm scripts, i.e. that they can only run after another job has finished, which might make several slurm scripts quite convenient as any individual stage can be re-run quite easily.

[tjhunter]

evaluation also has the same issue:

• evaluation has separate script

• keep same script for train + continue

ensure that the slurm config can be overridden from command line

[Jubeku]

The issue with train_continue script is that it should not again copy anything when only run_id is specified, and copy from the submitted run dir to a new run when parameters are changed.

[Jubeku]

@tjhunter, @clessig, do we agree on following logic?

	--run-id	--from-run-id	new directory?
1	no	no	yes, under generated run-id
2	yes (new desired name)	no	yes, under provided (new) run-id
3	no	yes	yes, under generated run-id, copied from old run-id directory
4	yes	yes	no, continue in old run-id directory (--run-id == --from-run-id)

For case 3, if we want to change any parameters, do we have to provide them all via command line as we don't want to mess with the files in the old repo?
Maybe we need another case, also for fine-tuning, where we copy from a new dir but continue from an old checkpoint. Not sure if we should still copy the backbone config from the old directory and only finetune-specific configs from the new dir.

[Jubeku]

@clessig, @tjhunter, another question:
Say we continue a previous run under a new run_id (case 3 above). Do we want to copy only the WeatherGenerator from dir of the previous run_id, or both WeatherGenerator and WeatherGenerator-private?
Maybe I want to change the slurm script for submitting and that could be independently defined from the WeatherGenerator-private of the previous run.

[clessig]

s run under a new run_id (case 3 above). Do we want to copy only the WeatherGenerator from dir of the previous run_id, or bo

Copy both, it's not many storage and files, and might save us trouble.

[clessig]

@tjhunter, @clessig, do we agree on following logic?

--run-id --from-run-id new directory?
1 no no yes, under generated run-id
2 yes (new desired name) no yes, under provided (new) run-id
3 no yes yes, under generated run-id, copied from old run-id directory
4 yes yes no, continue in old run-id directory (--run-id == --from-run-id)
For case 3, if we want to change any parameters, do we have to provide them all via command line as we don't want to mess with the files in the old repo? Maybe we need another case, also for fine-tuning, where we copy from a new dir but continue from an old checkpoint. Not sure if we should still copy the backbone config from the old directory and only finetune-specific configs from the new dir.

Case 3: This is a safety option to not invertedly continue in the same dir? Intuitively, I would have assume this options continues in the same dir
Case 4: generate when --run-id \neq --from-run-id

[Jubeku]

@tjhunter, @clessig, do we agree on following logic?
--run-id --from-run-id new directory?
1 no no yes, under generated run-id
2 yes (new desired name) no yes, under provided (new) run-id
3 no yes yes, under generated run-id, copied from old run-id directory
4 yes yes no, continue in old run-id directory (--run-id == --from-run-id)
For case 3, if we want to change any parameters, do we have to provide them all via command line as we don't want to mess with the files in the old repo? Maybe we need another case, also for fine-tuning, where we copy from a new dir but continue from an old checkpoint. Not sure if we should still copy the backbone config from the old directory and only finetune-specific configs from the new dir.

Case 3: This is a safety option to not invertedly continue in the same dir? Intuitively, I would have assume this options continues in the same dir Case 4: generate when --run-id \neq --from-run-id

Yes, case 3 is a safety option: If you really want to continue in the old folder you need to specify both run_id == from_run_id. In particular when sharing folders across multiple people, the idea is to create a new dir by default.