resubmit to queue with new job resources

[msiron-entalpic]

One useful feature would be to resubmit something to the queue with new resources. Sometimes on SLURM clusters, we realize after submitting that some partition are full but some partitions are available. Currently we can do something like this to update a job resources:

jf job set resources -did 541 '{"cpus_per_task": 56, "partition": "new_partition"}'

However, this will not go through if the job is already submitted to slurm and just awaiting for SLURM to schedule it. And we might not be able to edit the slurm partition if the requested resources differ between partition. Instead jobflow would return an error similar to:

[11:15:29] ERROR    Error while setting for job 541                                                                                                                                                       
                    ValueError: Job in state UPLOADED. The action cannot be performed    

or:

[11:18:55] ERROR    Error while setting for job 541                                                                                                                                                       
                    ValueError: Job in state CHECKED_OUT. The action cannot be performed    

It would be nice to have a feature which re-submits a SLURM job with new resource allocation if its already waiting to be scheduled/submitted. However this might involve cancelling the appropriate SLURM job in conjunction.

[JaGeo]

@msiron-entalpic i agree that this would be very useful

[gpetretto]

Hi @msiron-entalpic,

I see your point. Currently the error serves the purpose to make the user aware that, since the job has already been submitted, applying this change in the DB will not take effect on the actual submitted job.

At present, the simplest way to cancel the slurm job associated with a specific Job would be to stop the job because since the last version released a few days ago stopped jobs can be rerun. The downside is that I have just realized that for a STOPPED/USER_STOPPED job it is still not possible to set the resources, which I would change in the next version. So at the moment one possible way to achive what you want would be running:

jf job stop JOB_ID
jf job rerun JOB_ID
jf job set resources JOB_ID ....

I think for this it would be better to be able to set the resources before the rerun, because otherwise the runner could pick it up before the resources are changed.

Would this a good solution? Or are you instead suggesting of having the option to basically stop/rerun the job directly in the job set resources?