Consider using TTL more extensively in k8s executor

[BioWilko]

New feature

Currently ttlSecondsAfterFinished is an optional config option for k8s executor which adds metadata to jobs/pods which tells the k8s control plane to kill resources upon completion / failure after that many seconds.

However, the nextflow monitor pool loop will actively delete resources some amount of time post completion, this appears to be variable from my observation and if you have lots of fast running processes this can add up to lots of resources for the loop to monitor slowing the main process down. Also, if the control plane kills a job before nextflow tries to this will lead to an error like this:

Oct-06 18:32:43.421 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Unexpected error in tasks monitor pool loop
org.codehaus.groovy.runtime.InvokerInvocationException: nextflow.k8s.client.K8sResponseException: Request DELETE /apis/batch/v1/namespaces/<NAMESPACE>/jobs
/nf-62e16f1931488513c9732bfd1e7d5718-f9506 returned an error code=404

  {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {
          
      },
      "status": "Failure",
      "message": "jobs.batch \"nf-62e16f1931488513c9732bfd1e7d5718-f9506\" not found",
      "reason": "NotFound",
      "details": {
          "name": "nf-62e16f1931488513c9732bfd1e7d5718-f9506",
          "group": "batch",
          "kind": "jobs"
      },
      "code": 404
  }

        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:348)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
        at nextflow.processor.TaskPollingMonitor$_start_closure2.doCall(TaskPollingMonitor.groovy:323)
        at nextflow.processor.TaskPollingMonitor$_start_closure2.call(TaskPollingMonitor.groovy)
        at groovy.lang.Closure.run(Closure.java:505)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: nextflow.k8s.client.K8sResponseException: Request DELETE /apis/batch/v1/namespaces/<NAMESPACE>/jobs/nf-62e16f1931488513c9732bfd1e7d5718-f9506 re
turned an error code=404

In the log which puts the pipeline into a state where it will no longer spawn new jobs / pods but also not finish with an error.

I suggest that you consider refactoring some of the k8s functionality to make better use of k8s features (such as TTL), better utilise the control plane to cheaply achieve many of the same features, or at the very least better handle the case where a resource no longer exists when nextflow tries to delete it.

[bentsherman]

Okay I think I understand. The K8s executor does delete pods that complete successfully, because we observed early on that K8s was not typically cleaning them up on its own.

However, most executors don't do this -- they just let the job get cleaned up on its own -- so normally the task polling loop is pretty lightweight. But I guess with K8s the polling loop could get bogged down as it waits for each job to be deleted before checking the next job

As for the error, I suspect this can happen if you specify ttlSecondsAfterFinished, because then the job might get cleaned up by K8s before Nextflow.

We added ttlSecondsAfterFinished a few versions ago just to try it out, but if it would help to integrate it more deeply then I'm down to try. If you have an idea of what the best practice should be, I'm all ears

For example, a simple improvement that comes to mind is that if ttlSecondsAfterFinished is set, then we shouldn't bother deleting the job:

nextflow/plugins/nf-k8s/src/main/nextflow/k8s/K8sTaskHandler.groovy

Lines 513 to 534 in 253666d

	protected void deletePodIfSuccessful(TaskRun task) {
	if( !podName )
	return
	if( cleanupDisabled() )
	return
	if( !task.isSuccess() ) {
	// do not delete successfully executed pods for debugging purpose
	return
	}
	try {
	if ( useJobResource() )
	client.jobDelete(podName)
	else
	client.podDelete(podName)
	}
	catch( Exception e ) {
	log.warn "Unable to cleanup ${resourceType.lower()}: $podName -- see the log file for details", e
	}
	}

[BioWilko]

I think that would be an ideal approach to speed things up, I also think that you should consider setting ttlSecondsAfterFinished to a default value to improve the general speed of the executor, something like 10 minutes seems reasonable to me!