flux GPUs

[bgruening]

@sanjaysrikakulam this seems to work for me.

We could also pass $_CONDOR_AssignedGPUs to --gpus ... but this seems to be very tricky for quoting.

docker/docs#11010

[sanjaysrikakulam]

I tried to a dig a bit and did not come up with a cleaner HTCondor solution for this. Moving forward, for multi-GPU destinations with Docker containerized tools, we should explicitly add this ENV to ensure the tool sticks to the assigned GPU, rather than all the tools/jobs using GPU index 0.

[bgruening]

I guess there is no harm in adding it to the default Docker destination?

[sanjaysrikakulam]

I guess there is no harm in adding it to the default Docker destination?

I am unsure if TPV merges the docker_run_extra_arguments and singularity_run_extra_arguments when they are specified in multiple places (destinations, tools, tool_defaults, users, roles, etc.).

[sanjaysrikakulam]

I had a thought over the weekend on how we could add this to the destinations. We could try something like the following

rules:
  - id: has_gpus_all_flag
    if: "'--gpus all' in (entity.params.get('docker_run_extra_arguments') or '')"
    params:
      docker_run_extra_arguments: "{{ entity.params.get('docker_run_extra_arguments') or '' }} --env CUDA_VISIBLE_DEVICES=$_CONDOR_AssignedGPUs"

The above is a generic solution; if we have a GPU dedicated destination, then we don't have to check for --gpus all explicitly. This needs to be tested, though.