POC: Global Cache

[jorgee]

This PR is a test to validate the possibility of a global cache to share the task caching between executions. This is forcing to have the same cloud-cache, working dir and sessionId for a defined global-cache.It is able to detect when same task is executed by different workflows without using the resume option.

-with-globalcache=<cloud_path> parameter overwrites cloud cache path with <cloud_path>/.nf-global-cache and the working dir with <cloud_path>/work and creates a SessionId hashing the global cache path. This option implies -resume.

I tested the concurrent execution of tasks in two different workflow and it is detecting if it is the same and generating a new hash. When executing a third workflow with the same task after the execution of the task it detects that is cached and do not run the task again.

[bentsherman]

@jorgee since there is renewed interest in the global cache and you might be working on this soon -- I recommend implementing a TaskHasherV1 / TaskHasherV2 and adding a setting (e.g. environment var) for which task hash to use. Then the TaskHasherV2 simply needs to not include things like the session id, process name, etc

[bentsherman]

Updates:

• use NXF_GLOBALCACHE_PATH environment var instead of CLI option
• set session ID to all zeros
• remove session ID and process name from task hash when global caching is enabled

[bentsherman]

Extra note: set process.cache = 'deep' alongside this to enable global content-based caching.

[bentsherman]

Note about concurrency

There is a slight chance that two Nextflow runs try to run the same task at the exact same time, in which case they will use the same task directory and clobber each other.

We don't know how likely it is, but at a high enough scale I suspect it could happen enough to be very annoying. So, this POC is only recommended for testing at small scale. Otherwise, be prepared for weird concurrency errors