Enable publishing of files not in working directory

[inari-ldavey]

New feature

Hi, I love the new workflow outputs and the ability to publish from channels instead of just processes. However, one feature that is missing for me is the ability to publish files to the output directory that are not staged to the working directory. In the docs here it is stated that this feature is not supported, however it is not explained why. It also "fails" silently which might be confusing for many people (it was for me at least).

Use case

My use case is that I want to publish output files as well as input files to the same directory.

In the example below, running nextflow run main.nf does not publish test.txt to results/test.txt as you might intuitively expect.

main.nf:

#!/usr/bin/env nextflow

workflow {
	main:
	
	// test.txt is a random file in the current directory
	ch_test = channel.fromPath("test.txt")

	publish:
	test = ch_test
}

output {
	test {
	}
}

Suggested implementation

I don't know how to implement this, but you already have the file loaded in a Channel so I'm not sure why it can't be published and what the difference is with files in the working directory. Perhaps just a setting in file() or fromPath() to force stage a file could be a solution?

[bentsherman]

I'm also interested in enabling this, as you aren't the first person to request it.

The reason it hasn't been supported so far is because the file publisher uses the source path relative to the task directory to determine what the relative path should be in the publish destination. This mainly comes up when I publish a task output that is in a subdirectory, e.g. work/<hash>/foo/bar.txt. But files that aren't task outputs don't have this relative path because they aren't in a task directory.

So what should the relative path be? I think we can make it work by just using the file name. Just understand that if you publish a remote file e.g. http://example.com/foo/bar.txt to results, it will be published as results/bar.txt by default. You'll have to be explicit if you want to preserve the subdirectory bar:

workflow {
  publish:
  test = file('http://example.com/foo/bar.txt')
}

output {
  test {
    path { txt -> txt.parent.name }
  }
}