make main.nf process multiple datasets

[tischi]

Hi @BioinfoTongLI,

@sebgoti and myself were wondering how to process multiple images with the current workflow.

I guess this line of code would need some modification?

[BioinfoTongLI]

yes. but that is not the ultimate solution.
The ultimate solution would be writing all the metas and their corresponding paths into a json or yaml file and pass them as -params-file.
Then split them into channels using channel.from
I can commit an example file later

[tischi]

I am often using wild-cards:

nextflow run main.nf --inputPattern "IJ-TIFF/*_s01.tif"

And then inside main.nf:

images = Channel.fromPath("${params.inputPattern}")

Is that bad practice?

[BioinfoTongLI]

not at all. That's a clever way of doing it. But in our case here, we also have meta and output dir tightly attached to each input image folder. You will have issues say if you want to apply different blurring to different set of images. And the output are folders as well. This is what makes this more complex....

[BioinfoTongLI]

this commit should allow multidataset processing
2d7434e
the command line to run:
nextflow run . -params-file data/input_params.yaml
It should run ok for the first time. But it will throw error in the second run, saying the folder already exists etc. Just delete the output folder in that case.
Basically we need to find a better way for -resume/rerun etc.

[krokicki]

I think the nf-core way is to use a samplesheet CSV file. This is even tied into nf-core parameter validation now (see recent Nextflow Summit talk) so you can validate the internals of the samplesheet. Kind of off topic, but I think parameter validation would be really useful. It's very easy to create a schema using the web-based schema builder.

Here's an example of using samplesheets for processing multiple CZI/MVL files. Note that I still have the old-style validation in there with a custom Python script.

When I get time I can work on adding this, if you all agree it's the way to go.

[BioinfoTongLI]

I wasn't able to attend the summit. Bummer!
Yes, this looks great to me!