Why won't my workflow work when I use the syntax labels? (take and main)

[julia-luz]

Hey there! I'm not sure what I'm doing wrong. I'm reading the documentation on DSL2 Workflows and so far I have an implicit worflow that works, but only if I don't define the input channels with take: or use main: to define my workflow. Am I missing something?

This workflow works fine:

workflow {
    

    reads_ch = Channel.fromPath(params.reads_tsv, checkIfExists: true )
    .splitCsv( header: true, sep: '\t' )
    .map{ row -> tuple( row.sample_name, file(row.fq1, checkIfExists: true) ) }
    
    first_step(reads_ch, params.refgen_mmi)
    second_step(first_step.out.trimmed, params.refgen_fa, params.refgen_fai)
    last_step(second_step.out.outfinal)

}

I want to eventually make this a subworkflow, so I created channels for those parameters I was using (input_fa = Channel.fromPath(params.refgen_fa, checkIfExists: true) and so on and so forth) for the sake of modularity, defined them outside the Workflow block (later I plan on having those be defined on my main workflow) and changed the input channels of my processes accordingly . Then I changed the workflow definition to have the take: and main: blocks:

workflow{
    take:
        
    reads_ch
    input_fa
    input_fai
    input_mmi

    main: 

    first_step(reads_ch, input_mmi)
    second_step(first_step.out.trimmed, input_fa, input_fai)
    last_step(second_step.out.outfinal)
}

Trying to run this results on the error:
ERROR ~ Main workflow declares 4 input channels but 0 were given

What am I doing wrong? I tried naming that workflow and then adding another implicit wokflow calling that one as workflow{NAMED()}: same error. Now, if I try adding the same input channels I had defined in the take block:
workflow {NAMED(reads_ch, input_mmi, input_fa, input_fai)}

This at least begins to run, however, it fails on the second step - Seemingly because the first_step.out.trimmed that appears as the second process input is not in the inputs of the main workflow callling NAMED ...

Which seems wrong to me - nowhere in the documentation did I find anything mentioning this syntax is only valid for named workflows, and the examples of subworkflow invocations in the documentation don't really have process outputs being explicitly called again, or even being in the take block for that matter - not that I could find it, at least, I may just be missing a very obvious paragraph in the documentation.

Any help understanding this syntax better will be greatly appreciated!!

[bentsherman]

Hi @julia-luz , it's an understandable mistake as some of these rules are not clearly documented. I will make a note to expand the workflow docs on this point

The entry workflow can't have take: or emit: because you can't call it. Meanwhile, the take: inputs can be channels or even just regular values, so you have some flexibility in how you define the interface for your named workflow.

You could construct the input channels in the entry workflow:

workflow {
    reads_ch = Channel.fromPath(params.reads_tsv, checkIfExists: true )
        .splitCsv( header: true, sep: '\t' )
        .map { row -> tuple( row.sample_name, file(row.fq1, checkIfExists: true) ) }
    NAMED(reads_ch)
}

workflow NAMED {
    take:
    reads_ch

    main:
    first_step(reads_ch)
    // ...
}

Or you could pass only the params and construct the channels in the named workflow:

workflow {
    NAMED(params.reads_tsv)
}

workflow NAMED {
    take:
    reads_tsv

    main:
    reads_ch = Channel.fromPath(reads_tsv, checkIfExists: true )
        .splitCsv( header: true, sep: '\t' )
        .map { row -> tuple( row.sample_name, file(row.fq1, checkIfExists: true) ) }
    
    first_step(reads_ch)
    // ...
}

So it's a matter of what you consider to be "in scope' for your named workflow. The second option would allow you to re-use the subworkflow with any input channel, not necessarily one from a csv file, which is why I tend to take the second approach. But it's up to you.

Hope that clears things up for you.

[julia-luz]

Hi there @bentsherman ! Thank you for your speedy reply!

The entry workflow can't have take: or emit: because you can't call it. Meanwhile, the take: inputs can be channels or even just regular values, so you have some flexibility in how you define the interface for your named workflow.

Oh wow. Yeah, that's really helpful, I think the documentation doesn't mention this. Good to know!

Also, I just want to make sure: Does one need to add the process output channels that are chained into the next process' inputs ( in my dummy code, those would be first_step.out.trimmed and second_step.out.outfinal) in the workflow's take block too? I understand you are only showing the reads_ch being used for convenience's sake, but this was another confounding factor in the workflow composition/syntax for me, and I'd really appreciate it if you could shed some light on how those channels would "behave" when using this syntax.

Many thanks!

[bentsherman]

Does one need to add the process output channels that are chained into the next process' inputs ( in my dummy code, those would be first_step.out.trimmed and second_step.out.outfinal) in the workflow's take block too?

No, the take: section should only have the inputs that you pass in when you call the workflow, it's like function parameters and arguments in other programming languages (and in Nextflow too, but that's another topic). The process .out channels are essentially intermediate variables that exist only in the workflow after you call the process.

If it would make things more clear, you can also write the process calls like this:

    first_out = first_step(reads_ch, input_mmi)
    second_out = second_step(first_out.trimmed, input_fa, input_fai)
    last_step(second_out.outfinal)