Multiple files emitted as single output

[Edoardo Giacopuzzi]

I have a simple workflow in DSL2 that takes an input file (a tab-separated text file) and processes it using a script to generate multiple .yml files which are passed as the output channel.

process1 {

output:

    path '*.yml'

  }

This output channel is then passed as input to a second process that use them to perform some analysis. 

process2 {

input:

    file(config_file)

}

However, the first process outputs all generated files as a single emission apparently, since the second process is invoked a single time with all files from process 1 as input.

This happens with nextflow 20.10 and 21.10. Any idea of what I'm doing wrog here?

My actual code is as follows

```

workflow {

if (params.exomiser) {

output_exomiser.mkdirs()

output_exomiser_settings.mkdirs()

appsetting_exomiser = file(params.exomiser_appsettings)

hpo_file_exomiser = file(params.HPO)

input_file_exomiser = file(params.exomiser_input)

exomiser_template = file(params.exomiser_template)

configure_exomiser(exomiser_template, input_file_exomiser, hpo_file_exomiser)

exomiser(configure_exomiser.out, appsetting_exomiser)

}

}

process configure_exomiser {

publishDir "$output_exomiser_settings", mode: 'copy'

input:

file(template)

file(input_file)

file(hpo_file)

output:

path '*.yml'

script:

// This python script creates multiple .yml files in the folder and I can see them

"""

python $projectDir/bin/Configure_exomiser.py \

-t $template \

-i $input_file \

-p $hpo_file

"""

}

// This process is supposed to be invoked one time for each .yml from the process above,

// Instead all .yml files are passed together and this process is running a single time taking all of them as input

process exomiser {

publishDir "$output_exomiser", mode: 'move'

input:

file(analysis_setting)

file(application_setting)

output:

file '*.tsv'

file '*.json'

script:

"""

java -jar ${params.exomiser_cli} \

-analysis $analysis_setting

"""

}

```

[Jing Yu]

What if you “flatten” the output from the first process?

https://www.nextflow.io/docs/latest/operator.html#flatten

Kind regards,

Jing

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/14eb625f-864f-4969-8c31-aa337cf8e011n%40googlegroups.com.

[Edoardo Giacopuzzi]

Great thanks! It worked perfectly!

I supposed the glob pattern would have resulted in multiple emissions instead is a single emission as Array / Collection... I see the point for this (in case process 1 takes multiple inputs as well), even if it is not intuitive 

Thanks again!

You received this message because you are subscribed to a topic in the Google Groups "Nextflow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nextflow/OPmwnHKMBiY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/F1471F95-2AEA-40B5-BD1D-F6D2F77C215F%40googlemail.com.