Mysterious missing-output-files/UnixPath issue

[gnet...@gmail.com]

Hello Team Nextflow,

I am running into a recurrent  'Missing output files' issue when names and locations of output files are not explicitly stated in the exec code block of the script directive.

I have included two Nexflow code snippets - one that works and another that reproduces the phenomenon.

The following one works as expected and is straightforward, I believe, since the location and name and pattern of output files is clearly stated in the 'echo' line as hello_${lane}.txt

lanes = Channel.from(1..4)
barDir = '/mnt/home/ngebremedhin/outputs/Nextflow/Barcodes'


process processA {


        publishDir "${barDir}/fastq"


        input:
        val lane from lanes
        output:
        file("hello_${lane}.txt") into welcomeFiles


        script:
        """
        echo "Hello">hello_${lane}.txt
        """


}


process processB {




        publishDir "${barDir}/fastq"


        input:
        file x  from welcomeFiles


        output:
        file "${x}.pdf" into finalOut


        script:
        """
        mv ${x} ${x}.pdf
        """
}

However, if I changed the script directive in processA to use a bash script that writes multiple output files in sub-directories the script itself creates, I run into a 'Missing output files `hello_1.txt` expected by processA' error.

The files have, in fact, been created in the publishDir and exit code is 0.

Caused by:
  Missing output file(s) `hello_1.txt` expected by process `processA (1)`


Command executed:


  /mnt/home/ngebremedhin/novaseq_pipe/create_files.sh 1 /mnt/home/ngebremedhin/novaseq_pipe/work/fastq


Command exit status:
  0


Command output:
  (empty)

Problematic:

lanes = Channel.from(1..4)
barDir = '/mnt/home/ngebremedhin/outputs/Nextflow/Barcodes'


process processA {


 publishDir "${barDir}/fastq/${lane}", mode: 'copy'


 input:
 val lane from lanes
 output:
 file("hello_${lane}.txt") into welcomeFiles


 script:
 println workDir
 """
 /mnt/home/ngebremedhin/create_files.sh ${lane} ${barDir}/fastq
 """


}


process processB {




 publishDir "${barDir}/fastq"


 input:
 file x  from welcomeFiles


 output:
 file "${x}.pdf" into finalOut


 script:
 """
 mv ${x} ${x}.pdf
 """
}

Where create_files.sh is something as simple as: 

#!/bin/bash


for i in {1..10};
do
 mkdir -p ${2}/${1}
 echo "Hello">${2}/${1}/hello_${1}_${i}.txt;
done

Thank you,

Net

[Paolo Di Tommaso]

That's expected because NF processes can only write in the task working directory i.e. you should not use absolute paths.

Hope it helps.

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[gnet...@gmail.com]

Thank you Paolo for the swift reply!

I am afraid it is still not clear to me.  I changed the output directory of the script to be relative to the workDir but still got  a 'File X is out of the scope of process working dir Z' error

/mnt/home/ngebremedhin/novaseq_pipe/create_files.sh ${lane} ${workDir}/../../outputs/Nextflow/Barcodes/fastq

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

The `workDir` variable refers to the workflow scratch directory which different from the task work directory. 

A core concept of NF workflows is that task executions are isolated each other, each task run in its own working directory that is assigned by the framework and outputs can only be produced that directory. 

Therefore how to write in that directory, just using a relative path instead of an absolute one i.e. 

touch foo.txt

instead of 

touch /some/path/foo.txt

If your tool requires an absolute path use the $PWD environment variabile (don't forget to escape the $), for example: 

process foo {

  output: 

  file 'foo.txt' into foo_ch 

  """

  touch \$PWD/foo.txt

  """

}

Hope it helps 

p