Clarification on publishDir usage with path: does it overwrite the directory, or the files inside?

[Matteo Schiavinato]

I am running a process that has a whole directory as output, which I declare as "path".

While the output directory name remains the same every iteration, thus being always found and saved in the publishDir, the content of the directory is not constant. In fact, the directory contains an arbitrary number of files named "pool_1", "pool_2", ... , "pool_n" which are generated by the program, but the numbering depends on the order of completion. Hence, pool_1 could be two different pools in two different runs.

When you declare a path as output, how does publishDir work? Does it overwrite the whole directory each iteration, or does it check if it exists and potentially overwrite the files inside according to their name?

[Paolo Di Tommaso]

A good approach is saving each of them into separate directories. Have a look here 

https://nextflow-io.github.io/patterns/index.html#_store_process_outputs

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/994fe9eb-0f47-4351-a877-00a49aab6b6en%40googlegroups.com.

[Matteo Schiavinato]

Not sure I get it: do you mean having a publishDir that has a mnemonic code attached to it, depending on the run? Because my problem is that every time I rerun the pipeline I get a new set of pools, but "pool_1" of run #1 is not the same as "pool_1" in run #2 (pools are generated independently, and there's a stochastic step involved so two runs are never identical). If I get it right, you're suggesting to attach a code to the publishDir (e.g. the run mnemonic, let's say "happy_fermi") so that the output directory can be pinpointed to the specific run?