scratch tmp folder in the specified directory should be removed ?

[Pau Carrio]

Hello,

In my nextflow configuration I use a scratch specific directory.
This folder never gets cleaned, so I am not sure to open a feature request  to remove $NXF_SCRATCH directory  or  get advice on where to do it inside my pipeline ?

Thanks
Pau

[Paolo Di Tommaso]

Hi, 

Nextflow uses the temp folder provided by the underlying resource manager which in principle should delete that directory. 

In any case you can use the `afterScript` directive to delete temp results when the task complete, using something similar in your configuration file: 

process {

  scratch = true

  afterScript = 'rm -rf *'

}

Hope it helps.

Cheers,

Paolo

 

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Pau Carrio Gaspar]

Hi Paolo,

thanks for your reply Paolo.
My point was not clear.
If using "scratch = true" then the /tmp ( or $TMPDIR ) is used and there is a possibility that the underlying resource manager takes care to deleted at some point, as you described. But control of this system behavior looks to me out of scope of a workflow developer.

If using "scratch = /some/scratch/directory " I do not observe ( at least in SGE) that mktemp created directories are deleted automatically after some time.

I adapted your snippet to

process {

  scratch = true

  afterScript = 'rm -rf $NXF_SCRATCH'

}

that does what i expected to happen.

In order to keep resources clean, and that scratch temp directories are not traceable (i.e. on trace file there is no link to that file, and accessing cluster resources might be tricky ) you might consider to delete them after the calculations.
It is also fine to do it via afterScript, but some reference in documentation could help.

Thanks

Pau

--
You received this message because you are subscribed to a topic in the Google Groups "Nextflow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nextflow/VtzFHEEhJIA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

Hi Pau, 

Yes, this is something that we are planning to improve.

https://github.com/nextflow-io/nextflow/issues/165

For now the `afterScript` could be a workaround. 

Cheers,

Paolo

[Pau Carrio Gaspar]

great that is on planning!