Running out of /tmp space

[Carlos Guzman]

I've been getting this relatively often now that i'm running larger samples through my pipeline. Originally I went ahead and added a config file with

NXF_TEMP = "$pwd"

so that I could use the larger disk for temporary file storage and it worked fine. However, for some reason, i've now been running into the problem that my /tmp is still filling up and I have no idea why this is. This loses hours of progress every single time because my pipeline just stops dead in its tracks. Should I set a maximum queue size? Do I just run less samples through the pipeline at once?

In this case i'm only running 13 samples and still running into the problem.

[Carlos Guzman]

Adding to this .. the -qs option does not seem to work. I set it to 10 and it's still running 13 processes.

[Paolo Di Tommaso]

This looks unusual. You should try to understand what tool is using all that space. 

Running `du -h -d 1` in the /tmp directory can help to troubleshoot the problem.

Cheers,

Paolo

On Wed, Sep 14, 2016 at 5:55 PM, Carlos Guzman <cguzma...@gmail.com> wrote:

Adding to this .. the -qs option does not seem to work. I set it to 10 and it's still running 13 processes.

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Carlos Guzman]

It appears that several tools are running and creating too many temporary files in the /tmp folder. I've fixed the ones that have options to change their temporary file storage, but there are several others that by default store temp files in /tmp and offer no way to change that. One solution was to change the TMPDIR env variable itself.

Is it possible to change the $TMPDIR env variable from within a nextflow script?

Would using the scratch '$NFX_TEMP' be a possible solution?

On Wednesday, September 14, 2016 at 12:12:28 PM UTC-5, Paolo Di Tommaso wrote:

This looks unusual. You should try to understand what tool is using all that space. 

Running `du -h -d 1` in the /tmp directory can help to troubleshoot the problem.

Cheers,

Paolo

On Wed, Sep 14, 2016 at 5:55 PM, Carlos Guzman <cguzma...@gmail.com> wrote:

Adding to this .. the -qs option does not seem to work. I set it to 10 and it's still running 13 processes.

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

I would try use a different TMPDIR for each task executed in your pipeline, and clean it up when the task complete. 

For example: 

export TMPDIR=$(mktemp -d)

<your script>

rm -rf $TMPDIR 

 Eventually you may use beforeScript and afterScript directives for that. 

p

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Carlos Guzman]

Went ahead and did this to every process:

    beforeScript 'export TMPDIR=$(mktemp -d)'
    afterScript 'rm -rf $TMPDIR'

But i'm still seeing significant amounts of /tmp being filled up 'tmp' directories now though the pipeline worked this time around. Could you explain what exactly your code is doing? I understand the majority, I just have very little experience with the mktemp command.

[Paolo Di Tommaso]

Hi, 

mktemp -d creates a unique directory, thus you are allocation a new temp directory for each task execution, that is deleted when it completes. 

Cheers,
Paolo

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Carlos Guzman]

Is it possible to modify the code above to make a TMPDIR outside of the /tmp directory? Perhaps create a directory for temporary file in the present working directory?

[Paolo Di Tommaso]

Yes, check the manual for the mktemp Linux tool. 

I don't want to be impolite but there different options depending the actual Linux distribution. 

Cheers,
Paolo

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.