Hanging in parallel processes

[Hugues Fontenelle]

Hi

Using nextflow 0.17.0. It seems that sometimes, some processes hang.

Here is my (simplified) pipeline:

  A

/ | \

B C D

| | |

E F G

\ | /

  H

Running with queueSize = 1,

sometimes nextflow runs the processes in this order:

a) A,B,C,D,E,F,G,H and all is fine, the pipeline succeeds.

sometimes nextflow runs them in another order (more often):

b) A,B,E,C,F hangs. Processes E, F are finished but D never starts.

I can reproduce the hanging when nextflow chooses (a) (therefore not always)

The time it hangs varies between a few minutes (in rare cases it recovers) and forever (at least I get bored after a few hours).

I have experienced this behaviour with previous versions, and upgrading did not help.

My question is: what should I look at in order to debug? Alternatively, which information should I provide to get help?

All the best,

Hugues

[Paolo Di Tommaso]

It would help a lot if you could share the `.nextflow.log` file generated when the pipeline hung.

Cheers,

Paolo

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Paolo Di Tommaso]

However from your description it looks there something weird with your script. Do you have a public repository for it, or can you share it? 

p

On Mon, Jan 11, 2016 at 2:26 PM, Hugues Fontenelle <hugues.f...@gmail.com> wrote:

--

[Hugues Fontenelle]

Just for the world to know: I shared my pipeline with Paolo privately.

Also sent him two illustrative .nextflow.log, one for when the pipeline succeeds and the other when it fails.

I got some suggestions back and we're working on it now.

Thank you!

[Hugues Fontenelle]

I the description I wrote

b) A,B,E,C,F hangs. Processes E, F are finished but D never starts.

This is not correct. I should have written:

b) A,B,C,E,F hangs. Processes E, F are finished but D never starts.

[Hugues Fontenelle]

To summarise the private discussions:

At the core, there were two issues.

1) One nextflow exception not handled properly.

I believe this got addressed in https://github.com/nextflow-io/nextflow/commit/ec5df276af0ace3baadca9dea9ebbf83af87ec0e

2) More importantly, a racing condition due to csutom Groovy code. The lazymaps created by JsonSlurper are not thread safe, and we didn't really think of that in combination that we would be multithreaded when creating the process. We tried to read out the value we needed in the global scope instead, and that works every time. ( https://issues.apache.org/jira/browse/GROOVY-6934 ). Alternatively we used JsonSlurperClassic which is thread safe.

Thank you for the time and fruitful discussions.