nextflow doesn't complete and is working on tasks very slowly

[Pranathi Vemuri]

Hi

I am trying to run a job with a lot of files about 50K. It is getting stuck and does not do anything, once I ctrl+c it and restart it it seems like it is doing faster and completes a few (about 100s of files). But a few hours later it is just stuck until the next day. I am not sure what's happening. This is for the kmermaid pipeline on nf-core and data and computation is on our local servers

Attaching nextflow log for you review.

Thank you,

Pranathi

[Jason Steen]

your submitting all of these to local.  are you sure you are overloading the server you are working on causing it to crash/run slowly?  what does `top` say is actually running.  have you tried setting "queueSize" in your config file to something that matches the number of CPUS you have.  I tend to use queueSize = 32 if i'm submitting jobs to local.

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/336e4ba4-9576-4527-97b0-b2761040338an%40googlegroups.com.

[Pranathi Vemuri]

Hi, 

Thanks a lot for your response and sorry for the delay in mine. I was not setting queueSize to 32 seems like the default is 100. would you mind explaining the difference between cpus and queueSize? I assume from the definitions here https://www.nextflow.io/docs/latest/config.html#scope-executor they are how many cpus are available vs how many of the tasks are on each of those cpus in parallel. could you please explain?  below is my nextflow.config file

executor {

         cpus = 32

         time = 99999.h

}

process {

  withName: sourmash_compute_sketch_bam {

      cpus = 1

      time = 48.h

      }

  withName: translate {

      cpus = 1

      time = 488.h

      }

[Jason Steen]

queueSize is the total number of jobs that nextflow will try and run in parallel. if you leave it default, nextflow will run 100 at a time, meaning that it is trying to assign 100cpus worth of work to do at the same time. if your machine only has 32 cores, it will bog down and run very slowly.

To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/75836213-9308-4d95-93ee-5752885777d3n%40googlegroups.com.

[Pranathi Vemuri]

Hi, 

Thanks again! wondering if anyone has seen any difference in execution time when a command is run using a process on nextflow vs without nextflow. 

For this pipeline on my end, when I tested a command to run by itself it took 10 minutes but when I checked the time for the process on the same single file from execution_trace.txt it took 8 days. wondering why that would happen?

Could you please advise, thank you!

Best,

Pranathi

[Abhinav Sharma]

Hi Pranathi,

 

I think that this might be because of the difference between “CPU time” and “Wall time”. PFA a screenshot for a run of nf-core/rnaseq-v2

 

 

Best,

Abhinav

[Abhinav Sharma]

Whoops, this one here :)