server performance issue

[Xin Gong]

Dear Nextflow,

A performance issue occurred on the server for my nextflow pipeline. My pipeline was very simple, it ran bcftools against 1300 chr21 vcfs with -resume option. When about 200 samples had finished, I observed the vm for submitting job  was not responsive. The active java process for nextflow was using 171 threads. I had to stop my pipeline then VM was responsive.

here is the process:

process BCFtoolsIsec {

  cache true

  cpus 3

  executoe 'lsf'

  queue 'largemem'

  scratch '/data'

  stageInMode 'copy'

  stageOutMode 'move'

  clusterOptions '-M 5G -e./log.err -o ./log.out -R "select[avx2]"'

  

  input:

  set id, file(a_vcf), file (b_vcf) from map_ch

  script:

  """

  source activate /path/vcf-env/ 

  bcftools view -s ${id} $a_vcf > ${id}.a.vcf

  bcftools view -s ${id} $b_vcf > ${id}.b.vcf

  more commands

  """

}

Thank you for any help you can provide,

Xin

[Xin Gong]

Stephen Kelly's comments below:

Stephen Kelly @stevekm 10:08

@xg203

my pipeline had performance issue. The java process ate 171 threads. login node was frozen. I had to stop it so other people could use terminal.

This can happen with Nextflow since I believe it spawns a lot of threads both to monitor tasks and to copy files to publishdir. I think I used to use this to keep the number of file copy threads down;

// number of Nextflow process threads for moving files; should match Makefile 'submit' threads

filePorter.maxThreads = executor.$local.cpus * 3

where executor.$local.cpus = 8 or some smaller value that I passed in via CLI params

it can also help to limit your executor.queueSize, I think

[Xin Gong]

Stephen Kelly @stevekm 19:28

@xg203 @micans the place where I last used the filePorter config was here; https://github.com/NYU-Molecular-Pathology/NGS580-nf/blob/3ba2f970c3fbee56080ba60727f7bf43cb1be3b2/nextflow.config#L344

note that this is inside the nextflow.config file, not in the pipeline.nf file and not inside a process. This is a global setting for the entire pipeline. Also the example shown there is referencing the local executor which I had already set but if you do not have that set explicitly then I would think it would be better to just try something like filePorter.maxThreads = 8 or similar first just to see if it works. I dont have a setup at the moment to test it myself on the latest versions of Nextflow so I am not sure if anything has changed since I wrote that one there in the link

Stijn van Dongen @micans 19:35

@stevekm thanks! something for @xg203 to try.

[Xin Gong]

Dear Nextflow community,

Wherever I submit my nextflow pipeline, the max threads allowed is 13 in my lsf cluster environment. I need at least 25 threads for my pipeline. For example, after I submitted the pipeline I do the following check the total threads used for the pipeline:

$ps aux | grep nextflow #grep pid

$ps -o thcount <pid> 

I see it is 40 at beginning then stable at 25. How do i limit the max threads to 13? I tried to put 'filePoter.maxThreads = 13' in configure file, didn't work.

Best,

Xin  

[Xin Gong]

Deal All,

I am still have this problem. Would you please kindly help? The thread of submitted lsf job should be less than 13 according to policy. My nextflow has thread great than 300. the detail below:

1. in nextflow.config

executor{

  queueSize = 40

}

2. in .nf file

filePorter.maxThreads = 13  

//error, filePorter not found. I am using nextflow-20.10.0-all

3. run nextflow

export NXF_OPTS="-Xms1G -Xmx4G" && bsub -R rusage[mem=4000] nextflow run…

4. Resource usage collected using command bsub -l pid

The CPU time used is 507 seconds.

MEM: 968 Mbytes;  SWAP: 0 Mbytes;  NTHREAD: 349

PGID: 32976;  PIDs: 2290 32976 32977 32979 33034

How to limit the threads below 13?

Best regards,

Xin

[Paolo Di Tommaso]

Try adding to NXF_OPTS the option `-Dnxf.pool.type=sync`

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/f0aac266-b687-445f-8a8c-84f95235e01bn%40googlegroups.com.

[Xin Gong]

Thank you Paolo !

I had split the pipeline to several pipelines. I am running the first pipeline, the number of thread is about 45.  

Then I tried "nextflow -Dnxf.pool.type=sync run pipeline.nf" and  "nextflow -Dnxf.pool.type=sync -Dnxf.pool.maxThreads=12 run pipeline.nf" the number of thread is still around 45

Did I do anything wrong?

Best,

Xin

[Xin Gong]

Hi Paolo,

I found your comments on gitter below:

"

you can control with the system properties nxf.pool.type and nxf.pool.maxThreads

refer to this code

https://github.com/nextflow-io/nextflow/blob/master/src/main/groovy/nextflow/util/CustomPoolFactory.groovy#L36-L68

"

the link seems broken.

Best,

Xin

[Alan Hoyle]

https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/util/CustomPoolFactory.groovy

Looks like Paolo forgot the "modules/nextflow" after "master"

--
  -  Alan Hoyle  -  al...@alanhoyle.com  -  http://www.alanhoyle.com/  -

To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/a9390fe0-f9c5-4bd6-be33-386d8811f28bn%40googlegroups.com.

[Paolo Di Tommaso]

Try to use maxThred 1 or 2, but I have no idea if it can impact the correct execution or solve your problem. 

p

To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/6fe877f6-5b7e-47eb-b83c-db14a26c4126n%40googlegroups.com.