Using more than one node with Nextflow executor="local" (independent jobs)

[Matteo Schiavinato]

I often use the "local" executor in my pipelines since I don't often require more than a few tens of cores for them to run efficiently.

However, I recently embarked in developing one that could (potentially) run more than 30k independent jobs at the same time. 

Each of these jobs is very cheap: 1 cpu, little memory. The pipeline works fine if I run it with the "local" executor, but my nodes have only up to 48 CPUs, so it takes forever.

I am wondering if there is a way to trick nextflow into thinking that 4 nodes are a single node where to spawn jobs "locally". I am not an expert when it comes to mpi & co, so I would welcome any direction / tip.

Matteo

[Alan Hoyle]

Nextflow, by default, will only submit 100 jobs.  

https://www.nextflow.io/docs/latest/config.html#scope-executor

executor.queueSize = 100

--
  -  Alan Hoyle  -  al...@alanhoyle.com  -  http://www.alanhoyle.com/  -

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/c3ef4157-f318-4430-b2c4-7eec9ccd59ffn%40googlegroups.com.