restricted number of jobs being submitted to resource manager

[ben curran]

Is there an option somewhere that restricts the number of jobs submitted to a resource manager?

I have a working pipeline that submits jobs to a slurm resource manager. At the moment, it has 14 jobs but is only submitting 2 at a time to slurm.

Cheers

Ben.

[Simon]

On Wednesday, January 30, 2019 at 8:59:08 AM UTC+11, ben curran wrote:

Is there an option somewhere that restricts the number of jobs submitted to a resource manager?

Yes - by default, Bpipe restricts how many jobs it will submit. It's designed mainly as a protection against accidentally overwhelming the system Bpipe is running on. So it will not use more total threads than it thinks the computer it is running on has. For example, if it is running on a computer with 8 cores, it might limit the total cores to that. If your jobs both used for threads / cores it might only submit two jobs.

You can raise that limit by adding the -n flag:

bpipe run -n 32 pipeline.groovy ....

Note that this can also be controlled in some other ways (for example, setting concurrency in a bpipe.config file), so if that doesn't explain it you could dig a bit deeper into those other options.

Hope this helps & let me know if it's not the answer in your case!

Cheers,

Simon

[ben curran]

I got it working by setting the concurrency in the bpipe.config. I've got the run command in a pipeline file, calling the -n flag from within the file didn't seem to work. All good though, it works now.

Cheers

Ben.