Nipype: Creating a custom template for PBS

[rickys]

Nipype users,

I'd like to use our cluster and PBS to parallelize using nipype. Everything works great, but the only thing is that our cluster only allows 4 jobs under different jobnames/ids per user to be submitted at a time (the rest are queued). So, all jobs are submitted but then only 4 run and the rest are queued. So this is not allowing me to take advantage of the parallelization. Normally, we have a script that uses pbsdsh to send individual tasks to reserved nodes, and this is how we use the cluster routinely.  

I was wondering if there is a way to overcome this? One thing I thought of might be to create a custom template and use pbsdsh to allocate tasks to reserved nodes. Does anyone have an example template of this or something similar? Or does anyone see an alternate solution that I am missing?

Thanks very much,

-Ricky

[Satrajit Ghosh]

hi ricky,

nipype does allow using a custom template, but doesn't allow changing the command. as long as you call qsub you can use it with a custom template with the current pbs plugin. otherwise you will need to write a new plugin (probably would look very much like the current pbs plugin).

cheers,

satra

--
 
 
 

[rickys]

Hi satra,

Right, thanks very much. I also found a few things to be very helpful, if it might be to others in the future:

1. passing in the qsub_args the -q quename argument allows me to submit to a queue on the cluster that allows more concurrent running jobs. We have a quickrun queue that allows more running jobs to be allowed. 

2. passing in plugin_args the max_jobs argument which limits the max number of jobs submitted.

But yes, I was running an analysis on a dataset with 135 subjs and this was submitting a lot of jobs. I still have asked the IT admin to give permission to allow more concurrent jobs!

Thanks again.

-Ricky