limiting number of jobs run by 'jug execute'

[my....@gmail.com]

Hi Luis,

Thanks a lot for writing jug and making it freely available.  I've been
using it recently to parallelize some python code on my cluster (I was
using PP before but it was a bit finiky).  I'm running into a problem
which is that I have an 8 hour limit to run jobs on my cluster and so
I'm worried that when I run 'jug execute' it will get killed in the
middle of a job.  Each of my jobs takes less than 8 hours but I have
lots of jobs and each instance of 'jug execute' will keep picking up new
jobs until its killed by the cluster controller.  This would leave the
last job in a half-finished state. Is there any way to control the
number of jobs a 'jug execute' instance runs?

If not and if you have a bit of time would you mind recommending a
simple work-around or code hack to the jug code itself?  I'd appreciate
any help.

thanks a lot

[Luis Pedro Coelho]

Thanks for your kind words.

I know your problem because I've dealt with it exactly.

I have a couple of hacks to jug to deal with this. I don't really have
yet a clean way to do it, so I didn't make it part of the master version.

I attach two patches to jug/jug.py (against current github). The first
will detect whether a file called jug_exit_trigger.txt and exit whenever
a task has just finished. The second will finish after running just 1
task (but you can edit it to be a number N of tasks by changing the if
condition). Once you've seen these, you can probably think up
interesting conditions that fit you.

Then I run a little script in bash to submit new jobs whenever an old
one exits, monitoring the queue.

*

Perhaps seeing that others have the exact same problem will make me
clean up this code and put it into jug proper.

Luis

--
Luis Pedro Coelho | EMBL | http://luispedro.org

[Sven Augustin]

Hi!

Actually I do have to same problem regularly. I "solve" it by guessing how many jobs I can run in one go, hope that my guess was good enough and resubmit when done.

So a better solution sounds great to me!

Best!
Sven

--
You received this message because you are subscribed to the Google Groups "jug-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jug-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.