Limiting the number of local jobs

[rupert thorpe]

Sometimes I create quite allot of local jobs in gaffer,

because I do something like creating proxy plates, and I have 1 job per shot.

so for a 40 shot project I end up with 40 local jobs.

I like each shot getting its own job because then if there is an error its easy to see which shot has the problem, and an error on one shot doesn't stop others from rendering.

only issue is that they all run at the same time - and this can result in my machine running out of ram.

is there a way to limit how many local jobs run at the same time?

[John Haddon]

Hi Rupert,

There isn't any way of limiting the number of concurrent local jobs at the moment - we've always assumed that local jobs are mostly for testing before distributing the full workload to a farm. What you're doing makes complete sense though, and improved concurrency limits for local jobs would be a good feature.

Would you mind opening an issue for this on GitHub? Of particular interest would be how you want multiple jobs to be scheduled. Should one job run to completion before the next starts? Should we make some progress on all jobs at once? Should each task have some concept of how heavy it is, so for instance we could do multiple file copy tasks at once but only one render?

Cheers...

John

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/af1a1423-fb09-4877-8c21-a2be343a56aan%40googlegroups.com.

[Carlo Giesa]

Hi there!

If I remember well, in Tractor, there is a notion of available slots per render blade and each job can define the number of minimum slots available to be picked up by a blade. This way, we could control the number of parallel jobs in a very dynamic way depending on the render blade configuation and the type of job. I think that we had something between 1000 and 2000 slots per blade and small jobs took something between 20 to 100 slots and big render jobs something between 500 and 2000.

Greets,

Carlo

To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/CAB8pVgJSEy8v%2BMf69Yue0oxeqn9t9pybG7mhrBrmXwh0TpsTCw%40mail.gmail.com.