Automatically switching max Clients / Abort after frame for wrangling

[Matthias Backmann]

Hey Knight,

Our small farm has some issues during the day, which oftentimes needs a lot of render wrangling:

If artists send some heavy fx/cg stuff in the morning, due to longer project loading times, they often set it to 1-15 frames/machine. This leads to machines rendering up to 15 frames with long render times >0.5h - ergo those are blocked the whole day, if not manually stopped (if I understand it correctly, they will only switch after finishing their current assignment and look for higher prio jobs). And because the max client is set to maximum, this sometimes occupies the whole farm.

Right now, I can think of two ways to ease this issue and not affect the renders too much (e.g. abort after time could stop long fx renders completely before finishing each frame):

- A script, which switches the max client on cg-jobs with lower prio during working hours to a lower number and gives those max clients during the night again

- A way to check for high prio renders after each frame and abort then (without losing render time), otherwise continue and avoid reloading the whole Maya/Houdini-Scene.

- Perhaps additonally a "wrangle shortcut" which does both: Set the max clients down to a lower number and aborts all machines above that number, that finish a frame and could now switch to the next job.

I know, that sounds pretty complicated, but with a small farm, it's not possible to really split between comp and cg pools, because every machine counts. So I want to avoid blocking machines completely for the CG artists, just to have some quick comp jobs running through.

Thanks for your help in advance

Matthias

[RR, Schoenberger]

Hi

First some question.
Is it possible to know if the job has a long render time based on these three settings

1. Project Name
2. Render type (3d, comp, archive)
3. Layer name ( „beauty“, „AOV“)

 

E.g.
Regarding „ProjectY“ only:

All 3D render jobs with layer name „SubsurfaceScattering“ or „Volume“ have high render times.

All other 3D jobs have low render times.
And Comp jobs have tini render times.

 

 

 

>just to have some quick comp jobs running through.

Ok, if the issue is just the comp departement

and not that some 3D jobs block 3D jobs of an other user, then there are some solutions build in:

1. Multiple job threads.
   Setup your machines to start 2 different jobs (rrConfig, per client config).
   Then set 3D for the first thread, comp for the second.
   Of course your machines require to have enough RAM to run 3D and comp at the same time.
   But it work well on other sides, 3D jobs take mostly CPU and RAM and Comp jobs TCP traffic.
   
   
2. Average clients between render apps.
   So Comp and 3D get the same ammount of clients (IF they have jobs).
   rrConfig, tab Jobs.
   Choose Average in group 4 or 5.
   In the middle column
1. disable the first Abort if job is a received
2. enable the the second abort and change it to every 2(?) hours.
3. Enable third Abort after Frame

 

 

All other build in solutions will abort directly.

(job „abort low prio jobs“ or privileged clients)

 

 

 

 

 

About your requested functionality:

It would be possible to implement it as a python script.

Either as a right-click in the job table in rrControl or as an external script that could run 24/7 to automatically change the jobs.
Please see SDK\External\Python\ or \plugins\control_job\once\ for a starting point.

Once you know more, we can get into details.

 

PS:

No matter which way you choose, I recommend to change the file

RR\sub\cfg_user\userInformation.htm

to know the current state.

Just use rrControl menu Main/Set User Information to create the file to see how it works.

 

 

regards,
Holger Schönberger

 

Please use the rrKnights Tavern
or our support system for new questions.

[Matthias Backmann]

Thanks for the quick response.

Right now, one can assume the longer render time from the used application - So Maya+Arnold or Houdini+Mantra/Arnold in comparison to Nuke (We did some tests with copycat, hopefully that won't change too much). The rest is done by prioritizing. So if a shot is not due that day, Prio is 10 or 30 and those shots should then be the ones who have to be toned down during the day.

Multiple job threads would be a solution, but there have been some issues on several machines, so we use single threads on most blades. Perhaps we should have another look into that. Otherwise I'll have a look into the config for averaging the machines, as well as into the SDK to perhaps script a few useful things.

[RR, Schoenberger]

Hi

> Nuke (We did some tests with copycat, hopefully that won't change too much).

Do you mean you want to copy Nuke to the local machine?

Please see
"Shared Exe" installation

http://www.royalrender.de/help/Nuke.html

[Matthias Backmann]

No, the CopyCat-Node in Nuke 13 for Machine Learning roto and stuff. This one could perhaps increase render/work times for nuke jobs. But we might solve this with a seperate pool.

[RR, Schoenberger]

 

Ah, sorry, mistaken with with some other copy tool.

I remember.

Yes, it takes a long time and only one machine can compute all frames.
And afaik it requires a lot of GPU  power, perhaps the artist should be logged out.

[Matthias Backmann]

Hey,

I know talked to our IT-guys about your suggestions to average between render apps. Right now, our priorisation is set up like this:

So right now we average between all projects, which might be something we can change. Oftentimes there are only a few shows really running on higher prio and so it makes less sense to give projects, which are still in RnD or already renderings low prio breakdowns, the same amount of machines. (And perhaps, if I can dive a bit into the scripting part, I might be able to setup a connection of the config with shotgrid, so that producers can set a priority in there, which directly reflects in the exclusive priority and therefore gives those shows quick access to a part of the farm without long waiting times)

I still have a question about the First Render order modifier "Av. level A: Abort a few clients for averaging if a job is received":

Is this one directly aborting jobs (resulting in broken frames) if e.g. the full farm is only rendering jobs for one project and then a job for another show drops in? I had the feeling, that this wasn't the case. Or is it only aborting, if those jobs are low prio (<20)?

And how does the "Abort after current frame" work? Will this take into account the 2nd step (abort after 8 hours)? And does this work with Maya/Arnold or Houdini/Arnold-Renderings?

Sorry for asking, but in the helpfiles I could only find the example for this second checkbox and so am unsure how these three work together.

Thanks again

Matthias

[Matthias Backmann]

And one more addition:

Another way I thought of may be to give the nuke renderapp an exclusive priority for a part of the farm (say a third of all machines). Thereby, all machines are still available for all jobs, but Nuke gets a direct boost. But this is only an option, if it doesn't lead to a direct abort on all other renderapp-machines, but to an "abort after frame".

[RR, Schoenberger]

Hi

Please note that you have two average options.
Prio level 4 and 5.
So each project can get the same ammount and inside each project each render app.

Or the other way round if you do not use that many render  time for comp jobs.

So each render app gets same ammount and inside each render app each project.

> sense to give projects… low prio breakdowns…the same amount of machines
Note that you have set that jobs with priority lower than 19 are not taken into account for averaging.
So these jobs get clients AFTER all other projects.
And jobs with prio higher than 90 as well.

 

>exclusive priority
That is an option as well beside prio higher than 90.

 

 

>Is this one directly aborting jobs (resulting in broken frames)

It should be used for the „average every x hours“ only.
We could change that.

> I had the feeling, that this wasn't the case

Perhaps there is a bug in this functionality.
I will do some tests.
Averaging itself was tested quite often, but probably not these options.

 

> And how does the "Abort after current frame" work?
>Will this take into account the 2nd step (abort after 8 hours)?
>And does this work with Maya/Arnold or Houdini/Arnold-Renderings?

It should work with almost all renderings.
Afaik it is required to have „Local Render Out“  enabled for the job.

 

> Another way I thought of may be to give the nuke renderapp an exclusive priority for a part of the farm
Yes, you can either set Exclusive Priority on x machines.
Or if you hvave machine with 128 cores and some with 32 Cores,

it might be better to use the client „Preferred for“ (Nuke) setting on the 32 Core machines.

> lead  to an "abort after frame"
This is just a priority boost, no „abort after frame“

 

 

 

Back to old topic

> assume the longer render time from the used application - So Maya+Arnold or Houdini+Mantra/Arnold

Note that you can set different sequence divide for Comp and 3D.
Or even different per renderer Maya/Houdini.

 

 

PS: Working on a new feature to use the last 50 jobs of a project to „guess“ the right sequence divide for new jobs. But very early alpha stage.

 

 

 

regards,
Holger Schönberger

 

Please use the rrKnights Tavern
or our support system for new questions.

From: rrkn...@googlegroups.com <rrkn...@googlegroups.com> On Behalf Of Matthias Backmann
Sent: Friday, September 16, 2022 10:41 AM
To: Royal Render Knights Tavern <rrKn...@googlegroups.com>
Subject: Re: [rrKnights] Automatically switching max Clients / Abort after frame for wrangling

 

Hey,

I know talked to our IT-guys about your suggestions to average between render apps. Right now, our priorisation is set up like this:

--
If you reply, the message is send to the user group which is sufficient and desired.
("Reply All" might send the message twice to the last author which is not required.
Replying "in private" prevents other users to see the answers and might not be seen by the single receiver if he has email rules in place.
If you want to talk to us in private, please use support@RoyalRender instead)
---
You received this message because you are subscribed to the Google Groups "Royal Render Knights Tavern" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rrKnights+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rrKnights/91d612bc-2520-44fa-a772-12281f1686d4n%40googlegroups.com.

[RR, Schoenberger]

Hi

 

I have found an issue with the average abort options.

It will be fixed for the next update.

[RR, Schoenberger]

Hi

 

„Abort after job was received“ is fixed, an update will be released soon.

Note that the „Abort After Frame“ setting will affect the „Abort after job was received“ only.