Rez on Deadline Farm

[Tony Dorfmeister]

Cheerio everyone,

I'm currently tinkering with rez and trying to incorporate it in our Shotgun-based pipeline. Our studio is using windows throughout.
I can already launch DCC's, maya for now, via Shotgun Desktop in a rez resolved environment. 

Now I want to re-resolve that same environment on any render node that picks up the job. 

The idea is to store the resolved packages in the Deadline job's custom key-value store and read those packages as an argument for rez-env on a render node. This is already working however, I'm hitting a wall when executing that commmand.

How are you guys doing this? Are there any caveats to be aware of?

Here's what I did until now:

- inject a PreJobScript that adds a new callback to the plugins ModifyCommandLineCallback. This callback returns the path to a custom batch script which shall take care of running rez

- the batch script gets executed before the render starts. It configures PATH in order to run rez from the network installation of rez

- when the batch script gets executed the job fails with the following error message:

WARNING: Cannot write stdin to process because it is no longer runnning.

WARNING: Monitored managed process MayaBatch is no longer running

Any help or pushes in the right direction are much appreciated.

Cheers,

Tony

[Nico Van den Bosch]

While I can’t help you with your particular implementation Tony, lemme share our studio’s approach to deadline + rez integration.    So far we’ve successfully sidestepped the need to invoke rez wrappers directly on the farm.   Instead, we’ve landed on doing all of  rez resolves locally.   Upon submission to deadline, we have a lightweight submission event plugin that checks the local environment  for $REZ_USED_RESOLVE,  gathers all the environment variables that are set by this resolve, and copies them into the deadline job’s environment property.  Deadline automatically handles appending these to the environment at render time.    Granted, this approach assumes a fairly homogenous studio infrastructure without workstations and render nodes having different platform dependencies.

Best of luck,

Nico

[Tony Dorfmeister]

Hey Nico,

thanks for sharing these really helpful insights!
I'll definately try this out now.

Out of curiosity... did you try to run rez on the render nodes or did you decide for the local approach right from the get go? 

[Nico Van den Bosch]

We never attempted to run rez on the render nodes, but since our current implementation is still relatively fresh (<6 mos),  I can’t confidently say we’ll never have occasion to do so.  For now, the submission plugin implementation was very simple and got our deadline jobs up and running in rez environments without issue.

Cheers

N

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/3c524fae-f5d5-4daf-b6e9-cbe8015eb9d9o%40googlegroups.com.

[Dani Asztalos]

Hi,

although I can't help either with the error, I can describe our setup. 

We have an "environment system" built on top of rez, you can think of an environment as a list of requested packages, and you can tell the system to resolve an environment for Maya, for XYZ project. It then calls rez in the background with the required packages.

1. The environment (list of the requested packages) is passed in an ExtraInfoKeyValue
2. We have a GlobalJobPreload script that:
1. reads this key and resolves the environment
2. runs a python script in this resolved environment that prints all the environment variables to the stdout in a json format, so the full rez command looks like this: rez-env <requested_packages> -- python -m enviroment_printer_script
3. then it parses this json and sets all the environment variables with deadline_plugin.SetProcessEnvironmentVariable

Dani

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/BC3D78EE-AF17-4376-AC17-760B69748827%40gmail.com.

[Rob Rob Rob Rob Rob]

We serialize our environment, and send it to the farm, that way the job executes in the exact environment people are expecting. (the one it was produced in) 

[Pablo Giménez]

What we do is to pass the packages requested by the user, $REZ_USER_REQUEST, and pass it as an EnvironmentKey in Deadline, then workers can just execute something like:

rez env $REZ_USER_REQUEST

This is a simplified explanation but the idea is pretty much that.

Our workers run in the linux farm while workstations can be a mix of linux and windows, so it is better for us to run rez env, some packages resolve different depending on the OS.

Cheers

[Rob Rob Rob Rob Rob]

It might be worth using time-stamping if your not specifying the exact resolve to use. Not knowing what code a farm job will execute could be a major source of chaos and confusion.

[tony.dor...@gmail.com]

Hey... sorry for getting back at you guzs that late. First of all big thanks to all of you. Didn't expect to receive so many good tips!
I finally had the time to get back to this again and I managed to get it working by just resolving the env as a pre job callback and inject those env vars to the deadline job. Works like a charm :)

Thanks again! :)

[Pablo Giménez]

We are using Pipeline Bootstraping at the moment, works fine as well: 

https://docs.thinkboxsoftware.com/products/deadline/9.0/1_User%20Manual/manual/application-plugins.html#pipeline-bootstrapping