Cloud rendering with Gaffer

[William Eguienta]

Hi,

I'm having some issues manage scenes for cloud rendering, mainly because I need to setup rendernodes each time, as gaffer is not supported by any cloud renderfarm I know.

anyone is using cloud rendering with gaffer? What's your process ?
For now i'm working on sending an imagedisk to AWS nodes but it asks a lot of works (too me) for any update

thanks

[eric...@gmail.com]

I've used cloud rendering with Gaffer, specifically on AWS. I used Deadline which helps a LOT with setting up the infrastructure. Here's my general process:

1. Go through Deadline's setup process for getting AWS communicating with the on-premises network. You can replace this with your own manual setup if you aren't using Deadline. If you've already gotten through this part without throwing your computer out a window, bravo.

2. Start up an EC2 machine with a Linux template image. I used the AWS Linux machine image and it worked fine at the time. Deadline has a variety of pre-configured image combinations with the usual suspects like Arnold, Maya, Houdini pre-installed. One thing that made it a little more difficult is that image uses a fairly old version of glibc (2.26). I don't know what version the official Gaffer releases are built with. I know Ubuntu 16.04 is works (glibc 2.23).

3. Install all the needed software and configuration and do a small test to make sure it's working right. Utilities like 'scp' on Linux help with this. The way Deadline sets it up there are a couple relays you have to go through to get software onto the worker machine: on-premises workstation > AWS gateway server > EC2 worker.

4. When it's setup and working, make a snapshot through the AWS portal interface. This saves the machine state at that moment. Every time you start a new instance through the AWS portal, you choose that image and it will restore that state.

For changes I boot up a new EC2 instance, make any updates and then do a new snapshot.

I hope that helps!

- Eric

[Aaron Powell]

I can second Eric here as well - I set up my own AWS/Deadline render farm using his GafferDeadline plugin earlier this year. It took some trial and error, because you need to set up AWS exactly as they tell you to or else you'll have a bad day. But once I figured it out it all worked and made sense. I'm happy to help guide you through any specific steps if you need more help.

I'll parrot some of what Eric has already said, but my process was:

1. Install Deadline on a computer on your local network. It's easiest if you install the repository on the same computer you're submitting jobs from, but if you have a network share you use to store project files it can also be configured to use that. There are a ton of great tutorials out there on how to get Deadline set up, and their customer support team is pretty stellar as well. https://www.awsthinkbox.com/resources/deadline/aws-portal
2. Find an existing EC2 AMI with the exact version of Deadline you installed on your machine (this is important). You can start with this walkthrough to get started: https://docs.thinkboxsoftware.com/products/deadline/10.0/1_User%20Manual/manual/aws-custom-ami.html
3. Boot into your new EC2 instance and install Gaffer. If you're not used to installing on Linux, I'd be happy to talk you through that as well - there are some additional things you need to configure depending on the render engine your using as well.
4. Install Eric's GafferDeadline plugin on both your local machine and the EC2 instance. https://github.com/hypothetical-inc/GafferDeadline
5. Once you test to make sure Gaffer can render a script properly, create a new AMI and save it to your account.
6. Now you should be able to spin up new instances through the Deadline interface!

AWS infrastructure is expensive - I'd suggest only turning on an infrastructure/spot fleet when you actually need it, and turn it off after you're done. Otherwise your costs are going to skyrocket. Good luck!

Aaron

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/997453a7-64cb-44f9-b789-222d28df503an%40googlegroups.com.