How to use the Render Farm function?

[Jeff Wallin]

Someone got wind of the cartoon Ive been working on and offered me a really fucking insanely powerful (to me) set of servers in return to let him voice act. I THINK in total I have 46 1.6 Ghz cpus and something like 128 GB in ram in a set of 8 server nodes. That might be the wrong wording but basically 8 seperate computer boards in a server rack.

Anyways, either Im gonna end up doing alot of guesswork and making tutorial videos on how to use the render farm functions or preferably one of you nice folks knows about it and will share your knowledge. 

[gary.c...@cartoonsaloon.ie]

I'd also be very interested in this. Either info on how to use the render farm executable (tfarmcontroller.exe and tfarmserver.exe I believe) or even commands that can be called with OpenToonz.exe to get it to render from the command line.

[Dave Amies]

Hi Jeff,

Did you ever get this working? I have tried unsuccessfully over the last few days with the latest version, so I am interested to know if it worked at all and in which versions.

Dave.

[Rodney]

Are you guys able to use the Farm for rendering tasks on the Local machine?

I assume you can.

That'd be step one in the process of moving toward full farm rendering on external systems.

Also:

@Gary

Insight into command line usage can be gained by observing the instructions the Farm Render panel presents.

For example, in a quick test of rendering on my local machine I see the commands as follows:

tcomposer.exe "C:\OpenToonz 1.1 stuff\sandbox\scenes\building000.tnz" -tmsg 192.168.1.139 -o "Z:\Farm\a imageout 0000.png" -range 1 31 -step 1 -shrink 4 -multimedia 0 -nthreads single -maxtilesize none

We can therefore assume that tcomposer.exe is the batch executable.

If we run that same set of commands from a coomand line (making sure Windows knows where the executable is located we get the same rendered results:

C:\Program Files\OpenToonz 1.1\tcomposer.exe "C:\OpenToonz 1.1 stuff\sandbox\scenes\building000.tnz" -tmsg 192.168.1.139 -o "Z:\Farm\a imageout 0000.png" -range 1 31 -step 1 -shrink 4 -multimedia 0 -nthreads single -maxtilesize none

Bam.  Done.

In this way... noting the  we can even set up a batch file (text file) that will launch a set of renders for us.

Although, it'd probably be better to save those as Render Tasks because of the additional information saved in those files (in XML format).

Those are saved with a .tnzbat extension.

Added: Actually using the Render Tasks... saving and loading them... has a definite advantage because of the information stored in the document; Who submitted the task, when it was started, when completed, how long it took, etc.  It even makes note of failures and successes.

@Jeff/@Dave,

Now obviously that is on the local machine so we need to look deeper at the actual Render Farm executables.

I haven't delved into that because I don't have external machines for that purpose.

Most users likely don't require large renderfarms but for those that do it'll be good to publish/share that information.

@All

It's a little hard to know how far you've delved into use of the OpenToonz Farm so the more information you can provide the better.

[Dave Amies]

@Rodney

Thanks for the reply,

Does this mean that you got tfarmcontroller and tfarmserver working on your local machine? on which version of OT? I gather from your post you are using windows, which version. 

About me:

I am not an animator, I am an IT professional, my daughter wants to be an animator, she has known that for several years. She finished high school last year and has started animation at uni this year. she has already done a few stop motion animations (some you can see on her web site http://sunshineamigurumi.com) as well as some hand drawn (using a wacom tablet on a mac) animations in clip studio paint and flash. she hates flash and is a big fan of Studio Ghibli.

My primary goal with the the render farm is to learn how to set one up with several machines (preferably linux) so that I can support her with her future projects, I have several servers that I use for my own testing and it often sits idle, so sharing these cpu's with a render farm is no issue for me.

My other goal is to create Debian packages for OpenToonz, which will make life easier for people to install OpenToonz on many Linux distributions like Ubuntu. Initially I just want to get a single package working that has everything, then I will try to split them up so tfarmcontroller and tfarmserver are separate packages from the OpenToonz package and be run independently.

So hopefully now you can see why I'm keen to find out what configurations tfarmcontroller and tfarmserver are known to work, If I can replicate that, then I can try to make it work in more configurations and hopefully make it easier for everyone.

Dave.

[Dave Amies]

As a follow up, this is where I am at with tfarmcontroller and tfarmserver on linux:

_____________________________________________________________________________________________________________________________

/opt/opentoonz/bin/tfarmserver -console

Unable to start the Server

The directory specified as Local Root does not exist

_____________________________________________________________________________________________________________________________

/opt/opentoonz/bin/tfarmcontroller -console

Fri Mar 3 09:50:46 2017 INF:

ToonzFarm Controller 1.0

Unable to get FARMROOT environment variable

_____________________________________________________________________________________________________________________________

I have tried setting in 

- "$HOME/.config/OpenToonz/SystemVar.ini"

- "./OpenToonz_1.0.app/Contents/Resources/configfarmroot.txt" (this is what is in the code that is checked into the master tree for non windows)

I also tried re-compiling with "./OpenToonz_1.0.app/Contents/Resources/configfarmroot.txt" changed to "$HOME/.config/OpenToonz/SystemVar.ini" and similar variations.

On windows I found that FARMROOT is hard coded to the registry location "SOFTWARE\\OpenToonz\\OpenToonz\\1.0\\FARMROOT" I set this by copying the valuse from "SOFTWARE\\OpenToonz\\OpenToonz\\1.1\\FARMROOT" etc, and then got tfarmcontroller and tfarmserver running, however they didn't seem to talk to each other and OpenToonz still gave me a message about needing tfarmcontroller running on port 0 when I tried to select it, even though tfarmcontroller says its running on port 0.

I think on windows I might be missing some configuration information somewhere that tells each app about where to find the other?

Dave.

[Dave Amies]

I had some success, 

with help from procmon and reading the code I got it working on 1 machine on windows....

1) in the toonzfarm directory you need to create a config directory

2) in the config directory create a text file controller.txt, this just needs to contain the hostname of the machine running tfarmcontroller

3) restart tfarmcontroller (on the machine mentioned in controller.txt), it now comes up saying starting on port 1 not port 0

4) restart tfarmserver (it won't tell you its working yet)

5) select RenderFarm in OpenToonz, nothing happens (no error any more) so this is a good thing.

6) start a render task, now stuff happens:

 - tfarmcontroller displays Task 0 assigned to <tfarmserver hostname>

 - tfarmserver displays the tcomposer.exe message

 - tfarmserver displays Task completed at Fri Mar 3 10:45:46 2017

 - tfarmcontroller displays Task 0 completed on <tfarmserver hostname>

I will set up some windows vm's over the week end and see if I can build a farm, with the information above I expect it should be quite easy to get working on Windows, if so I will use that to have another crack at Linux and MacOS. I'm making progress at least :)

Dave.

[Dave Amies]

Good news, after much fiddling and using strace I think I have it working on linux, but can't really test it from here, will try tomorrow. fingers crossed.

[Dave Amies]

Good news, thanks to @konero, more info is available here: https://github.com/opentoonz/opentoonz/issues/1070

[Dave Amies]

I succeeded in getting the farm working with windows only so far, I will update the issue (https://github.com/opentoonz/opentoonz/issues/1070) as I progress.