[Maya-Python] Setting up a virtual renderfarm

[Marcus Ottosson]

Hi all,

I’m looking to set up a dozen virtual computers to act as my renderfarm whilst developing tools for task distribution.

I’m having a look at http://docker.io, does anyone have experience with it or something similar/better suited?

Each worker will essentially run time.sleep(10) commands to simulate workload, and I’d need them all to run on a single physical workstation and I’ll need at least 10-20, ideally 500-1000 of them running at once.

Thanks,
Marcus

​

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

I was playing with docker earlier this week since I also want to integrate it at work. Although the only thing I see it being related to for your question is being able to distribute a ready made container that will just do what you want when it starts up. Is that what your question is about?

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOB-Jq-EHYmiw%2Bnyn3_knxd29rh5s%2BhJ3VyDn6LJMULatQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

I would assume so, not sure yet of absolute requirements, I'm expecting to find out as I go. But I would imagine the requirements would be similar to workers in any traditional renderfarm.

Is docker suited for something like that you think? I'm interested in it having its own dependencies too and possibly reserved performance from the host system. Ultimately coming as close as possible to working against an actual farm of computers.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0nU3OJTkPXYX9OCd0t8mAg1Ee4gL9T7ZLOD2rECYavQg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

Docker looks well suited for deploying an application that is already set up in its own environment by a developer. I like the idea because it moves the build step over into the developer's responsibility. So like, if I set up my docker container exactly how I want, and test it, and it works, then I know it is going to work when it gets deployed to some other machine. This would replace the process where you have an application and you have to get someone to run your build script to provision the machine and install libraries and hopefully avoid conflicts.

You also get the benefit of cgroups so you can control resource usage. I've been working with them as well in my own applications. 

It does require a bit of extra work for OSX and Windows though, since you first have to install a virtualbox/vagrant thing and then run docker through that. 

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBsvA3ykDs0Gf8aQqXhKxjFMSuYgwd-DG5N4aTdTAbiyg%40mail.gmail.com.

[Marcus Ottosson]

This would replace the process where you have an application and you have to get someone to run your build script to provision the machine and install libraries and hopefully avoid conflicts.

This sounds awfully close to being an alternative to Rez. If Rez configures a state completely out of thin air, with only the required dependencies, then Docker could instead be used to set-up pre-configured, more course-grained states that could run in parallel on a single worker, especially since it seems capable of sharing resources across states (or “containers”). Thoughts?

You also get the benefit of cgroups so you can control resource usage. I’ve been working with them as well in my own applications.

That’s cool, haven’t heard of cgroups before, but it sounds like it’d deal with partitioning resources then.

It does require a bit of extra work for OSX and Windows though, since you first have to install a virtualbox/vagrant thing and then run docker through that.

Yeah, it seems rather specific to Linux. I’m having a go in Ubuntu and so far haven’t experienced any hiccups.

​

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA11d1%3Du9p3b4UMUZZ3ix9oYX6CGcJ4t%3DHL8Jd3VcwKukA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

On Jun 2, 2014 3:37 AM, "Marcus Ottosson" <konstr...@gmail.com> wrote:
>>
>> This would replace the process where you have an application and you have to get someone to run your build script to provision the machine and install libraries and hopefully avoid conflicts.
>
> This sounds awfully close to being an alternative to Rez. If Rez configures a state completely out of thin air, with only the required dependencies, then Docker could instead be used to set-up pre-configured, more course-grained states that could run in parallel on a single worker, especially since it seems capable of sharing resources across states (or “containers”). Thoughts?
>>

It isn't anything like Rez other than maybe some very high level features. Rez is managing packages from an environment standpoint.
Docker uses kernel features like Cgroups and Linux containers to give your container it's own process space, ethernet device, root environment, etc. You have to explicitly map a virtual resource to give the container access to the host filesystem. And you have to map ports out of the container to expose them to the host. This gives you the ability to build the container and not even consider if it will have library and dependency conflicts with other things on the system. Because there isn't a need for dependency resolution. It doesn't have to play with other shared libraries and deps.

I would make the analogy of saying that Res makes applications play nice within the same environment on the same system. Docker tells your application to have fun running off in its own world as a god with 40 virgins.

>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

>>>>> To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOB-Jq-EHYmiw%2Bnyn3_knxd29rh5s%2BhJ3VyDn6LJMULatQ%40mail.gmail.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

>>>> To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0nU3OJTkPXYX9OCd0t8mAg1Ee4gL9T7ZLOD2rECYavQg%40mail.gmail.com.
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> --
>>> Marcus Ottosson
>>> konstr...@gmail.com
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

>>> To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBsvA3ykDs0Gf8aQqXhKxjFMSuYgwd-DG5N4aTdTAbiyg%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA11d1%3Du9p3b4UMUZZ3ix9oYX6CGcJ4t%3DHL8Jd3VcwKukA%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Marcus Ottosson
> konstr...@gmail.com
>
>
> --
> Marcus Ottosson
> konstr...@gmail.com
>

> --
> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBkYQNNAUqmB5o--ykrEV3-VwrsC%2BR64eQOmb7VRmo3_A%40mail.gmail.com.

[Chad Dombrova]

On Jun 2, 2014 3:37 AM, "Marcus Ottosson" <konstr...@gmail.com> wrote:
>>
>> This would replace the process where you have an application and you have to get someone to run your build script to provision the machine and install libraries and hopefully avoid conflicts.
>
> This sounds awfully close to being an alternative to Rez. If Rez configures a state completely out of thin air, with only the required dependencies, then Docker could instead be used to set-up pre-configured, more course-grained states that could run in parallel on a single worker, especially since it seems capable of sharing resources across states (or “containers”). Thoughts?
>>

It isn't anything like Rez other than maybe some very high level festures. Rez is managing packages from an environment standpoint.

Docker uses kernel features like Cgroups and Linux containers to give your container it's own process space, ethernet device, root environment, etc. You have to explicitly map a virtual resource to give the container access to the host filesystem. And you have to map ports out of the container to expose them to the host. This gives you the ability to build the container and not even consider if it will have library and dependency conflicts with other things on the system. Because there isn't a need for dependency resolution. It doesn't have to play with other shared libraries and deps.

Justin,

do you know if it is possible to use docker to run GUI applications like Maya?  I only ever hear about it in the context of running services / daemons / servers, etc

-chad.

 

[Justin Israel]

It should be able to run any type of application really. The only limitation by default are what ports and access to the host filesystem that are made available to the container. But you can export whatever ports you want and mount the host file system. To the host, it all looks like a docker instance process. But inside, the application still have full use of the system, just in its own environment space, with its own process space. 

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7GSaR5xn4hB9hcweexJLH9ZA8kUzTT2%2BsD%2Burj7j4%3D-Fg%40mail.gmail.com.