[ANN] Plow: Open Source Render Farm Software for VFX

[Justin Israel]

Hi Everyone!

I wanted to announce a project I have been working on for a while now. It is still in dev status, but has made enough progress towards being production ready to reach out to enthusiastic developers.

Plow - A render farm management software specifically designed for VFX workloads.

While there are a few open source projects out there with a similar theme, they are either not very good, or not specifically focused on VFX workloads. 

Members of the Plow project have previously written 1+ proprietary Render Queues still in active production by one of the largest VFX studios in the world.

The software consists of:

1. Plow server, written in Java, using a PostgreSQL database 
   
2. rndaemon - Render Node Daemon, a process manager that runs on each node which should be made available to run tasks. Reports stats of both the system health, and all running tasks (written in python)
3. python client - Plow uses thrift, so generating client bindings for any language is quite easy. We use python as the official client API. One could choose to write client code for Javascript, C++, Objective-C, etc.
4. plow-wrangler - A PySide client application for managing and monitoring the queue (still work in progress)

We have separated our job description library into a project called Blueprint. Via Blueprint, you can define job "template" modules, which make submitting to Plow really easy, via blueprint files.

Currently we have Plow running jobs such as Blender, Maya, arbitrary commands, and command ranges. With Blueprint, its a minimal amount of code to create a new module to support custom required/optional args, and how to ultimately end up with a Job that can be submitted.

Some of the features:

• Project / Folder structure - Supports multiple "projects", each with various "folders" to filter jobs into, with individual resource and priority control.
• Job/Layer/Task inter-dependencies (i.e. have a task wait on a layer, or a job wait on a job)
• Resource and Historical stats-based scheduling - Server-smarts in how it decides which task to dispatch to which node based on a number of strategies: Available resources, past runs of previous version of the same task, service and tag based, ... i.e. fit as many tasks onto as many cpus as efficiently and comfortably as possible.
• Clusters - organize render nodes into logical clusters, and associate resource quotas between projects and clusters.
• Service-based layers - A Job consists of 1 or more layers which consists of 1 or more tasks. Layers of a job can be associated with service definitions, such as Maya for example. A service can dictate how few or many resources its tasks can get, and what kind of hardware then end up running on.

I'm mentioning the project here because I know there are a lot of smart developers in this group. If you are interested in getting involved with a seriously awesome system, there is plenty of cool stuff to do ranging from server => client => gui => integration with different applications (Maya, Nuke, Houdini, ...). 

To develop or use, make sure to build from source. Features and fixes are happening so rapidly that we are not really at the stage to be providing regularly update binary released. But it is the plan to do that eventually.

Links:

• Plow Repo - https://github.com/sqlboy/plow
• Blueprint Repo - https://github.com/sqlboy/blueprint
• Pretty empty placeholder site - http://www.plowrender.com 
  
• Plow User Group - http://groups.google.com/group/plow-render
• Dev Tips Wiki - http://code.google.com/p/plow/wiki/dev_tips

If you want to get involved, you can fork the repos on github and submit pull requests. Definitely hit me up with any "getting started" issues. Hope this intrigues some of you out there!

-- justin

[Justin Israel]

After realizing that I wrote this following line in haste, I figured I would rephrase it:

"While there are a few open source projects out there with a similar theme, they are either not very good, or not specifically focused on VFX workloads."

Plow aims to satisfy as much of these areas as possible (but not limited to): 

• Scalability for very large farms
• Scheduling, dispatching, and constructs that make sense for VFX (to handle wildly varying resource, run time, dependency, and priority requirements for tasks)
• Flexibility of integration to existing pipelines (environment management, statistics reporting, submission / job description, tracking / asset management)
  
• Decent client tools
• Decent configuration options

There... I feel better about that replacement description :-)

[Matt Estela]

Looks pretty cool!

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To post to this group, send email to python_in...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Marcus Ottosson]

Exciting!

[Ævar Guðmundsson]

Looks very cool indeed

[Marcus Ottosson]

Hey Justin, I noticed this isn't active anymore. Would you mind talking about your experiences with this, how come it isn't active and perhaps a post-mortem on what to do/avoid next time?

On 14 June 2013 16:18, Ævar Guðmundsson <aevar.gu...@gmail.com> wrote:

Looks very cool indeed

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To post to this group, send email to python_in...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

Hey Marcus,

While I would be more than happy to write a post-mortem on the topic, to be honest it is actually not dead. The open source project is coming back around summer time. 

The reasons behind why it disappeared are related to employment contract issues, and stuff that I can't get too detailed on. But it just meant it had to be made private for the time being. It is a fantastic project, and I'm really hoping to see it back online soon, with some people even from this group jumping into development. 

I can definitely notify you guys on here once it is made public again.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOD2ry1GPqHv9_%3DoVKVd9ucWj-qWAVTQsMkq4vxx%2BcG0%2Bw%40mail.gmail.com.

[Marcus Ottosson]

Oh, sorry, as soon as I hit "Send" I realised this may have been the case, but I was too quick to assume otherwise.. I too thought it looked promising and was surprised to see it gone.

Looking forward to reading more about it this summer.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1kGVmk6oCc-d6eE_T%3D0_9Wr3%3D1ZQcbpM2kvBzk%2BtW0Pw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

Thanks! Would be cool if you were able to contribute. 

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAP0jnQ1bvJufMQnSXfR%3DVyBD1_h4STYGHaqoyNgDwqAg%40mail.gmail.com.

[Marcus Ottosson]

I have very little experience contributing to oss, but I'd love to give it a try. Let me know when you're out looking.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1ONeD5ZEP0FThbCxLfNZ9CZDOU38%2BhiRQ8TeNB0FP_LQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Marcus Ottosson
konstr...@gmail.com

[Chad Vernon]

Hey Justin,

I'd be interested in contributing whenever the project starts back up.

Chad

On Wednesday, January 22, 2014 10:26:42 PM UTC-8, Justin Israel wrote:

Thanks! Would be cool if you were able to contribute. 

On Jan 23, 2014 7:23 PM, "Marcus Ottosson" <konstr...@gmail.com> wrote:

Oh, sorry, as soon as I hit "Send" I realised this may have been the case, but I was too quick to assume otherwise.. I too thought it looked promising and was surprised to see it gone.

Looking forward to reading more about it this summer.

On 23 January 2014 03:08, Justin Israel <justin...@gmail.com> wrote:

Hey Marcus,

While I would be more than happy to write a post-mortem on the topic, to be honest it is actually not dead. The open source project is coming back around summer time. 

The reasons behind why it disappeared are related to employment contract issues, and stuff that I can't get too detailed on. But it just meant it had to be made private for the time being. It is a fantastic project, and I'm really hoping to see it back online soon, with some people even from this group jumping into development. 

I can definitely notify you guys on here once it is made public again.

On Thu, Jan 23, 2014 at 1:13 AM, Marcus Ottosson <konstr...@gmail.com> wrote:

Hey Justin, I noticed this isn't active anymore. Would you mind talking about your experiences with this, how come it isn't active and perhaps a post-mortem on what to do/avoid next time?

On 14 June 2013 16:18, Ævar Guðmundsson <aevar.gu...@gmail.com> wrote:

Looks very cool indeed

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To post to this group, send email to python_in...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOD2ry1GPqHv9_%3DoVKVd9ucWj-qWAVTQsMkq4vxx%2BcG0%2Bw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1kGVmk6oCc-d6eE_T%3D0_9Wr3%3D1ZQcbpM2kvBzk%2BtW0Pw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

[Dillon Bailey]

Im also a Qube Certified Admin and would love to check this software out!

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/178c4b3c-abff-41b4-bd74-20ae19a52f60%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Justin Israel]

Thanks for the interest, guys!
I'm hoping we can get this posted up again very shortly.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAAaXfNrDN%3DL4g8cM0Q%2BDGpt07DK6GHcQnzGE27cmBo8UiHDucQ%40mail.gmail.com.

[郭金锋]

Hey,

Seems I have some problem getting to the repo, github told me 404 page not found...

Is it still there? the repo Justin posted in the first floor

[Justin Israel]

The official repo is not available anymore at this time. But Chad Vernon did put up the latest fork that was available :

https://github.com/chadmv/plow

It probably lost its fork reference when the original was made private.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/b7dd6165-b839-4a8e-be2d-30c8bd033c7a%40googlegroups.com.

[郭金锋]

Thanks, very interested.
Are you officially changing plow to another site or is plow
temporarily not available?..

[Chad Vernon]

Yes I started doing some plow development/investigation.  Feel free to take a look and participate.  I'm by no means the plow expert.  I also forked the blueprint repo.

--
You received this message because you are subscribed to a topic in the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python_inside_maya/rPJLZ_DeROA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACRmBzkWjn9_g5gPTWTwMob1JE1DEFjpOMwJSiEMyxgMj94p6A%40mail.gmail.com.

[Justin Israel]

"Official", being the original repo for Plow, is just no longer publicly available at this time. We can't work on it in an Open Source capacity because of conflicts of interest / contractual limitations. But forks exists and can continue to progress if they want to. Unfortunately, myself and the original developer can't contribute to the Open Source version right now. 

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFi_QywGq%2B5-YKoSEobo9wT9PKMzVqR8g2-YrfUKrdGmewWeZg%40mail.gmail.com.