Tutorials & Tips on getting started with Tasks

[statix vfx]

Hi All,

Hope this is an appropriate forum to ask a few questions and learn more about Gaffer.

Specifically I'm interested in tips and examples/documentation (if there are any around) on using Tasks for pipeline automation.

We are currently using lots of shell scripting to make our pipeline do lots of things, ingest, conversions, dailies, publishing etc. But its quite limiting to use bash and python and would like to explore Gaffer to simplify and unify more of our scattered old ways.

For instance, are there ways of collecting a list of quicktimes based on a system command, and feed that list into another node (another system command) and have them be executed in parallel? Like Houdini's PDG? I can do a chain of sys commands that dump data in /tmp/ and passes the data that way but it seems dirty and not very Gaffer like.

We are also using Afanasy/CGRU and have written a hacked dispatcher (it just executes the graph on the farm sequentially, completly ignoring complex relations and task dependencies) and we are definitely not using Gaffer to the fullest potential, but are stuck with no real examples to look at.

Are anyone willing to share/shed a bit more light practical examples (that are obviously not tied to NDA'd pipelines etc)

Thanks, hope its ok to ask all these noob questsions.

[alex...@cinesite.com]

Hi,

Here are a few practical tips for getting your Gaffer automation off the ground:

automation using out-of-the-box nodes
1. you can use Python expressions connected to Wedge or other Task nodes to build dynamic automation jobs. This approach will only work if the job structure (e.g. number of files to be processed in parallel) is available at job dispatch time. For example, you can use a python expression with os.listdir(quicktime_dir) to drive Strings plug on Wedge node. This will create a task branch for each quicktime.

2. if the number of tasks is not known at the dispatch time, you can process tasks in batches. For example, you can create a job with 10 tasks and each of these tasks will be processing some of quicktimes in parallel (0, 10, 20 for the first task, 1, 11, 12 for the second task, etc). You will not achieve maximum parallelism this way, but it is still a practical solution

automation with custom nodes

To get the most of automation, you should create your own nodes and apps. You are right that there are not many open-sources examples of Gaffer task nodes, but Gaffer itself is open-source and we are constantly using Gaffer's source code as the source of inspiration for our tools. Most of the task nodes in GafferDispatch module are python-based - they are great examples for creating your own Task nodes

1. Most of our task nodes are uber-nodes build from other nodes and expressions. A lot of them are not actually implementing Task API, but simply connecting Task plugs. For example, our Arnold render-layer dispatch node is build of:
- 10 python expressions
- TaskList
- 5 scene nodes e.g. StandardOptions, ArnoldOptions, etc
- Arnold render
- 5 cs-python commands (a wrapper of PythonCommand) to run denoising, post-processing, error-checking, etc 

Task nodes use a combination of per-frame tasks, sequence tasks and immediate tasks. Immediate tasks are particularly useful for setup work that should be executed once during dispatch.

2. We have a few nodes that override parts of Task API:
- hash() function is usually overridden to enable/disable certain tasks programmatically

- execute() or executeSequence() is overridden to change the actual execution of the task or to add things like progress reporting. Our Nuke command node or our assetised ImageWriter node are examples of nodes that override these functions

3. We wrap execution of typical DCC commands into custom nodes to make automation templates simpler e.g. CsNukeCommand or CsMayaCommand

4. 'Scene globals' are a great way to pass pipeline data down the graph. For example, we rely on scene globals to pass information about render-layers down the graph. So our templates have a single render node, but it can deal with variable number of layers and layer setups.  I cannot overstate how much our pipeline (including automation) relies on passing data in Globals plug

5. Please remember that Gaffer supports arbitrary plug type connections, not just scene, image and  task ones. You can pass asset information between nodes using string, stringArray, dict or your own custom data type. For example, a render node can pass the asset data to downstream slapcomp nodes using plug connections

6. A more advanced automation pipeline will require passing dynamic data between task nodes using files or databases. There is currently no built-in support for this in Gaffer, but there are plans to support this in the future. At Cinesite, the passing of dynamic data between tasks is file-based. We have a couple of C++ nodes that deal with nitty-gritty aspects of data management and type conversions.

7. You might want to consider creating custom Gaffer apps to handle automation. Gaffer provides a bunch of built-in apps (e.g. gui or execute), but it is quite easy to register your own using GAFFER_APP_PATHS. You can run most of automation tasks using execute or python apps, but custom apps can make the pipeline cleaner

8. Some pipelines use custom dispatcher to take full control of dispatching and automation. We rely on the built-in dispatcher (Tractor), but we use preDispatch signal to tweak dispatched jobs. Many of our pipeline tasks are relying on self-expanding tasks (task that generate other tasks). I do not think gaffer support them out of the box, but you can add such functionality in preDispatch hooks.

regards,

Alex

[statix vfx]

Alex, this is fantastic! Thank you so much for the detailed writeup. Its exactly what I was looking for.

I've started to play around (prototyping) with using the -gui method to run scripts with gaffers boxes as the front end for submissions, love how easy it is to plug more tasks and subtasks into the graph. Makes it very clean to see what it does.

Hopefully I can post some of our tools down the line.

Thanks!

--
You received this message because you are subscribed to a topic in the Google Groups "gaffer-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gaffer-dev/SxoqZb7y_e4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/bcf74ca5-118b-475f-9564-13670f818677n%40googlegroups.com.

[statix vfx]

Just wanted to update and thank you again Alex.

We're in the processes of upgrading / rebuilding our pipeline. Removing old scripts and code here and there and upgrading out database backend (and frontend) with something more modern.

Having an absolute blasts with the tips you provided. Very surprised on how much stuff you can do with just a single wedge node, string plugs and a systemcmd task node.

So far I've rebuilt our 2d asset library ingestion system to:

* Recursively find the assets on disk (they are all EXRs)

* Convert them to quicktimes.

* Convert them to GIFs

* Upload and display them in our database frontend (nocodb)

This is presented as a standalone tool (using the mentioned -app flag) to ingest more assets as they come in using the file-browser (right click -> add to asset library)

I do have a pretty dumb "integration" with Afanasy (our render manager). But it currently just sends the entire graph to one machine. Not sure how to distribute them in a clever way.

I'm also hoping to maybe do a tutorial series on getting started with these discoveries. There's not a lot of info out there. And I've found Gaffers flexibility now to be very useful even for smaller companies.