Custom plug in Python and custom data in the graph

[Carlo Giesa]

Hi there!

After a long time, I'm finally getting my feet back into the Gaffer world and I had two basic questions that will have an impact on how to approach things in the upcoming weeks.

I'm wondering how "easy" it is to implement a custom value plug. I can't see any Python example in Gaffer's source code. So, I'm wondering if that is a restriction or more a performance reason. I would like to have a custom plug for displaying production and publish data on my nodes, it's not for any computation of the node's data. I might come along with only a custom widget and using existing plugs, but I thought it would be worth asking the question. In general, I would like to avoid starting implementing stuff in C++ (quite rusty on my end).

My second question is about custom data flowing around in the Gaffer graph. We have a publish data dependency system that makes sure to keep track of down- and upstream publishes. Since Gaffer is super-dynamic with its context system, I absolutely can't rely on connected nodes to detect upstream data. My idea was to 'inject' our internal uids of any loaded data into the graph, and when rendering out, I get all 'injected' uids that end up to the render node. Is that doable with only Python? I'm also wondering how merges would happen. Ideally, I would like to have all 'uids' merged together, like a 'set' data structure.

Thanks for any input!

Greets,
Carlo

[John Haddon]

Hi Carlo,

Nice to see you back in these parts!

It is actually possible to derive from ValuePlug in Python - see `ValuePlugTest.testDerivingInPython` for an example. There is currently a restriction though : the plug doesn't get to define a custom value type - it can only add child plugs to store values. So it's only useful for "compound" plugs. I think it is feasible to lift this restriction, but we'd probably want a fairly strong reason.

I think where custom plug types really justify themselves is when users will be tasked with making connections between them. Type-matching has proved to be the best way of enforcing constraints here - it is possible to use `Node.acceptsInput()` overrides, but it's fiddlier and less robust. If users won't be wiring these plugs together, then I think using a regular plug with a custom UI may be a decent option.

I suspect other folks are better situated to answer your question about dependency tracking, having implemented it for themselves. But to me there seem to be two main options for having this data flow through the graph :

1. Store it as custom options in the scene globals. The only merging logic for such options is to merge the top level "dictionary" of options. So you'd need to track each dependency using an individual option. Nodes like MergeScenes and CollectScenes can merge options from multiple sources, but you need to choose the right settings for that to happen. And because the data is in the scene globals, it won't be updated if someone prunes an asset out entirely downstream. Likewise, nodes like Parent and Subtree won't keep anything up to date.
   
2. Store it as attributes on the root location of each "asset". So you might have an attribute called "asset:uid" at `/world/robot1` and another at `/world/cowB`. Combined with a set containing all the locations of asset roots, you then have a data structure that will update naturally when scenes are modified, no matter what the user does (as long as they don't intentionally delete the attribute). Downstream, you could use a FilterResults to get all locations in the set, and feed into a Collect node to query the attribute values. That would give you a quick multithreaded way of querying all the assets in the scene.

Unless someone with more experience has other thoughts, I'd be tempted to pursue option 2.

Cheers...

John

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/920dbd73-0f67-44dd-8d64-0fc66ea839b6n%40googlegroups.com.

[dan...@image-engine.com]

ImageEngine has quite an elaborate asset management approach that does involve wiring asset management nodes together, and they're just using string plugs. There's nothing technically preventing users from connecting an asset management URLs plug to a regular strings plug - but they use custom widgets that display what users need to know, and a custom plug color to distinguish asset management nodes, and that seems to be working fine. Users just know "Green plugs are asset management URLs, if I connect it to a string plug, I'll get a meaningless string of numbers, so I don't do that, and just connect green plugs to other green plugs".

I don't have much input on your second question ( I'm not sure whether ImageEngine has this sort of dependency query in Gaffer, they often drive things the other way around, where the asset management database determines what assets are imported, and then Gaffer nodes automatically bring in whatever assets are already listed in asset management ), but John's option 2 sounds reasonable.

[Carlo Giesa]

Dear John and Daniel!

Thanks a lot for your input. Will test this out and let you know how lucky I'll be with that.

Greets,
Carlo