broad tips on developing a reader from an asset management system

[Mathew Mackereth]

Hey guys - still just kicking bits of gaffer and really enjoying it.

Just looking for a bit of help about where to look, a reading list perhaps, so apologise for the exceptionally broad question :)

Lets just say i have an asset management system that I can query to return a hierarchy of assets and some metadata/ attributes that should make up the shot i want to work on.  I want to take this information, and construct a gaffer scene/ bunch of locations in a hierarchy from it.

I currently do this in houdini for example by actually looping through and creating an alembic node and a group with shaders etc, one by one, resulting in a big group of nested nodes.  This works in Gaffer too, I can already create a ton of alembic nodes and hook them up with a python script and group them so it would 'look' like a nice neat hierarchy.  But surely the scene can be generated directly with all the data/ metatada/attributes/shader assigns needed without the big bunch of nodes?

the advantage of this would be, if i want to switch shots with a variable - i don't have to create or destroy anything in the ui as I flick between shots, only the scene being generated is changing....  

I don't yet have an understanding exactly of what all the nodes can and should do  -   perhaps an example of how to actually create a custom locationin the scene, or other nodes that already exist that i can look at the source for - i'm still crawling with gaffer :)  Is there a way to mock such things/nodes up with python?

thanks all for your time.

Macka

[Ivan Imanishi]

Hi Mathew,

Here at Image Engine, for a similar purpose, we built a node called SceneBuilder, based on the SceneLoop node.

The SceneLoop allows you to compute the upstream graph varying an "index" variable value, and you can then use that value to drive, for example, SceneReader file paths, and then combine them using the Parent node.

I have built a very basic example of what that would look like in the attached gaffer script inside the box called "ReaderWithSceneLoop".

It takes a list of file paths that can be read by a SceneReader.

In this particular example, I use the base name of the file as the group name in the scene hierarchy, but you can change the expression and inputs to suit your needs.

It will not handle arbitrary hierarchies that are not built into the files being read. At least in its current form. But it will give you some idea of what we do here.

You can also look at the DigiPro 2016 gaffer presentation: https://vimeo.com/201047816

At the end of that video (starting at about 18:20) Andrew talks about the our Asset Management nodes and the SceneBuilder.

However, we recently switched to using the newer CollectScenes node as the base for the SceneBuilder, which has some performance advantages over the SceneLoop node when what you need is just collecting scenes from different sources.

Nothing is fundamentally different, but basically the CollectScenes does both the loop and the Group/Parent operations internally.

I've also added an example of that in the attached file, in the "ReaderWithCollectScenes" box.

Note that in both boxes I'm pointing to files that should be distributed with gaffer. But you can replace the paths with whatever files you want (as long as the SceneReader can handle them).

In both boxes, you could easily drive the promoted "filePaths" plug via an expression node based on your Asset Manager information.

You could also add extra data as plug values that are then used in the internal expressions to drive other operations like creating Sets.

If you need to add a lot of data per iteration, you could consider using a Gaffer.AtomicCompoundDataPlug() instead of the Gaffer.StringVectorData plug I'm using.

CompoundData is a dictionary-like data type from cortex.

I'd like to point out however, that we also use event-based graph-building nodes here as well as the fully dynamic SceneBuilder.

That's because our Look Dev is published independently from the geometry as a gaffer graph box, which is loaded as Reference node, added to the scene data after we've loaded the geometry (using a SceneBuilder).

A SceneLoop/CollectScenes type of iteration cannot change the graph, only the computed information.

Therefore, loading a different Reference file for each iteration is not allowed.

What we do here is to have another node that, based on events (user input, pre-dispatch signal, etc) and the Asset Manager information, will rebuild an internal graph that includes loading or reloading any Reference nodes as well as setting up SceneBuilders.

However, wherever possible, we do use the SceneBuilder by itself, since it immediately updates to new input, and it fully supports things like wedging (where a context variable defined by a Wedge node can be used to drive the scene being built).

Does this help you in any way?

Let me know if something isn't clear, or if you need more details.

Ivan

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mathew Mackereth]

Hi Ivan, 

Sorry for the Late reply. That is absolutely Awesome, end exactly what i need to get my head around it all - thank you so much for all the detail!

Macka 

[Miguel González Viñé]

Hi Ivan,

I'm totally new to Gaffer and I'm in love with it :)

Thanks a lot to the team, and for the examples you guys are attaching to this list.

I'm trying to replicate by myself your examples in sceneBuilder.gfr and I've a question:

How do you create the "filePaths" plug inside the "Settings" tab in the "SceneLoop" node? I can create similar ones in a "Box" node through the "Edit UI..." menu, but this menu doesn't allow me to create them in the "Settings" tab of a "SceneLoop" node.

So the way I did it is creating it in a "Box" and then saving the scene and looking to the script .gfr file.

Now I can do it with:

script['ReaderWithSceneLoop']['SceneLoop1'].addChild( Gaffer.StringVectorDataPlug( "test", defaultValue = IECore.StringVectorData( [ ] ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )

Gaffer.Metadata.registerValue( script["ReaderWithSceneLoop"]['SceneLoop1']["test"], 'plugValueWidget:type', 'GafferUI.FileSystemPathVectorDataPlugValueWidget' )

But, is this the only way to do it or is there another one I didn't know?

Thank you,

Miguel

Btw, I've searching for documentation about how to create AtomicCompoundDataPlug, but I can't find anything about it...

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

[Daniel Dresser]

Hmm, that's a good question Miguel - I'll ask Ivan on Monday, but I'm not quite sure why he would have added that plug directly to the SceneLoop.

The usual ways of doing this would either be:

A) add your new plug as a child of the user plug on SceneLoop.

This can be done either with:

script['ReaderWithSceneLoop']['SceneLoop1']["user"].addChild( Gaffer.StringVectorDataPlug( "test", defaultValue = IECore.StringVectorData( [ ] ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )

Or with the "Edit UI ..." menu on the gear icon on the SceneLoop

B) If you want to make your "SceneLoopWithExtra" more convenient and reusable, you could package it inside a box, promote all the plugs, and then also add your new plug to the box.  This would give you the option to publish it as a reference.

Ivan's approach of adding directly to the root of the SceneLoop node I don't think is something we would encourage in user scripts - since it isn't very editable in the UI.  But it does work, and it might be convenient if you're building the whole graph from a script anyway ( I suspect Ivan may have been modifying a graph from IE which is automatically built by some fairly complex asset management code ).

As for AtomicCompoundDataPlug, we seem to currently be only using it for internal processing, so we haven't really exposed it yet.  You can create one like this:

script['Box'].addChild( Gaffer.AtomicCompoundDataPlug( "testAtomicCompound", Gaffer.Plug.Direction.In, IECore.CompoundData(), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )

And then set it like this:

script['Box']['testAtomicCompound'].setValue( IECore.CompoundData( { "a" : IECore.IntData( 3 ), "b" : IECore.StringData( "foo" ) } ) )

And you can read it in an expression.

But I don't think there is any UI yet to allow a user to directly view or edit the value of an AtomicCompoundDataPlug - we currently only use it for internal processing on the C++ side.

-Daniel Dresser

[Miguel González Viñé]

Hi Daniel,

thanks a lot for your long and detailed reply. I like the option B, it looks more easily to read. it seems to me that plugs in the "User" tab are very hidden.

Now AtomicCompoundDataPlug is more clear, thanks! But I think I'm going to be away from it for now :)

I have two more questions that I hope you can clarify them to me when you have time:

1.- You said in this comment: https://github.com/GafferHQ/gaffer/pull/2629#discussion_r203515797 that it's good to keep the geo loading and shader assignment together. So in the example of this thread, I should also assign the shaders in the loop to make the graph more efficient?

This is the comment in case the link doen't go to the correct one: 

"We also want to discourage people from doing two separate phases, first generating all the geometry and then applying all shaders - this is what often happens in the IE pipe, and it's really inefficient. It's better to have lots of small branches feeding into your graph, each of which loads some geometry and sets up shaders on just that geo."

2.- When you want to promote plugs to their parent Box, sometimes it creates automatically "BoxIn" nodes connected to the promoted plug, that makes the nodes network easy to read. (this is the case for promoting the specular of a shader, it creates a "BoxIn" node called "parameters_Specular"). But there's a lot of promotes plugs (like a "fileName" plug of a "SceneReader" node) that doesn't create this "BoxIn" node and it's hard to find which plug is the promoted one. This can be even harder if you rename the plug name in the "Box" (for example, renaming "fileName" to "filePaths" or "assetFiles" or whatever) Is this behaviour going to change in the future, showing always which plug are promoted, or is there an easy way to find them? I think it's easy to get lost when you open an old gaffer script.

Thanks again.

Miguel.

[Ivan Imanishi]

Hi Miguel,

Sorry for the delay in replying.

Regarding the `filePaths` plug, I had just created it using python directly. And yes, it is mostly based on how our internal nodes are set up, where they are created as their own custom node classes (derived from the base Gaffer node types.

So, I didn't use the Edit UI to add the plug, but that should be totally valid. You just need to set up the expressions to look it up.

> 1.- You said in this comment: https://github.com/GafferHQ/gaffer/pull/2629#discussion_r203515797 that it's good to keep the geo loading and shader assignment together. So in the example of this thread, I should also assign the shaders in the loop to make the graph more efficient?

If you can, that would definitely be more efficient.

Note however that you cannot change the graph itself during a loop. That's what I was referring to in the end of my original reply to this thread.

At least in the way we store our shaders here (as References), it would not be possible to update to different shaders during the loop, because that would alter the graph.

John has been considering ways of allowing that though.

Ivan

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.

[dan...@image-engine.com]

> it's good to keep the geo loading and shader assignment together.

To expand on this a bit - generally, the less geometry flowing through your processing nodes, the better.  It's often a bit hard to pin down the exact performance characteristics of this stuff ( the Gaffer cache covers over a lot of stuff ), and depending on the structure of your shader assignments, it may not make a huge difference.

But to give a silly example where it would make a huge difference: suppose I have two assets: a pigeon, and a fully detailed 747.  I could assign the eyeball shader on the pigeons with the filter ".../eyeBall*".  This wildcard will have to search the whole hierarchy for a name match - if it's the pigeon flowing through, this isn't so bad.  But if we merged the scene first, and then used this filter, it would waste a lot of time checking the name of every single part in the 747 to conclude that none of them are eyeballs.

Usually you would be more specific than that about targetting your shaders anyway, but it always safer not to have piles of unrelated geometry in the hierarchy while doing the bulk of your processing nodes.

As Ivan points out, it may be tough to do your lookdev assignments inside the SceneLoop - if you have to do them afterwards, it should be possible to get acceptable performance, you just have to be more careful.

2) Currently we only show BoxIn nodes for plugs that have nodules ( the little circles on the nodes that allow you to make connections in the Graph Editor ).  I agree that there isn't enough indication something is happening for plugs without nodules when they get promoted.

It used to be that there was never much indication when a plug without a nodule was connected to something.  We've only recently added the green lines to the center of a node showing that something on the node is driven by an expression.  Now that we have this, I guess we could create BoxIns and then we'd show a green line from the BoxIn to the target node.  Not sure if this is the best UI.  But yeah, basically, we've been working on this, but we haven't really thought about the Box case yet.

-Daniel

[Miguel González Viñé]

Hi Ivan,

thank you for taking the time to answer.

> Note however that you cannot change the graph itself during a loop. That's what I was referring to in the end of my original reply to this thread. At least in the way we store our shaders here (as References), it would not be possible to update to different shaders during the loop, because that would alter the graph.

Ok. Now I understand that part of your original reply. I thought that I could change the path inside the loop of the "Reference" node to the different LookDev graphs.

Hi Daniel,

Thanks for the eyaball example. Now it's totally clear.

The green arrows to the center of the node would be great. Or anything you guys think that could be better to follow the promoted plugs.

Thanks to both of you,

Miguel