Layered USD in Gaffer and Scene Assembly

[Sachin Shrestha]

Hey folks,

I'm trying out the USD support in gaffer and its awesome that gaffer can both read and write usd* files. I was trying a simple gafferBot example and overriding the transforms on some parts of the model and then writing that back out as a usda file. It seems like gaffer does not reference the original usd file in the exported usd file or there is no control over exporting only the overrides as a separate usd file? Instead it writes out the entire graph as a flat usd schema again. My understanding is that this is because the final geometry with the applied transforms is already available to gaffer in its context and the entire scene's final state is exported as is into the usd. The official USD plugin for Maya or the AL_USD or RodeoFX's Walter usd plugins for maya export those overrides as a separate usd file that can be layered on top of the base usd scene. Also, conversely, if one were to generate separate usd with geometry and transform overrides from the said plugins, how would I layer it back in gaffer? Or is this something currently not possible in gaffer?

On the same topic, how would one go about building a set assembly within gaffer's paradigm? Our Maya set assembly workflow goes something like this (note that the word "assembly" here does not refer to the Maya node, instead its a generic term to denote a set layout):

• Each sub-asset that belongs to a set assembly is modeled separately (as a prop) and the model published at the world origin.
• Each set prop gets a simple rig hierarchy (parent null controls introduced directly above the model group) to transform it within maya.
• This prop rig is then referenced by the artist in the set assembly rig. A sub-asset may get referenced multiple times within the same set assembly copies of chairs, rocks, etc.)
• This set assembly rig is then published for layout/anim/lighting teams to use in shot production - at that point when the set assembly rig file is referenced, the props referred within it become nested references. We track the namespaces of the set props, store their edits in our pipeline as per our conventions and apply them back in downstream departments along-with their materials (multiple references of the same prop asset can share the same shader reference or could use a different shader per prop reference).

This seems like a good candidate for usd and hence I was exploring the USD workflow in gaffer. It could work as a nice consistent interchange format between layout/anim who would use Maya rigs and color teams who would interface with the same set assembly but in gaffer.

usdTest.gfr - Gaffer scene with usd model (gafferBot_model.usda) and applied transforms being exported again as a usd file (gafferBot_scene.usda)

gafferBot_sceneOverrides.usda - Transform override exported via the OpenWalter maya plugin. This contains only the xform section for the C_head_GRP and can be layered on top of the base usd model.

Cheers,

Sachin

[John Haddon]

On 20 August 2018 at 12:49, Sachin Shrestha <noizf...@gmail.com> wrote:

Hey folks,

I'm trying out the USD support in gaffer and its awesome that gaffer can both read and write usd* files. I was trying a simple gafferBot example and overriding the transforms on some parts of the model and then writing that back out as a usda file. It seems like gaffer does not reference the original usd file in the exported usd file or there is no control over exporting only the overrides as a separate usd file? Instead it writes out the entire graph as a flat usd schema again.

Currently it is only the SceneReader and SceneWriter nodes that "talk" USD. Once the USD file is loaded in Gaffer, it becomes a native Gaffer scene and all information about its origin and composition is lost. We're debating improved approaches to this, but it isn't something I expect to change in the near term. Gaffer has some pretty powerful tools for manipulating scenes already, and my concern is that mixing paradigms in one package will lead to confusion. And as far as I know, Gaffer allows a much broader range of edits than are possible in USD (correct me if I'm wrong, I'm no expert).

My understanding is that this is because the final geometry with the applied transforms is already available to gaffer in its context and the entire scene's final state is exported as is into the usd. The official USD plugin for Maya or the AL_USD or RodeoFX's Walter usd plugins for maya export those overrides as a separate usd file that can be layered on top of the base usd scene.

The Maya integrations you mentioned are designed on the assumption that Maya couldn't handle the scene if it were represented natively, but we are striving for the opposite in Gaffer. For most of our workflows, Gaffer is also the final consumer of the scene prior to rendering, so exporting layered edits to other file formats has not yet been a priority.

You might like to experiment with using a DeleteObject and DeleteAttributes node to delete everything except the hierarchy and the transforms though, giving you a "skeleton" scene you could write out as USD and layer over. I don't know enough to know if this would work, but it seems something worth experimenting with.

Also, conversely, if one were to generate separate usd with geometry and transform overrides from the said plugins, how would I layer it back in gaffer? Or is this something currently not possible in gaffer?

My understanding is that you would make a USD file which composes your original geometry and your transform overrides, and then load that file in Gaffer.

 

On the same topic, how would one go about building a set assembly within gaffer's paradigm? Our Maya set assembly workflow goes something like this (note that the word "assembly" here does not refer to the Maya node, instead its a generic term to denote a set layout):

To date, we have built sets in an external package such as Maya, and then imported them into Gaffer for lookdev only. Because different copies of the same asset will be present in different parts of the layout, we would typically use a SetFilter or a PathFilter using wildcards to assign to the assets within the layout.

I would love to explore more procedural approaches to building layouts themselves in Gaffer, but as you note, it is then necessary to find a way of exporting this to other packages within the pipeline. My most outlandish suggestion so far has been to create a file format plugin for USD, allowing .gfr scenes to be loaded within USD just as .abc files can be. It seems so far that most USD integrations are attempting to take over the host package from the inside out, whereas this would be the opposite :)

Cheers...

John

--

John Haddon - R&D Programmer

Image Engine

studio: +1 (604) 874-5634 | jo...@image-engine.com | www.image-engine.com



15 West 5th Avenue, Vancouver, BC, V5Y 1H4, Canada

If you are not the intended recipient, disclosure, copying, distribution and use of this email is prohibited. Please notify us immediately and delete this email from your systems. You may contact us at in...@image-engine.com if you do not wish to receive further commercial electronic messages. We may still send you messages for which we do not require consent.

[Luke Titley]

A diff/patch/delta node might not be such a bad idea for gaffer here ... ?

The idea being, you load in a usd file, transform it however you want in native gaffer, then at the end you diff the output of your graph against the original input and write out the delta as a separate usd file that would be used as a layer?

Not sure about what the structure of the delta would look like in gaffer but... it sort of makes sense...

Luke

[John Haddon]

On 22 August 2018 at 02:09, Luke Titley <luke....@dwarfanimation.com> wrote:

A diff/patch/delta node might not be such a bad idea for gaffer here ... ?

The idea being, you load in a usd file, transform it however you want in native gaffer, then at the end you diff the output of your graph against the original input and write out the delta as a separate usd file that would be used as a layer?

Yeah, we've been discussing something like that. It's not totally clear how we'd say "no diff" for things like transforms, or how far we'd want to go in terms of representing hierarchy changes. In the end, it's going to be hard to capture the full gamut of Gaffer's scene modification abilities in a simple diff format. That's one of the reasons we've tended not to bake the results of node networks, and instead prefer to run them live.

Cheers...

John

[Sachin Shrestha]

Hi John,

It seems that in the current usd implementation within gaffer, I cannot load a usd composition which has nested references to other usd or alembic files? When I tried with a sample usd file exported from maya, it loaded the transforms from the usd file in gaffer but it doesn't seem to load any actual geometry nor do the transforms inherit the values from the usd file (I selected them in the Scene Hierarchy and checked the Scene Inspector). The top level Instead, if I export a flattened usd file (where the geometry is in the same usd file instead of being referenced), then that works in gaffer. Is this a limitation of the current implementation?

It would be nice to have the SceneReader load the nested usd references so that way we can simply export usd layout files from Maya (we also build our set assembly/layout in Maya) without needing to override anything in the gaffer context. The usd in question looks like this:

#usda 1.0
(
    defaultPrim = "MySet"
    upAxis = "Y"
)


def Xform "MySet" (
    kind = "assembly"
)
{
    def Xform "BeechHangerHeroA_standin" (
        prepend references = @assetA_model.usd@
    )
    {
        uniform token[] xformOpOrder = None
    }


    def Xform "BeechHangerHeroB_standin" (
        prepend references = @assetB_model.usd@
    )
    {
        double3 xformOp:translate = (0, 0, 1034.839013964176)
        uniform token[] xformOpOrder = ["xformOp:translate"]
    }
}

Cheers,

Sachin

[John Haddon]

Hi Sachin,

This definitely isn't intentional, but its early days in the USD support in Gaffer, so we may very well have overlooked something (we're learning as we go). If you could share a minimal broken example including the referenced file, we can take a look.

Cheers...

John

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sachin Shrestha]

I have attached the simple repro here. Once you extract it please cd into the "usd_repro" directory and start gaffer from there.

|-- usd_repro
|   |-- maya
|   |   `-- MySet_Assembly_v001.ma
|   |-- usd
|   |   |-- Cube_Model_v001.usd
|   |   |-- MySet_Assembly_v001.usda
|   |   `-- Sphere_Model_v001.usd
|   `-- workspace.mel

The MySet_Assembly_v001.usda file has nested references to the Cube_Model_v001.usd and Sphere_Model_v001.usd. The references are from relative paths.

Cheers,

Sachin

[Andrew Kaufman]

Testing here in Gaffer 0.48.1.0, I get the same problem with the usda, but when I try to load the Cube_Model_v001.usd or Sphere_Model_v001.usd, my SceneReader actually errors saying:

USDScene::Reader() Failed to open usd file: '/disk1/usd_repro/usd/Cube_Model_v001.usd

Are those files working as expected on your end? What version of USD where they created with? The Kitchen usda is loading fine for me, and the concept seems to be the same, so maybe its just a version mismatch?

Andrew

[Sachin Shrestha]

Hi Andrew,

That could indeed be a problem as I compiled USD and the maya translator plugins with their latest master so it would be v18.09 whereas gaffer dependencies seems to use 0.8.2. I’m not in front of a workstation right now but I’ll check the files again at my end tomorrow at work.

If I had to bump up the USD version for gaffer (if there was no API incompatibility) then I can do that in the buildUSD.sh file by pointing to the latest version’s tarball, right?

Cheers,
Sachin

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

[Andrew Kaufman]

Yes, I think that's right, but I suspect there are API incompatibilities. I seem to remember a discussion about changes to their Collections API. We have an internal ticket to get onto the latest USD, but it doesn't have much priority here as USD is quite experimental in our pipe thus far, and actual production priorities have to come first.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.

[John Haddon]

There are definitely API incompatibilities - Cortex currently doesn't compile with the latest USD. Patches very welcome!

[Sachin Shrestha]

Hi John, Andrew,

So it did turn out to be version incompatibility issue. I built the older USD version that gaffer uses and it worked fine with my example scene after re-exporting the usds from Maya.

USD is a bit new for me so I’m not sure if I’ll be able to figure out the compatibility issues but I’ll give it a shot!

Cheers,
Sachin

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

John Haddon - R&D Programmer

Image Engine

studio: +1 (604) 874-5634 | jo...@image-engine.com | www.image-engine.com



15 West 5th Avenue, Vancouver, BC, V5Y 1H4, Canada

If you are not the intended recipient, disclosure, copying, distribution and use of this email is prohibited. Please notify us immediately and delete this email from your systems. You may contact us at in...@image-engine.com if you do not wish to receive further commercial electronic messages. We may still send you messages for which we do not require consent.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.