Sample USD data

[Tom Cowland]

Hey folks,

We are working on improving USD support in Gaffer/Cortex. Whilst our
priorities are somewhat dictated by the needs of the stakeholder
studios, we'd love to make sure we are handling a wider variety of data
as well as we can.

Do any of you have any USD assets that you could share with us?
Especially any that don't load into Gaffer properly at present.

If it's possible to include a reference image/render or any extra info
on how the data is meant to look, that would be a great help. It will
allow us to verify any changes we make.

We're particularly interested in how the various DCCs author custom
attributes/primvars and referenced or instanced data sets.

Any models that include shader setups would be great too.

Many thanks in advance,
Tom

[Carsten Kolve]

The nvidia usd attic sample seems potential useful?

https://developer.nvidia.com/usd

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/4226c82de032e808509a0ba59cdd18f86809e183.camel%40cinesite.com.

--

// carsten kolve - www.kolve.com

[Heribert Raab]

houdini export of a character, there is usd export from blender
https://www.dropbox.com/sh/73w6y73xghj484p/AAB7hhozIlBNb1MQU9moQDd4a?dl=0

also a alembic, gaffer can't read faceset either from usd or abc. its
important for character loookdev.

[Heribert Raab]

houdini usd export example with groups facesets etc :
https://www.dropbox.com/sh/1f5t3zt5ffbjhb1/AADV89kCSzYQbsHCv7KKYxtWa?dl=0

------ Original Message ------
From: "Tom Cowland" <to...@cinesite.com>
To: "gaffer-dev" <gaffe...@googlegroups.com>
Sent: 2020-09-16 6:39:47 AM
Subject: [gaffer] Sample USD data

[Heribert Raab]

houdini usd spline example with custom attr and colors.

https://www.dropbox.com/sh/rdx886x3z3ulg2u/AABplatKrtK5Ve12YbWlI_Swa?dl=0

[Tom Cowland]

Thanks Heribert, they're going to be really useful. Appreciate your
time in putting them up for us.

> also a alembic, gaffer can't read faceset either from usd or abc.
> its important for character loookdev.

Yeah, Gaffer doesn't support face sets presently, regardless of the
format.

What exactly are you using them for in your character lookdev? We have
some ideas for a basic workflow, but it depends on what you need.

Also, do you make use of overlapping face sets?

[Tom Cowland]

Good tip, thanks Carsten!

[Heribert Raab]

there is a general problem with the vanilla gaffer to handle data. i know, there limited resources on development and time. but from an end-user standpoint, Gaffer is quite a hassle with data.

the Pixar and Nvidia examples always work, but if don't have the export tools from Picar or Nvidia, you out of luck.

i can read and export any ABC or USD in Houdini, Maya,3dmax, Blender, substance designer etc...

no problem, you can do a full rounds trip. but with vanilla gaffer its a different game.

for example alembics files.

* I got a camera export of ABC, which just a camera in it. gaffer could read it, no problem but i soon i hit render on Arnold or gaffer Cycles --> instant crash.  the solution was, load this ABC into Houdini and export it again, read into gaffer --> it works.

reading attr into gaffer, it takes vector x,y,z, standart vector. arrays like color[0],color[1],color[3] are ignored.

quaternion or vector 4 not readable, but you need quaternion for instancing right? so have assembled it first with a quaternion node.

once you manage to export a point cloud or particle in way gaffer can read, the attr comes in as facing varying?  with only points in the cache file.
-->you can't use unless you add osl object node and convert everything to vertex varying.

read spline is more a gamble game.  polylines, b-splines, NURBS curves? something all works something none of it works. you lucky if in Houdini and easily shuffle data around, but if use any other software, you out of luck.

alembics with hierarchy is another gambling game.

some with USD, basic polygon you may be lucky enough to get the data in from Houdini. USD from Blender, Maya etc. it won't read.

 there one workflow, load everything into Houdini, split all data in single floats and assemble the data in gaffer with OSL. the only which works reliably.

it sounds a little bitchy but that's the general user experience, I know some people on this list, which gone through the same.

but I really appreciate all the works you guys put this into the OpenSource Project, thanks for that!

so if guys find time to generate a solid Data reader, that would very much appreciated. if I have time on the weekend, I will assemble more examples files.

 

thanks,

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/3b8db21978768c40023c1cb9d15a9c4062d61c09.camel%40cinesite.com.

[Heribert Raab]

default Houdini USD export with camera.

https://www.dropbox.com/sh/8ha3xh4t120qku4/AADDipysw9Hp9xLdvGrObKj8a?dl=0

in current gaffer version the camera but not recognized as camera, it works with the alembic version  

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/4226c82de032e808509a0ba59cdd18f86809e183.camel%40cinesite.com.

[John Haddon]

On Fri, 18 Sep 2020 at 20:10, Heribert Raab <heribe...@gmail.com> wrote:

i know, there limited resources on development and time. but from an end-user standpoint, Gaffer is quite a hassle with data.

Historically, the studios funding Gaffer development have used their own geometry formats or Gaffer's native `.scc` format, so focus on industry formats like Alembic and USD has been lacking. This is something we're keenly aware of, and I am currently scheduled on work to improve the situation. Hence this thread.

 

* I got a camera export of ABC, which just a camera in it. gaffer could read it, no problem but i soon i hit render on Arnold or gaffer Cycles --> instant crash.  the solution was, load this ABC into Houdini and export it again, read into gaffer --> it works.

I believe the issue here is trying to render from a camera that is not in the `__cameras` set. See :

https://github.com/GafferHQ/gaffer/issues/3378

https://github.com/ImageEngine/cortex/pull/1034#issuecomment-584096107

I should be looking at populating the `__cameras` set automatically from USD and ABC data soon.

reading attr into gaffer, it takes vector x,y,z, standart vector. arrays like color[0],color[1],color[3] are ignored.

We don't have any array-of-arrays data type in Gaffer. Are the arrays you are interested in always of a fixed length, say, 2, 3 or 4? If so we may be able to convert to arrays of V2f/V3f/Color4f for you. Bear in mind though that it is preferable to encode the type (vector/normal/point) explicitly as it has a bearing on how data is transformed.

 

quaternion or vector 4 not readable, but you need quaternion for instancing right? so have assembled it first with a quaternion node.

Quaternions are already supported for reading in Gaffer for both USD and ABC files. If you have example data that doesn't load, please provide it. Bear in mind that Houdini's writing of quaternions to Alembic is/was broken - they round trip in and out of Houdini OK but the quaternions are stored in the wrong order according to the Alembic spec :

https://www.gafferhq.org/community/?place=msg%2Fgaffer-dev%2FoK79ijYEyr0%2F8DHNqUZGBwAJ

I believe the SideFX issue for that is #92479 if you want to see if it has been fixed. Gaffer's Orientation node has a mode that you can use to fix the loaded data if necessary :

once you manage to export a point cloud or particle in way gaffer can read, the attr comes in as facing varying?  with only points in the cache file.
-->you can't use unless you add osl object node and convert everything to vertex varying.

Can you provide some example data to demonstrate this? I've seen similar things in the past, and it has always been the case that the Alembic file specified that, and rightly or wrongly we were loading at as specified. Note that Gaffer has a ResamplePrimitiveVariables node that is an easier and more efficient way of modifying the interpolation than going through OSLObject.

 

read spline is more a gamble game.  polylines, b-splines, NURBS curves? something all works something none of it works. you lucky if in Houdini and easily shuffle data around, but if use any other software, you out of luck.

I have recently improved our USD curve loading considerably, but it hasn't been released yet :

https://github.com/ImageEngine/cortex/pull/1070

What I don't have yet is an example of `pinned` curves. Are you able to export some bSpline or catmullRom splines out of Houdini in pinned form?

 

alembics with hierarchy is another gambling game.

Please elaborate. Both USD and Gaffer support the idea of the same location having both geometric data and a transform, partly for convenience and partly for performance. Alembic has no such concept - it is closer to Maya, where a location is either a shape or a transform but not both. When loading Alembics, we collapse single shapes into their transforms wherever possible. Is that what you're talking about?

 

it sounds a little bitchy

It does a little, but we can deal with that. The vagueness of some of your assertions is more of a problem. We already know we need to improve. What we need now are specific examples that show us exactly where and how.

 

but that's the general user experience, I know some people on this list, which gone through the same. but I really appreciate all the works you guys put this into the OpenSource Project, thanks for that!

Thanks for providing the example data so far - I appreciate it. If you can follow up with data that shows the specific problems mentioned above then that would be even more valuable.

Cheers...

John

[John Haddon]

I have implemented reading of USD cameras today :

https://github.com/ImageEngine/cortex/pull/1082

With that, the USD camera loads and matches the ABC one. We are planning to release these improvements in Gaffer 0.59.

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/CALNO3Yfdv7iA%3DJ4v4tpJ%2BgCYyA9a20Rpq_PiMd0h38ninJnhvA%40mail.gmail.com.

--

John Haddon - R&D Programmer

Image Engine

studio: +1-604-874-5634 | jo...@image-engine.com | www.image-engine.com



15 West 5th Avenue, Vancouver, BC, V5Y 1H4, Canada

If you are not the intended recipient, disclosure, copying, distribution and use of this email is prohibited. Please notify us immediately and delete this email from your systems. You may contact us at in...@image-engine.com if you do not wish to receive further commercial electronic messages. We may still send you messages for which we do not require consent.

[Aaron Powell]

I don't know if it helps - I know we're probably getting into more Arnold territory over Gaffer - but it looks like I'm getting this warning from the render engine.

WARNING [Arnold]  light:/mesh_light_geo: for MIS to work properly, polymesh linked to mesh_light should at least be visible to shadow rays

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/CAEVDW43LRKJNoZJh3kdzjo6bDveFC38BkOgdvz2Hmwq8ZiODwQ%40mail.gmail.com.

--

Aaron Powell

Partner, Luna Digital, Ltd.

60 Carlson Road, Ste 7

Rochester, NY 14610

Office: (585) 433-5044 ext. 101

Mobile: (585) 410-0392

www.lunadigital.tv

[Carlo Giesa]

Hi there!

If there is possibility to give some request of further alembic/usd improvements, I would vote for the animated visibility support! Or has this already been taken care of?

Greets,

Carlo

[John Haddon]

Hi Carlo!

I added support for animated visibility from USD last week, and it should appear in Gaffer 0.59 : https://github.com/ImageEngine/cortex/pull/1079. We haven't yet tackled the same for Alembic though...

Cheers...

John

[Sébastien Desmet]

Hello,

we are using USD in gaffer for our current project and we have couple of issues.

The UVS

- interpretation of UVs who come from houdini are not working (we didn't test with other DCC ). we must use oslobject node to convert primvar from uv to st to make it work

The subject was already discussed here with Bruno Pollet : https://groups.google.com/g/gaffer-dev/c/NSlwO9KczqQ/m/J_ATagx8BAAJ

we also get this annoying warning message all the time : WARNING : USDScene : Unknown type texCoord3f[] on PrimVar primvars:uv

I don't know if this message is the problem, on our project, we have thousands of USD instanced houses (USD instance is not working, see below) and we get this warning for every single object in the scene. Lot of the times the render stop randomly during the warning messages and the render can't continue. It just stop, no error are visible in the terminal and I have to kill Gaffer process to continue working.

- USD INSTANCE

Here is a post we already did on gaffer-dev : https://groups.google.com/g/gaffer-dev/c/7UxtpGPyngA/m/nKkePtOQBQAJ

This is the post we had send :

Hi everyone. I'm bumping this thread because we're in trouble with our project and usd instancing in Gaffer.

We have to replicate 300 source buildings on 27000 instances. We have a setup in Houdini that does it, and when exporting the usd, we end up with a very small file size (under 1Mb) tha seem to correctly reference our houses. However when loading it in gaffer and rendering it with Arnold it seems to unique every instance. Which according to this discussions is actually what happens.

Then I read this line from Don :

"It looks like it should be quite simple to use Prim::IsInstance and Prim::GetMaster in the USDScene::objectHash to prevent multiple objects being created on USD scene read."

Is this something that can be done without coding ? Because right now the process is really long and memory intensive, we sometimes wait over 45 minutes before the render actually starts, and it seem to totally fail on computers with a lower ram budget.

I have upload 2 USD files(one is the instance house, the others the instancer). I don't know why but gaffer crash directly when loading the USD instance file.

But I let you take a look to it, maybe be there is a read bug behind this issue.

The houdini file is also uploaded so you can see how it is setup.

I will try to send a working instance file ASAP.

We also tried to instance the houdini usd files with the gaffer instancer but we also get some issues to read some primvar needed to match the objects' index

Here is the post where I talked about it https://groups.google.com/g/gaffer-dev/c/b5L-D03L_UM/m/UY5LfdCtBQAJ.

It would be nice to have a fix for that.

Here is a drop box with the sample files :

https://www.dropbox.com/sh/oopl5170gkhnkuj/AAAJH6BLcHxxrsioIbnV0t_Xa?dl=0

Thx for your help,

Seb

[John Haddon]

On Thu, 24 Sep 2020 at 09:12, Sébastien Desmet <s.des...@gmail.com> wrote:

Hello,

Hi Sébastien

 

- interpretation of UVs who come from houdini are not working (we didn't test with other DCC ). we must use oslobject node to convert primvar from uv to st to make it work

I believe I have fixed this already. USD's standard name for texture coordinates is "st" and Gaffer's is "uv", so we now rename "st" to "uv" on loading. We also automatically correct the type (your file has a `float2` type whereas `texCoord2f` would be more accurate). With my development version I now get this screenshot from your example file (after disabling the OSLObject workaround), does that look correct?

 

we also get this annoying warning message all the time : WARNING : USDScene : Unknown type texCoord3f[] on PrimVar primvars:uv

This is also fixed, in that we do support the `texCoord3f` type now, so the warning is gone. But note that the warning is coming from a primvar called "uv" in the USD file. Because we now rename the "st" to "uv" on loading, the original `texCoord3f` "uv" is getting stomped over. Do you need us to retain it, and if so, what should we call it?

Then I read this line from Don :

"It looks like it should be quite simple to use Prim::IsInstance and Prim::GetMaster in the USDScene::objectHash to prevent multiple objects being created on USD scene read."

This is still on my todo list, but I should be getting to it soon. If you have any example data for this I would love to see it. (the example you sent already is a USDGeomPointInstancer, but what Don was talking about was USD native instancing : https://graphics.pixar.com/usd/docs/api/_usd__page__scenegraph_instancing.html).

We also tried to instance the houdini usd files with the gaffer instancer but we also get some issues to read some primvar needed to match the objects' index

Here is the post where I talked about it https://groups.google.com/g/gaffer-dev/c/b5L-D03L_UM/m/UY5LfdCtBQAJ.

It would be nice to have a fix for that.

I've done some work so that we now load USDGeomPointInstancers as a pointcloud that is more compatible with Gaffer's Instancer node. The only thing you should have to tweak now is to set the `prototypeMode` plug to the "Indexed (Roots Variable") on the Gaffer Instancer. With that, I get this from your example data :

Does that look like what you expect?

All these improvements are slated to be released in Gaffer 0.59, which we don't have a firm date for yet. But if there is some value in getting an early preview to you I should be able to sort something out.

Cheers...

John

[LJY]

Our team have an unresovled issue relates to point instancer in Gaffer. Dynamic point instancers in Maya won't be read correctly after importing to Gaffer, they remain static. 

We tried several methods to walk around this issue,  but all failed in the end. 

Hope you guys can fix this. 

The example files can be reached here in dropbox:

https://www.dropbox.com/sh/nrg3vsqa3plofpa/AABpQZigxsWhGJIdUFWzhiUaa?dl=0

[John Haddon]

On Fri, 25 Sep 2020 at 03:20, LJY <ljy...@gmail.com> wrote:

Our team have an unresovled issue relates to point instancer in Gaffer. Dynamic point instancers in Maya won't be read correctly after importing to Gaffer, they remain static.

 

Hi, thanks for the example data. I can confirm that this bug has been fixed already and the fix will be included in Gaffer 0.59 when it is released.

Cheers...

John

[s.des...@gmail.com]

 

- interpretation of UVs who come from houdini are not working (we didn't test with other DCC ). we must use oslobject node to convert primvar from uv to st to make it work

I believe I have fixed this already. USD's standard name for texture coordinates is "st" and Gaffer's is "uv", so we now rename "st" to "uv" on loading. We also automatically correct the type (your file has a `float2` type whereas `texCoord2f` would be more accurate). With my development version I now get this screenshot from your example file (after disabling the OSLObject workaround), does that look correct?

Yes it seems to be  fine

 

 

we also get this annoying warning message all the time : WARNING : USDScene : Unknown type texCoord3f[] on PrimVar primvars:uv

This is also fixed, in that we do support the `texCoord3f` type now, so the warning is gone. But note that the warning is coming from a primvar called "uv" in the USD file. Because we now rename the "st" to "uv" on loading, the original `texCoord3f` "uv" is getting stomped over. Do you need us to retain it, and if so, what should we call it?

I don't need to see the warnings 

Then I read this line from Don :

"It looks like it should be quite simple to use Prim::IsInstance and Prim::GetMaster in the USDScene::objectHash to prevent multiple objects being created on USD scene read."

This is still on my todo list, but I should be getting to it soon. If you have any example data for this I would love to see it. (the example you sent already is a USDGeomPointInstancer, but what Don was talking about was USD native instancing : https://graphics.pixar.com/usd/docs/api/_usd__page__scenegraph_instancing.html).

 

I have to check with our FX lead if he can do a example scene

 

We also tried to instance the houdini usd files with the gaffer instancer but we also get some issues to read some primvar needed to match the objects' index

Here is the post where I talked about it https://groups.google.com/g/gaffer-dev/c/b5L-D03L_UM/m/UY5LfdCtBQAJ.

It would be nice to have a fix for that.

I've done some work so that we now load USDGeomPointInstancers as a pointcloud that is more compatible with Gaffer's Instancer node. The only thing you should have to tweak now is to set the `prototypeMode` plug to the "Indexed (Roots Variable") on the Gaffer Instancer. With that, I get this from your example data :

Does that look like what you expect?

I will need to do a better examples as our main issues is to have the correct instance on a specific point. in this example there is only one house 

All these improvements are slated to be released in Gaffer 0.59, which we don't have a firm date for yet. But if there is some value in getting an early preview to you I should be able to sort something out.

I have lot of issues with my rendering, some frames randomly stop during the warning messages (there are thousands of them) and there is no error message. But I don't know if the warning messages is the problem.

It would be interesting to see with your update if the problem will be fixed or not. The sooner for me is the best :)

 

Thx a lot,

[Heribert Raab]

I believe the issue here is trying to render from a camera that is not in the `__cameras` set. See :

https://github.com/GafferHQ/gaffer/issues/3378

https://github.com/ImageEngine/cortex/pull/1034#issuecomment-584096107

I was using the cameras set. and it worked in interactive rendering , but the batch render drop seg fault instant crash. Unfortunately, i can't provide the example file anymore.

I should be looking at populating the `__cameras` set automatically from USD and ABC data soon.

that's great.

reading attr into gaffer, it takes vector x,y,z, standart vector. arrays like color[0],color[1],color[3] are ignored.

yes, fixed-length for 2,3,4 would great some DCC using it as default. it should read the data anyway then we able to convert/use the data with OSL etc.. Houdini style, read any data and then wrangle your data into the correct space

We don't have any array-of-arrays data type in Gaffer. Are the arrays you are interested in always of a fixed length, say, 2, 3 or 4? If so we may be able to convert to arrays of V2f/V3f/Color4f for you. Bear in mind though that it is preferable to encode the type (vector/normal/point) explicitly as it has a bearing on how data is transformed.

 

quaternion or vector 4 not readable, but you need quaternion for instancing right? so have assembled it first with a quaternion node.

Quaternions are already supported for reading in Gaffer for both USD and ABC files. If you have example data that doesn't load, please provide it. Bear in mind that Houdini's writing of quaternions to Alembic is/was broken - they round trip in and out of Houdini OK but the quaternions are stored in the wrong order according to the Alembic spec :

https://www.gafferhq.org/community/?place=msg%2Fgaffer-dev%2FoK79ijYEyr0%2F8DHNqUZGBwAJ

I believe the SideFX issue for that is #92479 if you want to see if it has been fixed. Gaffer's Orientation node has a mode that you can use to fix the loaded data if necessary :

the main problem to get vector4 data into gaffer. it was quite complicated to the right way to convert vector4 in Houdini. fixed array 2,3 or 4 like Houdini does by default would cool. basically, we just wanna get default data from Houdini into gaffer without wrangle data types.

once you manage to export a point cloud or particle in way gaffer can read, the attr comes in as facing varying?  with only points in the cache file.
-->you can't use unless you add osl object node and convert everything to vertex varying.

Can you provide some example data to demonstrate this? I've seen similar things in the past, and it has always been the case that the Alembic file specified that, and rightly or wrongly we were loading at as specified. Note that Gaffer has a ResamplePrimitiveVariables node that is an easier and more efficient way of modifying the interpolation than going through OSLObject.

 

read spline is more a gamble game.  polylines, b-splines, NURBS curves? something all works something none of it works. you lucky if in Houdini and easily shuffle data around, but if use any other software, you out of luck.

I have recently improved our USD curve loading considerably, but it hasn't been released yet :

https://github.com/ImageEngine/cortex/pull/1070

What I don't have yet is an example of `pinned` curves. Are you able to export some bSpline or catmullRom splines out of Houdini in pinned form?

it seems gaffer expect very clean data, but it should able to load the data from any mayor DCC like Houdini, Maya, blender, substance designer etc sometimes you export the data after working with the scene and you can't get the data into gaffer anymore. maybe there is extra metadata or wrong datatype in the file. but it's hard to find out why the files not coming in anymore. basically, a solid roundtrips of data would be cool. data is coming from a different department with different DCC's  

 

alembics with hierarchy is another gambling game.

Please elaborate. Both USD and Gaffer support the idea of the same location having both geometric data and a transform, partly for convenience and partly for performance. Alembic has no such concept - it is closer to Maya, where a location is either a shape or a transform but not both. When loading Alembics, we collapse single shapes into their transforms wherever possible. Is that what you're talking about?

 

this is an export problem, from Houdini it exports the shape without a transform node. it would cool if the gaffer could group nodes, so data don't get loaded at first sight, only after expanding the group in the hierarchy view

thanks for listening, i will have more example files later on...

[br1...@gmail.com]

Then I read this line from Don :

"It looks like it should be quite simple to use Prim::IsInstance and Prim::GetMaster in the USDScene::objectHash to prevent multiple objects being created on USD scene read."

This is still on my todo list, but I should be getting to it soon. If you have any example data for this I would love to see it. (the example you sent already is a USDGeomPointInstancer, but what Don was talking about was USD native instancing : https://graphics.pixar.com/usd/docs/api/_usd__page__scenegraph_instancing.html).

 

I have to check with our FX lead if he can do a example scene

 

Hi John,

Here are some other instancing methods outputted from Houdini.

https://www.dropbox.com/sh/rd7epzey43e7dkg/AABT2PAMxqdM7qF1ouB_sc9-a?dl=0 

reference_houses.usd is using the method : reference in Houdini, here is the detail about this method from the Houdini documentation :

Creates a separate prim at each point that references in the "prototype" prim(s). These references take up more storage space than instanceable prims, but you can edit/override the descendant prims.  

instancable_reference_h…ses.usd is using the method : Instanceable Reference in Houdini, again here is the description from Houdini documentation :

Creates an instanceable prim at each point. These are separate prims in the scene graph tree, and can individual overrides on each top-level prim, but they share a "shadow" copy of their descendant prims, so their descendants are not editable or individually override-able.  

Here is a link to the documentation about the Instancer :

https://www.sidefx.com/docs/houdini/nodes/lop/instancer.html

If you need any sort of other method of instancing, feel free to ask, we'll upload them.

Cheers,

Bruno

 

[John Haddon]

On Mon, 5 Oct 2020 at 13:21, br1...@gmail.com <br1...@gmail.com> wrote:

Here are some other instancing methods outputted from Houdini.

https://www.dropbox.com/sh/rd7epzey43e7dkg/AABT2PAMxqdM7qF1ouB_sc9-a?dl=0

Thanks Bruno,

Using your `instanceable_reference_houses.usd` scene for testing, I've made some improvements to USD loading to take advantage of the instancing. For your example scene, this reduces cache memory usage in Gaffer by 90% and knocks about 20% off scene generation time. As you add more and more instances, I would expect these improvements to become more significant. You can see the changes here if you're interested :

https://github.com/ImageEngine/cortex/pull/1086

I've also found a simple improvement to Gaffer's SceneReader to take even better advantage of this - for this test scene that reduces peak memory usage by 60% and further reduces scene generation time by another 50%.

Cheers...

John

[Sébastien Desmet]

Hello,

Here is a feedback on the improvement of the USD in gaffer 0.59 beta2

We have send a render to the renderfarm with gaffer 0.59 beta2 and it took 15 minutes to render the images ! With gaffer 0.58 it took 1hour30 !

The USD now are loaded really faster.

Are there any chance to see an "official" release of gaffer 0.59 soon ?

Thx for the effort you put in the gaffer development !!

Cheers,

Seb

[John Haddon]

On Thu, Nov 12, 2020 at 8:53 AM Sébastien Desmet <s.des...@gmail.com> wrote:

Here is a feedback on the improvement of the USD in gaffer 0.59 beta2

We have send a render to the renderfarm with gaffer 0.59 beta2 and it took 15 minutes to render the images ! With gaffer 0.58 it took 1hour30 !

The USD now are loaded really faster.

Good news! Thanks for letting us know. I've done some other work that I think may speed things up even more when you're using a lot of USD instancing. I've been unable to get anyone to test it in production yet so it is sitting on a back burner at https://github.com/GafferHQ/gaffer/pull/3944. If you have a GitHub login you can go to the "Checks" tab on that page and download the build from the "Artifacts" menu. If you're able to test it at all I'd be really interested to see what you find.

 

Are there any chance to see an "official" release of gaffer 0.59 soon ?

 

We still don't have an ETA on that I'm afraid, but we have a meeting this week that may shed some light. If the beta is working for you then don't be afraid to use it though. We released it as a beta mostly because of the risk associated with the Qt 5.12 upgrade, and because we wanted to make the improved USD support available to you earlier. We may also make some ABI breaks before final, but those shouldn't affect source compatibility or present a barrier to adoption. In other respects it is no less risky than any other version.

Cheers...

John

[s.des...@gmail.com]

Hello John,

We have continued our test in gaffer 0.59b3 and we have a new issue.

It seems that the USD instances exported from houdini can't be rendered. we can see the hierarchy of the objects in gaffer but we don't see anything in the viewport or in the rendering.

Also, the hierarchy of the objects have changed. all the missing object are under a group called "prototype". For a reason that I don't know some objects are outside of the prototype group and these one can be rendered

Do you have an idea of what can be the problem ?

Thx,

Seb

[s.des...@gmail.com]

Ok we've fixed the issue, it was a houdini export issue. Now we can see all our city rendered. it took 30 min to load all the caches

There is only one thing who is not working and I don't really know if it is a bug but we can't see USD instances in the gaffer viewport (all objects under prototype group)

Seb

[John Haddon]

Hi Seb,

Glad you've got your city rendering. If you are able to send a small example USD file which shows the visibility problem then I will take a look. Also, I would love to know if the builds for https://github.com/GafferHQ/gaffer/pull/3944 give you any performance improvements when loading your city : in a test I did here it made quite a difference.

Cheers...

John

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/c923c02e-ed80-4a63-9f43-97f4f0bfd46dn%40googlegroups.com.

[s.des...@gmail.com]

Hello John,

I still have to search also, it is not yet clear for me what is visible or not. the USD prototype hierarchy is messing up a bit the original hierarchy (I have prototype group everywhere). I'm come back to you asap with small example

For the builds that you want me to test I will also try to si it asap. we have some deadline on our prod at the moment. it will be a bit later

Thx,

Seb

[Carlo Giesa]

Hi John!

This might be a stupid question, but I'm not very familiar with compilation and how the building procedure is setup for Gaffer. I can see that your pull request comes from your github, so I guess, as long as the merge has not taken place, that the GafferHQ github does not know about it. I tried to run following command in the cloned gaffer/build repo:

python ./build.py --version b25ec185da22aee2e3c1b6962a0717426c2e14fc

This is the hash key of your commit that you created the pull request from. It did compile and I can open this compiled Gaffer. But is this the correct one and how is it possible that this hash key is understood from the gaffer side?

During the building process, I see following line:

curl -L https://github.com/GafferHQ/gaffer/archive/b25ec185da22aee2e3c1b6962a0717426c2e14fc.tar.gz > gaffer-b25ec185da22aee2e3c1b6962a0717426c2e14fc-source.tar.gz

Is this a dynamic tar.gz that gets generated on the fly or are there archived tars for all commit which would sound crazy to me?

Sorry if this is all obvious.

Greets,

Carlo

You received this message because you are subscribed to a topic in the Google Groups "gaffer-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gaffer-dev/mH8KCMULURg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/656b187d-a992-4a59-99b2-4582e0e3f5f2n%40googlegroups.com.

[Tom Cowland]

Hey Carlo,

Glad you got it building.

> This is the hash key of your commit that you created the pull request
> from. It did compile and I can open this compiled Gaffer. But is this
> the correct one and how is it possible that this hash key is
> understood from the gaffer side?

Yeah that should be right. I think it's GitHub being nice as its
associated with an open PR. If you check the link in a browser it gives
you a helpful message:


https://github.com/GafferHQ/gaffer/commit/b25ec185da22aee2e3c1b6962a0717426c2e14fc

> Is this a dynamic tar.gz that gets generated on the fly or are there
> archived tars for all commit which would sound crazy to me?

Yeah, its a nice feature of GitHub, it does it on the fly, and lets you
easily grab the tree at any point.

If its any help, we usually have CI builds for all PRs (assuming they
haven't failed). Since we moved to Actions they're sadly a lot harder
to find though.

If you click on the `Checks` tab on a PR, then on the `CI` heading, you
should be able to download the artifacts, as long as you're logged in.
The ones for that PR are here, for example:

https://github.com/GafferHQ/gaffer/actions/runs/359351821

Annoyingly GitHub insists on wrapping the tar archive, in a zip, on the
fly, so it takes a while to build that before the download starts.

Thanks again,
'best
Tom

[Carlo Giesa]

Oh wow! Thanks a lot Tom for all those insights! I will check that out.

Greets,

Carlo

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/4e13ef088bc3744b123adca16eba8c35c4fbdae6.camel%40cinesite.com.

[s.des...@gmail.com]

Hello,

I got an issue with my USD caches and I'm not sure to know where the problem comes from ?

If is it a USD cache issue or a Gaffer issue?!

I have multiple usd cache with different types of building for each. if I render everything together (loading is really faster now compared to 0.58) but I have one cache where the building faces are popping during the render at random frames.

If I render the problematic cache alone, the render is fine.

All the caches comes under a specific group so there is no clashing names between the caches as far as I know.

Do you know what can be the problem ?

PS : I would like to post some print screen to show you what I'm talking about but it seems that since google group UI changes I can't add images or attachments to the messages anymore. The buttons are missing.

I don't have the problem on other google group so I guess it is a settings to switch on

Thx,

Seb

[John Haddon]

Hi Seb,

I've just been through all the settings for the group, and I can't find anything related to enabling images or attachments, sorry. And weirdly, I still have the attachment buttons in the UI here. Perhaps you are still able to paste images inline?

Is it at all possible that you have two copies of the same building in the same place? The z-fighting from that could possibly explain the symptoms you describe...

Cheers...

John

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/edad3769-005d-42dc-8d45-a33254e59c07n%40googlegroups.com.

[s.des...@gmail.com]

I can't add images, I have tried on 2 differents computer and one with firefox and the other with chrome.

Before google group change its UI it worked

here is a dropbox link with a render of the same house. you will see the popping of the face on one version and that when we render all the caches together

https://www.dropbox.com/sh/ayz789nobbxe1cd/AABBemKQhIoHRVluS3Z0j7yha?dl=0

Thx,

Seb

[John Haddon]

From the images, it looks as if all the walls on the second floor have disappeared. Is that correct? And in the bad frames, do they always consistently disappear together, or do some pop on and off independently? Are they all one object, or several objects?

I'm struggling to come up with a theory as to what might be happening here. It might be useful to try rendering with highly simplified shading to rule that out as a possibility. And to simplify the Gaffer network so it is as close to just a SceneReader as possible, to see if that reveals a culprit. You might also try repeatedly writing ass files with the ArnoldRender node, and doing a diff to see if anything changes between them. Or writing an ass file and repeatedly rendering it, to see if Gaffer/USD are consistent and there is an intermittent Arnold problem.

Sorry not to have any better ideas...

Cheers...

John

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/2620cc52-4456-4120-a091-afbd5fad2a77n%40googlegroups.com.

[Sébastien Desmet]

Hello,

all the objects are separated and you see it correctly the wall disappears but it is random, some frames are correct and on other frames it is an other parts who are popping...

I have disable one by one all the assets and it seems that when a particular asset is loaded I have the issue. but this asset is not linked directly to the procedural usd city.

All assets (camera, lights, assets in abc and usd, FX vdb) are connected to a parent node  

Maybe you can give me some tips because I'm not always sure what is the best technique to merge multiple assets together

Is it best to use mergeScene, groups or parents ?

I'm not sure to really control the mergescene node as sometimes I have to change the order of the plug to see all my objects. 

For instance, if I plug a camera, a light and some objects, I have to plug the camera it in the first plug to be able to use at render time

Thx, 

Seb

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/CAB8pVgKU5EGBYSvJ7fiTqHX1C3PiwhHEZbYe4dNbMs%2BVJ-PNLA%40mail.gmail.com.

[s.des...@gmail.com]

I have found  that in an asset I have some USD buildings placed by hand. The same buildings are used to generate the procedural city but there is no direct link between the Usd placed by hand and the ones generated procedurally. And the buildings who have issues are not the ones used on the other asset.

If I disable the USD placed by hand, the issues is gone...

Is is possible that there is kind of a link between the USD ?

thx,

Seb

[LJY]

Hi, it's me again. We've ran into an issue while loading usd file to gaffer. 

There are 3 mesh included in this file, but only one can be sucessfully loaded into gaffer. We‘’ve tried with several other usdf files, and it happened for all cases.

You can find the usd file here: https://www.dropbox.com/s/1fculafsg1pmer6/20210420_usd_primvar_issue.rar?dl=0

Best,

Mavis

[John Haddon]

Hi,

What's tripping Gaffer up here is the unusual `int[] primvars:displayColor:indices = [0]` on those meshes. If you remove those it doesn't change the meaning of the file at all, and it will load OK. Alternatively, you could use a DeletePrimitiveVariables node in Gaffer to delete "Cs" (which is what we convert "displayColor" to). We should probably try hard to accomodate this automatically in Gaffer, so please also feel free to open an issue on the GitHub tracker.

Cheers...

John