Support for Arnold 5 Procedural: namely Yeti

[Sam Hodge]

Hello Gafferians, 

I just put on 0.42.0.0 and it works a treat under Linux, for rendering out assets with Arnold 5.

I was so happy to be able to render VDB out of the box.

It seems that there are some useful utility nodes for shaders in the mix too between 0.40.0 and 0.42.0, which was the last version I tried

I have one more question. is there a way of hosting an Arnold procedural in a Gaffer graph?

With Alembic, OpenVDB we have most of our bases covered, but we will still need hair in the form of Yeti procedural, and I cannot see a way of plugging a procedural into the mix easily.

Thanks in Advance

Sam

PS. While I have you here, why dont Alembic camera work out of the box, it seems that you have to create a native camera and expression link it to the camera from an Alembic Reader.

PPS. I watch the 2016 Digi Pro video and read the paper today, thanks for that, so is the "no manipulators for lights in the open source version" still a thing, or has this been addressed?

[Sam Hodge]

I found this:

doc/gaffer/html/Reference/NodeReference/GafferScene/ExternalProcedural.html 

But I am a little muddled. In htoa I have followed an existing pattern to make it work pretty well and it is all wrapped up so you give it a .fur file and some attributes, and you can get nice motion blocks of fur.

I know about the fact that in Arnold 5 the behaviour of procedurals has become that each procedural is its own node.

But some gentle guidance would be excellent.

[Andrew Kaufman]

ExternalProcedural is indeed the node you want. You might want to see if you can get it working with a slightly simpler procedural (or at least one where you have source code access) before trying Yeti. As I recall with our brief experiment using ExternalProcedurals at IE, we ran into issues because of a difference in sample times between what Gaffer sends to Arnold and what the procedural expected (it might have been Golaem rather than Yeti). It should do the trick though, possibly with a few minor code changes.

>  why dont Alembic camera work out of the box

See a related post from Brian titled "Alembic files and SceneReader node" where he mentions sticking the Alembic camera into the (semi-secret) "__cameras" Set. You can do that with a Sets node for now. We'll need to make a change in Cortex to automate that in the future.

> is the "no manipulators for lights in the open source version" still a thing, or has this been addressed?

We do have basic manipulators in Gaffer now, you can find them on the left-hand side of the viewer. Just the basic TRS for now though, we don't have the fancier features yet (e.g. snap-to-vertex). Keep in mind, if you're reading lights from a cache file (as opposed to dropping down light nodes in the graph) you'll need to add a Transform node filtered to your light location. The manipulators will look upstream of the viewed node and make a best guess where to store the values. The biggest thing we're missing currently is a convenient way of moving a light (or camera) while you're looking through it. We are hoping to get that one in sooner, as its become a priority at IE, but I can't promise a timeline just yet.

Cheers,

Andrew

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrew Kaufman - R&D Lead

Image Engine

studio: +1 (604) 874-5634 | and...@image-engine.com | www.image-engine.com



15 West 5th Avenue, Vancouver, BC, V5Y 1H4, Canada

If you are not the intended recipient, disclosure, copying, distribution and use of this email is prohibited. Please notify us immediately and delete this email from your systems. You may contact us at in...@image-engine.com if you do not wish to receive further commercial electronic messages. We may still send you messages for which we do not require consent.

[Sam Hodge]

> is the "no manipulators for lights in the open source version" still a thing, or has this been addressed?

We do have basic manipulators in Gaffer now, you can find them on the left-hand side of the viewer. Just the basic TRS for now though, we don't have the fancier features yet (e.g. snap-to-vertex). Keep in mind, if you're reading lights from a cache file (as opposed to dropping down light nodes in the graph) you'll need to add a Transform node filtered to your light location. The manipulators will look upstream of the viewed node and make a best guess where to store the values. The biggest thing we're missing currently is a convenient way of moving a light (or camera) while you're looking through it. We are hoping to get that one in sooner, as its become a priority at IE, but I can't promise a timeline just yet.

Agreed, its been a few years since I have lit shots for a living but being able to place a light by looking through it is pretty important.

But if you can look through it an place an aim and position, its the same thing almost, just a bit clumsier.

No need to make a promise, its good to know it is on the cards, I have done some interactive OpenGL before, so if I am really desperate, I can work out how to build this thing and see if I can do it myself. What sort of context does the OpenGL in the viewport run in? 

[Sam Hodge]

Do the good folks at Gaffer HQ have an example of a gaffer scene that hosts the Arnold procedural Mandelbulb?

That would certainly help to cut my teeth so to speak on hosting a procedural in Gaffer.

Sam

[dan...@image-engine.com]

I don't currently have an env handy with HtoA available ( which I think is where mandelbulb comes from ), but it should be as simple as just dropping an ExternalProcedural node and setting "fileName" to "mandelbulb" ( the name "fileName" is a bit confusing in Arnold 5 where procedurals are specified via name rather than path, we do have a plan to fix this sometime).

I just tested with setting "fileName" to "cone" ( one of the built in shape types ), and that worked fine.

[Sam Hodge]

The cone tip was useful

so is this:

kick -info cone

node:         cone

type:         shape

output:       (null)

parameters:   24

filename:     <built-in>

version:      5.0.2.3

Type          Name                              Default

------------  --------------------------------  --------------------------------

BYTE          visibility                        255

BYTE          sidedness                         255

BOOL          receive_shadows                   true

BOOL          self_shadows                      true

BOOL          invert_normals                    false

FLOAT         ray_bias                          1e-06

MATRIX[]      matrix                            (empty)

ENUM          transform_type                    rotate_about_center

NODE[]        shader                            (empty)

BOOL          opaque                            true

BOOL          matte                             false

BOOL          use_light_group                   false

NODE[]        light_group                       (empty)

BOOL          use_shadow_group                  false

NODE[]        shadow_group                      (empty)

STRING[]      trace_sets                        (empty)

FLOAT         motion_start                      0

FLOAT         motion_end                        1

UINT          id                                0

VECTOR        bottom                            0, 0, -0.5

VECTOR        top                               0, 0, 0.5

FLOAT         bottom_radius                     0.5

FLOAT         top_radius                        0

STRING        name   

 samh@bladerunner ~/gaffer/projects/default/asses/fuzz/ run in htoa/latest yeti/latest : kick -info pgYetiArnold

node:         pgYetiArnold

type:         shape

output:       (null)

parameters:   33

filename:     /asset/common/software/thirdparty/yeti/2.2.3-build2/arch/linux-any/x86_64/maya2017/lib/pgYetiArnold.so

version:      5.0.1.3

Type          Name                              Default

------------  --------------------------------  --------------------------------

BYTE          visibility                        255

BYTE          sidedness                         255

BOOL          receive_shadows                   true

BOOL          self_shadows                      true

BOOL          invert_normals                    false

FLOAT         ray_bias                          1e-06

MATRIX[]      matrix                            (empty)

ENUM          transform_type                    rotate_about_center

NODE[]        shader                            (empty)

BOOL          opaque                            true

BOOL          matte                             false

BOOL          use_light_group                   false

NODE[]        light_group                       (empty)

BOOL          use_shadow_group                  false

NODE[]        shadow_group                      (empty)

STRING[]      trace_sets                        (empty)

FLOAT         motion_start                      0

FLOAT         motion_end                        1

UINT          id                                0

BOOL          override_nodes                    false

STRING        namespace                         

INT           frame                             1

INT           verbosity                         0

FLOAT         density                           1

FLOAT         width                             1

FLOAT         length                            1

STRING        filename                          

STRING        imageSearchPath                   

BOOL          frameRelativeSamples              false

BOOL          setDeformTimeSamples              false

BOOL          setFrameRelativeDeformTimeSam...  false

FLOAT[]       samples                           (empty)

STRING        name   

but the problem is how procedurals are used in gaffer, it assumes that there is a one to one relationship between a procedural and ginstance.

This relationship doesnt hold true for pgYetiArnold


you will get multiple curve nodes as follows

### created by instance:ceead697a99de4a4c02b3afcd84474b0

curves

{

 name |hair_grp|nodes|yeti_hair|yeti_hairShape|scatter1_geoShape_grow1

 visibility 0

 self_shadows on

 matrix

 1 0 0 0

 0 1 0 0

 0 0 1 0

 0 0 0 1

 opaque on

 motion_start 0

 motion_end 1

 num_points 49611 1 b85UINT

B!(4QdX$$%`q(4Qd

 points 644943 1 b85VECTOR

8.MU38t=GLb-9dV8.MU38t=GLb-9dV8/YhT8t=C)b-I%u8/JP*8t@R6b-X6r80gAi8t@QOb-gIM81'X58t>_wb-w)980C@v8t?J/b.1Vo813Xb8tE%.b.@@.83C,88tIM2b.Nrw84U.08tJ+Vb.^/684R=s8tIDEb.m[D83U'N8tHiWb/(6183U'N8tHiWb/(618>;0-8cX[a8>=I58>;0-8cX[a8>=I58=qYD8cQ+r8;HMO8>3w58cLM087q^28>xux8cBMH82H7c8?2G&8c2[_8-5]c8>Yu_8c'p'7xl)G8>=S58c$k77g)qB8>NMI8bs>d`d1/?8>IOb8bcVUa**A78>bS18bVGFa4jR`8>VrZ8bTJqa;vQg8>VrZ8bTJqa;vQgai1+h8kF//8NI4Bai1+h8kF//8NI4BaiT@Q8k?I)8MJKYai]T:8k>qr8L9%>aibq28k<S\8K&9Oaienh8k<+w8IfPAaimaJ8k>F]8HSa.aj)fq8k>D58GGDKaj9^.8k:lU8EiNtajD\i8k4Q38CNQ8ajOhD8k/am8A2tuaj`E;8k.=_8>o&vaj`E;8k.=_8>o&vaaO;x8q8Vrb-]rcaaO;x8q8Vrb-]rcaa@`78q8q_b-mh6aaG%/8q<pOb.'>[aaEAG8qBD(b.5-Faa@w38qES_b.CD3aaG%q8qF,Nb.S.taaKZ%8qHSbb.bAkaaJ4]8qMI4b.plDaaAdD8qQ1[b/*F.aa0>I8qQaeb/9D,a`xRW8qQOvb/HQAa`xRW8qQOvb/HQAaaO>38sdHZb)PS\aaO>38sdHZb)PS\aa:6V8saL6b)a;MaaL^28sbssb)qsGaaO+E8sf...

and the bummer is that the visibility of the procedural is 0 which is good because you need to pair it up the a ginstance.

So is there a way to turn off the instance behaviour for the ExternalProcedural node?

[john haddon]

On Tuesday, February 20, 2018 at 5:10:55 AM UTC, Sam Hodge wrote:

but the problem is how procedurals are used in gaffer, it assumes that there is a one to one relationship between a procedural and ginstance.

It's not necessarily one-to-one; if you have two identical copies of the same procedural in your scene, then you will get a single procedural and two ginstances. That's the reason we generate ginstances at all, to be able to more efficiently output scenes when there are duplicates...

 

This relationship doesnt hold true for pgYetiArnold

you will get multiple curve nodes...and the bummer is that the visibility of the procedural is 0 which is good because you need to pair it up the a ginstance.

Sorry, I don't understand the problem here, and I don't have Yeti to hand to make my own investigation. The procedural visibility is off because it is the "instance master" : we hide it so that it is only the ginstances that you see. It looks like your example output perhaps came from `kick -resave -forceexpand`? I'm not 100% sure that is accurate when instanced procedurals are involved. We have run into other bugs relating to procedural instancing in the past, because it seems we push Arnold harder in this respect than most exporters.

Assuming the visibility thing is a red herring, another thing to check is that the motion_start and motion_end parameters of your procedural are set to match Gaffer's shutter (which you'll see on the options node). I think this was the trickiest problem we encountered when setting up the Yeti procedural in the past...

Cheers...
John

[dan...@image-engine.com]

On Tuesday, 20 February 2018 01:41:38 UTC-8, john haddon wrote:

On Tuesday, February 20, 2018 at 5:10:55 AM UTC, Sam Hodge wrote:

but the problem is how procedurals are used in gaffer, it assumes that there is a one to one relationship between a procedural and ginstance.

It's not necessarily one-to-one; if you have two identical copies of the same procedural in your scene, then you will get a single procedural and two ginstances. That's the reason we generate ginstances at all, to be able to more efficiently output scenes when there are duplicates...

If I'm understanding Sam correctly, he's referring to us assuming that each procedural outputs a single node, which we can then ginstance.

If the Yeti procedural expands into many separate nodes, we would currently only ginstance one of them, right?  That sounds like a problem. 

[John Haddon]

On 20 February 2018 at 17:57, <dan...@image-engine.com> wrote:

On Tuesday, 20 February 2018 01:41:38 UTC-8, john haddon wrote:

On Tuesday, February 20, 2018 at 5:10:55 AM UTC, Sam Hodge wrote:

but the problem is how procedurals are used in gaffer, it assumes that there is a one to one relationship between a procedural and ginstance.

It's not necessarily one-to-one; if you have two identical copies of the same procedural in your scene, then you will get a single procedural and two ginstances. That's the reason we generate ginstances at all, to be able to more efficiently output scenes when there are duplicates...

If I'm understanding Sam correctly, he's referring to us assuming that each procedural outputs a single node, which we can then ginstance.

But we don't assume anything about what a procedural outputs do we? We just make the procedural node (and a single ginstance referring to it), and then leave Arnold to expand it later - we know nothing about the contents at all.

 

If the Yeti procedural expands into many separate nodes, we would currently only ginstance one of them, right?  That sounds like a problem.

No, we never ginstance the contents of the produral, regardless of how many separate nodes it might generate. Arnold alone deals with the contents...

[Sam Hodge]

I will see if I can get a smaller number of non show sensitive curve files and then I can walk through the issue I am seeing.

I am sure it is a case of me misunderstanding something.

Sam

[dan...@image-engine.com]

I did actually manage to find a Yeti cache that someone had lying around here, and I seem to be seeing the same issue you are with a visibility problem - though John's point seems correct, this should.

I'll try to investigate further later today.

[dan...@image-engine.com]

OK, got a somewhat clearer understanding now, though I'm not sure whether it's a bug in Arnold, or Yeti, or if we'll need to export procedurals differently in Arnold 5.

If we take the following ass file:

"""

pgYetiArnold

{

 name instance:yeti

 visibility 0

 verbosity 1

 filename "<someValidYetiFile>.fur"

 samples 0

}

    

ginstance

{ 

 name /group/procedural

 visibility 255

 matrix

 1 0 0 0

 0 1 0 0

 0 0 1 0

 10 0 0 1

 node "instance:yeti"

}

 

mandelbulb

{

 name instance:mandel

 visibility 0 

 gridsize 3

 chunks 1

}

 

ginstance

{

 name /group/testProcedural

 visibility 255

 matrix

 1 0 0 0

 0 1 0 0

 0 0 1 0

 10 0 0 1

 node "instance:mandel"

}

"""

And run it through this:

kick -forceexpand -resave /tmp/expand.ass /tmp/test.ass

expand.ass now contains:

"""

### created by instance:yeti

curves

{

 name |pgYetiMaya1|pgYetiMaya1Shape|scatter0_pSphereShape1_grow0

 visibility 0

 self_shadows on

 matrix

 1 0 0 0

 0 1 0 0

 0 0 1 0

 0 0 0 1

 opaque on

    <CURVE DATA>

}

### created by yetiInstance

curves

{

 name |pgYetiMaya1|pgYetiMaya1Shape|scatter0_pSphereShape1_grow0

 visibility 0

 self_shadows on

 matrix

 1 0 0 0

 0 1 0 0

 0 0 1 0

 10 0 0 1

 opaque on

    <CURVE DATA>

}

### created by instance:mandel

points

{

 visibility 0

 <POINT DATA>

}

### created by mandelInstance

points

{

 matrix

 1 0 0 0

 0 1 0 0

 0 0 1 0

 10 0 0 1

 <POINT DATA>

}

"""

So, we've setup the two procedurals identically: both have an invisible definition, and then a ginstance transforms by 10 units and turns on visibility.

The mandelbulb works fine, but the yeti instance does not pick up the overridden visibility.  Its also interesting that the instance of the yeti node gets the same name as the original Yeti node.

So it looks like it doesn't work to instance Yeti procedurals in Arnold 5.  ( I think we tested this at some point in Arnold 4 and it worked. )

So I guess the first question is whether instancing procedurals in Arnold 5 is expected to work - if so, maybe Yeti is doing something wrong?  I guess I should probably contact SolidAngle.

[Sam Hodge]

Legendary Effort Dan!

You see what I see so I didn't make a mess of it after all.

Can you cc me on or report back on any finding with Solid Angle or Peregrine. 

Thanks for all the work you have put in helping me out.

sam

[Sam Hodge]

The good blokes at Solid Angle got back to me and confirmed what they expected to be true is that you can instance a procedural that is hidden and the instances should override the visibility of the instancee, as you described with the Mandelbulb. They did this with an XGen procedural, which I assume spawns more than one polymesh. I havent confirmed this on my end.

So I will get in contact the Peregrine and see if Yeti needs as tune up, or if we missed an argument or some other source of usage error.

I will keep you updated.

[John Haddon]

Seems like we're getting somewhere. Daniel and I were discussing this last night, and we think it'd be quite reasonable to support a parameter on the ExternalProcedural to turn off instancing, for dealing with problems like this. If you don't get anywhere with Peregrine then I suggest we do just that. If you'd like to give this a go yourself, take a look here, where we decide whether or not we can instance something :

https://github.com/GafferHQ/gaffer/blob/master/src/GafferArnold/IECoreArnoldPreview/Renderer.cpp#L1451

You'd want to add a `runTimeCast<ExternalProcedural>()` to find external procedurals, and then check for a boolean parameter named "ai:allowInstancing" or something like that, and use that to drive the result. If that doesn't make sense just give us a shout, or we can take a stab ourselves...

Cheers...

John

--

You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

John Haddon - R&D Programmer

Image Engine

studio: +1 (604) 874-5634 | jo...@image-engine.com | www.image-engine.com

[Sam Hodge]

It is quite an undertaking to navigate the gaffer source code without documents to describe the class hierarchy.

Would it be possible to create Sphinx document from the souce code for gaffer much in the same way there are docs for the iecore docs?

Sam

To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sam Hodge]

that being said I was able to render some yeti in Gaffer with the following hack

diff -ru gaffer-0.44.0.0/src/GafferArnold/ArnoldAttributes.cpp gaffer-0.44.0.0-orig/gaffer-0.44.0.0/src/GafferArnold/ArnoldAttributes.cpp

--- gaffer-0.44.0.0/src/GafferArnold/ArnoldAttributes.cpp       2018-02-23 14:28:20.387979924 +1030

+++ gaffer-0.44.0.0-orig/gaffer-0.44.0.0/src/GafferArnold/ArnoldAttributes.cpp  2018-02-15 09:07:11.000000000 +1030

@@ -91,8 +91,6 @@

        attributes->addOptionalMember( "ai:shape:step_size", new FloatPlug( "value", Plug::In, 0.0f, 0.0f ), "volumeStepSize", false );

        attributes->addOptionalMember( "ai:shape:volume_padding", new FloatPlug( "value", Plug::In, 0.0f, 0.0f ), "volumePadding", false );

 

-        //Yeti Hack

-        attributes->addOptionalMember( "ai:allowInstancing", new IECore::BoolData( true ), "allowInstancing", false );

 

diff -ru gaffer-0.44.0.0/src/GafferArnold/IECoreArnoldPreview/Renderer.cpp gaffer-0.44.0.0-orig/gaffer-0.44.0.0/src/GafferArnold/IECoreArnoldPreview/Renderer.cpp

--- gaffer-0.44.0.0/src/GafferArnold/IECoreArnoldPreview/Renderer.cpp   2018-02-23 16:26:27.202067058 +1030

+++ gaffer-0.44.0.0-orig/gaffer-0.44.0.0/src/GafferArnold/IECoreArnoldPreview/Renderer.cpp      2018-02-15 09:07:11.000000000 +1030

@@ -50,7 +50,6 @@

 #include "IECoreScene/Camera.h"

 #include "IECoreScene/CurvesPrimitive.h"

 #include "IECoreScene/ExternalProcedural.h"

-#include "GafferScene/ExternalProcedural.h"

 #include "IECoreScene/MeshPrimitive.h"

 #include "IECoreScene/Shader.h"

 #include "IECoreScene/SpherePrimitive.h"

@@ -670,7 +669,6 @@

 IECore::InternedString g_linkedLights( "linkedLights" );

 IECore::InternedString g_lightFilterPrefix( "ai:lightFilter:" );

 

-IECore::InternedString g_allowInstancing("ai:allowInstancing");

 

 class ArnoldAttributes : public IECoreScenePreview::Renderer::AttributesInterface

 {

@@ -1449,25 +1447,6 @@

                }

 

        private :

-               template<typename T>

-               static const T *attribute( const IECore::InternedString &name, const IECore::CompoundObject *attributes )

-               {

-                       IECore::CompoundObject::ObjectMap::const_iterator it = attributes->members().find( name );

-                       if( it == attributes->members().end() )

-                       {

-                               return nullptr;

-                       }

-                       return reportedCast<const T>( it->second.get(), "attribute", name );

-               }

-

-                template<typename T>

-                static T attributeValue( const IECore::InternedString &name, const IECore::CompoundObject *attributes, const T &defaultValue )

-                {

-                       typedef IECore::TypedData<T> DataType;

-                       const DataType *data = attribute<DataType>( name, attributes );

-                       return data ? data->readable() : defaultValue;

-                }

-

 

                bool canInstance( const IECore::Object *object, const ArnoldAttributes *attributes ) const

                {

@@ -1480,23 +1459,6 @@

                                /// \todo Remove this workaround once the Arnold bug is fixed.

                                return false;

                        }

-                               const IECore::TypeId objectType = object->typeId();

-                       if( objectType == IECoreScene::ExternalProcedural::staticTypeId() )

-                       {

-                            /*

-                               //const IECoreScene::ExternalProcedural *procedural = static_cast<const IECoreScene::ExternalProcedural *>( object );

-                               if (!attributeValue<bool>( g_allowInstancing, attributes, true))

-                                {

-                                    return false;

-                                }

-                            */

-                            return false;

-                       }

-                       /*

-                        if( IECore::runTimeCast<const GafferScene::ExternalProcedural>(object) && !attributeValue<bool>( g_allowInstancing, attributes, true ){

-                                return false;

-                        }

-                        */

                        return attributes->canInstanceGeometry( object );

                }

[Carlos Sifuentes]

is working yeti - gaffer currently ??

[Carlos Sifuentes]

my test on Gaffer Windows 0.55.2.1-beta

arnold 5.4.0.0

yeti 3.5.3

ARNOLD_PLUGIN_PATH=D:\yeti_folder\bin

result not work possible bug gaffer windows ??

[Miguel González Viñé]

Hi Carlos,

sometime ago I tried to get Yeti working on Gaffer and the fur was exporting when rendering but hidden because a visibility attribute was exported as 0, i don't know why. I've not tested it again since then, and I only tested it on Windows so I don't know if it's working now or not.

You can read about my tests in this email:

https://groups.google.com/d/msg/gaffer-dev/rQraWNbDxaU/vpYEAVgKCQAJ

Good luck!

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/6e9e7eff-d05f-480d-bf98-4cffadea5d3c%40googlegroups.com.

[Carlos Sifuentes]

It would be interesting some official information to load xgen, yeti on gaffer