OSL in 3Delight.

[Aghiles Kheffache]

Hello Everyone,

We have got a solid OSL renderer implemented in 3Delight, finally. It contains a good implementation of both the GGX and the GTR BRDFs with some additional goodies. I have put a public page that describes some of the particularities of our implementation. You can read it under "General Guidelines" here:

https://3delight.atlassian.net/wiki/display/3DSP/OSL+in+3Delight

Our free Maya plug-in now allows users to render HyperShade networks by converting them to OSL networks. You can follow instructions on how to do this here:

1) Enable pre-release features: https://3delight.atlassian.net/wiki/display/3DFM/Pre-Release+Features

2) Create an OSL render pass: https://3delight.atlassian.net/wiki/pages/viewpage.action?pageId=38076614

For those interested on guidelines on how to write RIB files that contain OSL networks, let me know.

Thank you,

Aghiles

www.3delight.com

P.S. Note that we managed to properly compile the libraries on all platform. If Windows patches are needed we will be happy to provide them.

[Larry Gritz]

Very cool, Aghiles!

So many questions and comments...

Are you just using stock OSL, or did you find the need to change anything in the implementation?

I like what you have done with the reflection/refraction understanding backfacing on their own.

"passthrough" -- nice. Is there anything special about the implementation, or is it just setting a debug() closure and then passing the result on? (In other words, did you just implement passthrough in OSL itself in the obvious way?)

lockgeom -- I applaud your getting rid of this wart. So you are just saying that by convention, in your renderer you expect any connected geometric variable to go through a special node that uses a getattribute()?

Would it be helpful to you to get rid of that node and instead have a direct way through the ShadingSystem API to say that a particular parameter should be bound to a geometric variable if available? Would that enable us to get rid of lockgeom entirely? That might also allow a better way to OSLQuery the default value if the geometric variable is not present.

The "performance comparisons" page is private so I couldn't read that, but I'm wondering if you can generally comment on how you found performance and offer any guidance on where we should try to improve? I'm afraid SPI's renderer hasn't had a way to shade non-OSL for years now, so I no longer have a direct comparison to other systems. (Though at the time we got rid of the other shading methods, OSL was outperforming the alternative in our renderer.)

Windows -- I would be very, very interested in getting an up-to-date set of instructions for how to install dependencies and build OSL reliably on a Windows system, which I would be very happy to put in the installation nodes of the distribution. Also, I've had wonderful experience lately using Travis and would like to expand it to Appveyor for Windows CI. If you (or for that matter, representatives of any of the other places that need to build OSL for Windows) would like to collaborate on that, I would be very interested. I think the incentive from your point of view is that if we can make Windows-based CI a regular part of our development, it will save you heartache in the long run by preventing those of us on Linux/OSX from accidentally breaking the Windows builds.

-- lg

--
Larry Gritz
l...@larrygritz.com

[François Beaune]

Hi Aghiles, Larry,

I don't know if that's useful, but for appleseed we regularly build OSL and all its dependencies on Windows. It's actually fairly straightforward. We have this batch file that will build everything automatically in one go:

https://github.com/appleseedhq/appleseed-deps/blob/master/BuildAll.bat

Here is the OSL part:

https://github.com/appleseedhq/appleseed-deps/blob/master/BuildAll.bat#L243-L267

Best,

Franz

PS: congrats Aghiles for the amazing work on 3Delight+OSL!

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.
To post to this group, send email to osl...@googlegroups.com.
Visit this group at https://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

[Christopher Kulla]

This is really cool! Congrats on getting it out the door!

The newest closures in our renderer also follow the automatic eta inversion policy. In hindsight this is probably the only sensible solution, particularly when you get into nested dielectric tracking. We even went one step further and just merged reflection and refraction lobes into one closure.

I totally agree on lockgeom. There's some renderer specific reasons it still helps us in a few cases, but moving to get rid of the automatic binding mechanism would be a nice simplification to the library.

[Larry Gritz]

Chris, do you think it's practical (for us) to specify the connections between geometric variables and shader inputs at shader group declaration time (i.e., SS::Shader(), Parameter(), ConnectShaders(), etc.) and banish automatic name binding and lockgeom along with it?

If we have "connect earlierlayer.output mylayer.input" now, can we simply allow "connect geom.variable mylayer.input" (or some other choice of nomenclature), and call it a day?  (Presumably, if the geom doesn't have such a layer, the connection would not happen and the usual default value as specified in the shader would win the fight.)

What is the burden on lookdev (or the material/lighting tool) for requiring all geometric primitive variables to be so declared along with the shader group?

Is there merit to the shader (source) still being able to say "this parameter is bound by default to this geometric variable"? Or no?

-- lg

--
Larry Gritz
l...@larrygritz.com

[Christopher Kulla]

I don't think the shader declaration time logic would work easily with our lighting tool since it tries to share shaders between geometries. Information about the geometry attributes is also not handy during interactive updates.

My vote would be to get rid of the concept of geometric bindings altogether. Params are connected, instance value or default. We would turn what is currently:

float myparam = default_expr [[int lockgeom 0]],

into:

float myparam = getuserdata("myparam", default_expr),

getuserdata being a new call that would preserve the distinction between get_attribute callback and the get_userdata callback in RendererServices. Instead of an automatic binding, we would just be using the regular "default value with expression" mechanism.

We could have a separate discussion on the merits of getattribute vs. getuserdata -- but doing it this way would enable a backwards compatible transition I think.

[Aghiles Kheffache]

On Tuesday, January 5, 2016 at 6:58:07 PM UTC-5, Larry Gritz wrote:

Very cool, Aghiles!

Thanks Larry. After two decades of RSL, working with OSL is like landing in heaven.

 

Are you just using stock OSL, or did you find the need to change anything in the implementation?

Stock OSL.

We basically fetch the repo, apply some compilations patches and just build it as is. Any shader compiled with the standard OSL will work in 3Delight.

We are also using the standard "stdosl.h", but we might need to add some entries in there soon. I will let you know.  

 

I like what you have done with the reflection/refraction understanding backfacing on their own.

Thanks. The idea was to try to let go some of annoying stuff that we hated in RSL (like eta inversion). 

 

"passthrough" -- nice. Is there anything special about the implementation, or is it just setting a debug() closure and then passing the result on? (In other words, did you just implement passthrough in OSL itself in the obvious way?).

passthrough() is really to output shading components (it only works with closures). We will only allow such components to be output as AOVs (e.g. specular, diffuse, emission... ). Other "debug" AOVs such as fating ratio, normals and object ids will be provided as a renderer service and not a shading system service. We will still implement debug() if someone needs to output something specific for debugging purposes.

lockgeom -- I applaud your getting rid of this wart. So you are just saying that by convention, in your renderer you expect any connected geometric variable to go through a special node that uses a getattribute()?

Yes. That actually solves the problem quite nicely. One can see it as a connection-based lockgeom.

Would it be helpful to you to get rid of that node and instead have a direct way through the ShadingSystem API to say that a particular parameter should be bound to a geometric variable if available?

I guess that you have performance concern regarding the extra node ? One simple way is  to define a "standard getattribute() node" that renderers recognize. This means that the only work needed, as far as OSL is concerned, is to document such a node. 

 

Would that enable us to get rid of lockgeom entirely?

Both the node-based approach and the one you are suggesting would work I think! 

The "performance comparisons" page is private so I couldn't read that, but I'm wondering if you can generally comment on how you found performance and offer any guidance on where we should try to improve?

(The performance page compares many renderers together. It is not thorough enough and, as you now, this is a touchy subject so basically its private for now.) 

Regarding OSL: it's very fast. As you now, we have ridiculously vast experience with RSL, down to the JIT system, and I can say with confidence that shaders execute at least twice as fast on production scenes. Our simd RSL system compares well when shading large batches of shading samples but for single rays there is absolutely no comparison. 

Now, regarding C++ shaders vs OSL: from the testing I did with two other renderers (Arnold and PrMan, not to name them), it makes really very little sense to stick with C++. I foresee a total dominance of the OSL shading system in less than 2 years.

Windows -- I would be very, very interested in getting an up-to-date set of instructions for how to install dependencies and build OSL reliably on a Windows system, which I would be very happy to put in the installation nodes of the distribution. Also, I've had wonderful experience lately using Travis and would like to expand it to Appveyor for Windows CI. If you (or for that matter, representatives of any of the other places that need to build OSL for Windows) would like to collaborate on that, I would be very interested. I think the incentive from your point of view is that if we can make Windows-based CI a regular part of our development, it will save you heartache in the long run by preventing those of us on Linux/OSX from accidentally breaking the Windows builds.

We can help with that. I will gather the patches. We build the system quite often so we can SIGALRM when it breaks. 

 

--

Aghiles

[Aghiles Kheffache]

On Wednesday, January 6, 2016 at 3:32:25 AM UTC-5, François Beaune wrote:

PS: congrats Aghiles for the amazing work on 3Delight+OSL!

Thanks Francois!

--

Aghiles

[Aghiles Kheffache]

I must admit that I don't understand the advantage of this solution.  The node approach we followed has the following advantages

- You can still keep the folding optimisation when no primvars are present.

- Shaders programmers do not have to think about primitives variables which is sensible. 

- We use a mechanism that is readily available and actually would already work in all renderers with minimum work.

Since adding a primitive variable is a feature of a "parent" software, it makes sense to add the primvar reading functionality there too. In our case, the parent software has to add a connection to that variable that is intended as a receiver for the primvar. Again, there is absolutely nothing to do on the OSL side.

What I am trying to say is: the problem lockgeom is trying to solve doesn't really exist. It has always been solvable with very little logic at the shader tree creation phase.

--

Aghiles

[Aghiles Kheffache]

On Wednesday, January 6, 2016 at 4:09:38 PM UTC-5, Larry Gritz wrote:

Is there merit to the shader (source) still being able to say "this parameter is bound by default to this geometric variable"? Or no?

Frankly I think that would be the least logical place to make such a decision. I can't really come up with an example that would make sense. 

[Christopher Kulla]

What I am trying to say is: the problem lockgeom is trying to solve doesn't really exist. It has always been solvable with very little logic at the shader tree creation phase.

 

I agree. And we use the "node to read attribute" solution too for 99% of our cases too. I was just suggesting a possible way of keeping the ability to automatically bind variables if the shader writer really wants it, while still removing the concept from the library.

[Aghiles Kheffache]

On Wednesday, January 6, 2016 at 5:56:10 PM UTC-5, Christopher wrote:

What I am trying to say is: the problem lockgeom is trying to solve doesn't really exist. It has always been solvable with very little logic at the shader tree creation phase.

 

I agree. And we use the "node to read attribute" solution too for 99% of our cases too. I was just suggesting a possible way of keeping the ability to automatically bind variables if the shader writer really wants it, while still removing the concept from the library.

Understood.

 

[katsu...@gmail.com]

Congratulations, Aghiles. :)
I could build oslc.exe by myself, so I can play and dive into OSL with 3delight.

p.s.
Hi, Larry. Long time no see.
I would be happy if you remember me.
I'm sorry for my poor English. :)

--katsu

Katsuaki Hiramitsu

[Aghiles Kheffache]

Good news Katsu!

You asked us for a way to be able to add custom OSL nodes into Maya. This is now done. Todays build will contain the changes.

See you soon!

--

Aghiles

[Larry Gritz]

OK, so we all agree that when you have a shader parameter that you want to bind to interpolated geometric primitive data, rather than let the name bind automatically, you can simply connect it to a trivial upstream node that looks like the following, and never have to think about "lockgeom":

    shader bindfloat (string name = "", output float value = 0)

    {

        getattribute (name, value);

    }

There's a minor performance issue associated with overhead of having to call out to the upstream node, but if this is a common idiom I could certainly modify the optimizer to notice a node that does nothing but directly assign a getattribute to an output param, and move the call into its downstream connection, so there will be no overhead. (This is similar to how it's already the case that if the upstream node is simplified to merely assign a constant to its output, the downstream connection will have its input turned into that constant without needing to call the upstream node at all.)

The only other inconvenience is that you need a different variety of this node for each data type that you might want to pass. (One for floats, one for colors, one for strings, new ones you'll have to write for exotic cases...)

So Chris offers an embellishment:

float myparam = getuserdata("myparam", default_expr),

which consists of (a) a way to unambiguously retrieve just userdata (i.e., "interpolated geometric primitive data") without binding to renderer state attributes (is that good or bad? debate), and (b) operates as 'retrieved_value = get (name, defaultval)', which seems like a more convenient syntax for this use.

If we can get everybody to switch to this methodology, perhaps we can eventually the whole notion of automatic binding if nobody still requires it, and that will simplify some things.

The other advantage to Chris's approach is that this is a simple construct that can be recognized by OSLQuery, so it would be possible to get a clear notion of "this param binds to FOO, or else gets constant BAR" in a way that apps can understand. The two-node approach obscures this.

-- lg

[Larry Gritz]

Hello, Katsu! Great to see you here!

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.
To post to this group, send email to osl...@googlegroups.com.
Visit this group at https://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

--
Larry Gritz
l...@larrygritz.com

[Aghiles Kheffache]

Hello Larry,

On Tuesday, January 12, 2016 at 2:01:55 AM UTC-5, Larry Gritz wrote:

...

So Chris offers an embellishment:

float myparam = getuserdata("myparam", default_expr),

which consists of (a) a way to unambiguously retrieve just userdata (i.e., "interpolated geometric primitive data") without binding to renderer state attributes (is that good or bad? debate), and (b) operates as 'retrieved_value = get (name, defaultval)', which seems like a more convenient syntax for this use.

My fear is that in this situation, all shader writers will simply always add this expression to each parameter. Because at the end of the day, you want any parameter to be connectable to primitive variables. You can't guess a priori  which parameters needs that expression or not. This is will also uglify shaders a bit.

The main point here is that shader writers should be totally oblivious to primitive variables, an entity from another world.

--

Aghiles

[Larry Gritz]

I see -- in this form, users may be inclined to litter so many parameters with a getuserdata binding ("just in case") that it will totally defeat a lot of the best optimizations, akin to marking everything as lockgeom=0. An operation that disables so many optimization opportunities should take some extra effort so that it's only done intentionally.

Very well, then it seems the prudent thing to do for now is

1. Leave the semantics as they are.

2. Add the optimization I mentioned so that we know that there is NO additional performance penalty when you put the getattribute call into a separate node.

3. Reword any documentation or examples that talk about primitive variables so it's expressed with a node, to encourage people to adopt that idiom, and de-emphasize the role of lockgeom and auto-binding (it will still be explained somewhere, but as a disfavoured way of getting to primitive variables).

4. Periodically check in to see if all key parties (there are a growing number of them) ever get to a point where we could/should remove auto-binding and lockgeom entirely, without stranding anyone.

Sound ok to everybody?

--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+u...@googlegroups.com.
To post to this group, send email to osl...@googlegroups.com.
Visit this group at https://groups.google.com/group/osl-dev.
For more options, visit https://groups.google.com/d/optout.

--
Larry Gritz
l...@larrygritz.com

[Aghiles Kheffache]

I vote for this.

[Christopher Kulla]

Very sensible approach, sounds good to me too. As I mentioned earlier -- I suspect we (SPI) are the ones most dependent on the lockgeom=0 case right now.

As soon as we can transition away from our few uses of it, we could most likely remove it.

What do people think about the distinction between get_attribute and get_userdata? In our renderer the former is a bit more expensive than the latter - but that is really just an implementation detail. I am not sure its enough to merit maintaining the distinction. On the other hand its not a huge burden to keep get_userdata if its just exposed as another builtin.

-Chris

[Wayne Wooten]

 Pixar is actually very dependent on lockgeom=0 as well. We’ve always had pipelines that behave this way.

 Not to say we can’t change, but it will be a non-trivial effort.

   —Wayne

[Larry Gritz]

Wayne: No prob. There is no reason to break it now (or maybe ever).

Chris: The existing getattribute call already comes in a 3-argument variety getattribute(object,name,destination) where the object designation can be used to restrict the search. We generally use it to perform the lookup on a specific node (for example, getting information about a named camera). But we could decide that object "userdata" (or whatever name we like) is the way to take the shortcut and ONLY search userdata.

I also renew my offer for another alternative (not necessarily mutually exclusive): a group-assembly-time call or notation from the C++ API that lets you mark a particular parameter as binding to primitive data. This keeps it out of the shaders, and avoids the need to lay out additional nodes and connections -- making primitive binding no more onerous than setting an instance value. 

[Wayne Wooten]

On Jan 12, 2016, at 2:23 PM, Larry Gritz <l...@larrygritz.com> wrote:

Wayne: No prob. There is no reason to break it now (or maybe ever).

Chris: The existing getattribute call already comes in a 3-argument variety getattribute(object,name,destination) where the object designation can be used to restrict the search. We generally use it to perform the lookup on a specific node (for example, getting information about a named camera). But we could decide that object "userdata" (or whatever name we like) is the way to take the shortcut and ONLY search user data.

 This is exactly what we have done. In our build that is either “primvar” or “builtin” and let’s users grab builtins from our renderer (like normalized tangent) that aren’t provided in the default data structures of OSL.

I also renew my offer for another alternative (not necessarily mutually exclusive): a group-assembly-time call or notation from the C++ API that lets you mark a particular parameter as binding to primitive data. This keeps it out of the shaders, and avoids the need to lay out additional nodes and connections -- making primitive binding no more onerous than setting an instance value. 

  I like this. :)

[Daniel Heckenberg]

At animal logic we:
- always lockgeom.
- use getattribute to fetch geo data
- use an explicit node in our shader groups to do so
- allow this node to be added at shader group instantiation time

The changes described are welcome and in line with our usage.

Cheers,
Daniel

[Adrien Herubel]

A Ubisoft we use the getattribute() in explicit node approach.

The only case where we were forced to use lockgeom=0 is in this case :

float UV_0 = u,
float UV_1 = v,
Vec2 UV = {UV_0,UV_1} [[ int lockgeom = 0 ]],

But it might be a bug in the OSL version we use.

--Adrien

[Dan Kripac]

Similarly at dneg we have an enhanced OSL getattribute to access primvars and Prman RIS globals out-of-reach from the OSL::ShaderGlobals as well as prman attributes and options.

We prefix ours with "primvar:" or "globals:". 

I only favoured the string "globals:" as it's is eight characters long (but then so does "builtin:" so I should support it..) mostly so I can do a 64bit int compare on the first eight characters of an attribute name to differentiate it's meaning - slightly arbitrary reason though I know :)

I'm also in favour of support of the C++ side to mark parameters as "primvar compatible" regardless of the existence of lockgeom on a parameter.