Snippet composition

[Luke Campagnola]

We've had this discussion a few times, but I don't think we ever got very far. The wiki currently has this to say:

• Each visual comes with shader snippets rather than full-blown shaders. This makes it possible to compose multiple shader snippets into a single shader (essential for transformations).
• Shader snippets are composed through a directed acyclic graph.
• Each shader snippet is defined by:
• input variables
• output variables
• a code body

My response to this has always been that GLSL already implements an ancient, proven system for "composing multiple code snippets" called "functions". Functions have a name, input/output variables, and a code body. They can be recombined in arbitrary ways simply by linking the proper compiled shaders together, or in the case of OpenGL ES, by concatenating the function definitions together. No need for directed acyclic graphs because GLSL already handles combining the code in the correct way. 

But it seems that I am still the only one who thinks using functions is a good idea. Can anyone convince me that there is benefit to recreating this basic language feature in python? Or perhaps the code-snippet system you-all have in mind actually looks very similar to what I would propose?

[Nicolas Rougier]

I’ve no strong argument against this approach and it has the merit to be quite clear and “easy” to implement. I just wonder if this can handle most of the use-cases in the long term and my opinion would be to go for a try it on a moderately complex example and see it it stills work.


Nicolas

> --
> You received this message because you are subscribed to the Google Groups "vispy-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vispy-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Almar Klein]

One use case that I've encountered is that I had code for volume rendering, where a ray is cast through the volume in small steps. But then I wanted to do dynamic volume rendering using a motion field. The technique involved offsetting (i.e. bending) the ray according to a second 3D texture that represents the deformation field. To make stuff like this possible in visvis, I implemented a sort of templating mechanism, with which any existing line can be replaced with another (one or more) line(s). Sort of replace mechanism.

The problem with functions, but also with the currently described approach, I think, is that you need to place a hook for every place where you want to allow the user to introduce something custom. This might seem flexible, but it is rather rigid, because there is no way that we can predict what special stuff the user wants to achieve.

I think it would be best to try and use the simplest (easiest to maintain) system for internal use, but also provide a system that allows practically endless flexibility. It could be as simple as allowing replacing lines of glsl (but doing the replacement a bit smart, e.g. independent of whitespace).

I think using functions or the method in the wiki should be flexible enough for our internal use. I also am unsure about the advantage of the currently proposed method. In what way would it be more flexible?

- Almar

[Nicolas Rougier]

I don’t really believe we can provide “endless” flexibility for shader composition (lot of papers on that topic already) and I would really prefer to avoid an over-generic method that has no practical use in the end (but I may be wrong of course).

Also, maintaing both an internal and external system seems a lot bad idea to me.



Nicolas

[Cyrille Rossant]

Hi,
I'm currently travelling and I don't have time to reply in much
detail, but please feel free to edit the wiki to add your ideas and
suggestions too. The proposition is by no means definitive, I
essentially wrote it to trigger discussions!
Best,
Cyrille

2013/11/8 Nicolas Rougier <Nicolas...@inria.fr>:

[Cyrille Rossant]

>> So in my mind the combination of (1) and (2) together are flexible and
>> the "standard" way of doing things

> To rephrase it a bit, I think we have two options:
>
> * Use functions, try to anticipate many use cases and try to keep function
> bodies small so that they can be replaced without copying too much code.
> (GLSL compilers do inline functions, don't they?)
> * Use a string-manipulating service to provide a very high degree of
> flexibility for the users and use textual hooks to make things more
> explicit/stable for internal use.
>
> I think I like the first better, indeed :)

Agreed. (2) alone may be problematic: maybe in some situations, we
don't want to include a code snippet in the shader.. Think about
panning/zooming: if you want to add a static visual, you should just
be able to remove the lines of code that do that. If you copy and
paste this every time, what will happen when we'll change the
implementation of this feature later?

Could you precise what you have in mind for a mix with 1 and 2? A kind
of function-based templating system, where there are function calls at
many places, and one can just override this functions (that are
minimal by default)? And, if the general template doesn't work, one
can just create a new template?

Cyrille

[Cyrille Rossant]

2013/11/7 Luke Campagnola <luke.ca...@gmail.com>:

I think the system I described can work with either strings or
functions. In the first case, you have a system that concatenates
strings with code snippets. In the second case, each snippet is
encapsulated in automatically-generated functions (like Luke described
a few months ago IIRC). The problem with functions it that I'm not
sure how you can return more than 2 variables? Maybe a solution is to
use GLSL "out" or "inout" keywords? I don't know how well this is
supported.

The big problem with strings is that local variables can have
conflicting names between snippets. So functions should be fine if we
can find how to return several values.

[Luke Campagnola]

On Mon, Nov 11, 2013 at 11:11 AM, Cyrille Rossant <cyrille...@gmail.com> wrote:

>> So in my mind the combination of (1) and (2) together are flexible and
>> the "standard" way of doing things

> To rephrase it a bit, I think we have two options:
>
>   * Use functions, try to anticipate many use cases and try to keep function
> bodies small so that they can be replaced without copying too much code.
> (GLSL compilers do inline functions, don't they?)
>   * Use a string-manipulating service to provide a very high degree of
> flexibility for the users and use textual hooks to make things more
> explicit/stable for internal use.
>
> I think I like the first better, indeed :)

Agreed. (2) alone may be problematic: maybe in some situations, we
don't want to include a code snippet in the shader.. Think about
panning/zooming: if you want to add a static visual, you should just
be able to remove the lines of code that do that. If you copy and
paste this every time, what will happen when we'll change the
implementation of this feature later?

If the system we have set up encourages anyone to "copy and paste every time" then we have done something wrong. Ideally, one would only write a new shader if they have specific graphical or performance requirements that we do not yet support. In this case, pan/zoom would be handled by providing the shader with a different set of transforms to work with. For example, let's say I have a plot inside a pan/zoomable viewport and some text overlaid that is statically positioned. The scenegraph hierarchy might look like this:

    Scene

      -  Viewport

           -  Plot

      -  Text

So the chain of transformations for each visual is:

    Text -> Scene -> Normalized dev. coords. (-1 to 1)

    Plot -> Viewport -> Scene -> Normalized dev. coords. (-1 to 1)

..and the pan/zoom is implemented entirely in the Viewport transform (and thus doers not affect the text). These transformations might be represented as simple 2D translate/scale values, complete 4x4 affine transformations, quaternions, or whatever. The vertex shader code for both visuals will have a similar structure:

// overridable function prototypes

vec4 get_position();

vec4 map_local_to_dev(vec4);

void main(void) 

{

    vec4 local_position = get_position();

    vec4 dev_position = map_local_to_dev(local_position);

    gl_Position = dev_position;

}

So each shader would make a function call first to get the local coordinate position as a vec4, and a second function call to map from the local coordinate system of the visual to normalized device coordinates. Since these are function calls, we can override the function definitions for get_position() and map_local_to_dev() any way we want, without rewriting the main() function. 

Having a get_position() function means that the same core shader could pull its coordinates from a 2D buffer, 3D buffer, or even from a texture just by replacing get_position(). Likewise, whether the mapping from local to normalized device coordinates is best represented by a simple 2D scale+translate, or a matrix multiplication, or a series of transformations (possibly including log/polar/other transformations), we can simply define map_local_to_dev() accordingly. Ultimately, for each visual we just concatenate the core shader string with the strings of each function definition. 

There is an old example I was experimenting with (probably does not work anymore) of how this should look in visuals/line.py. In that example, XYPositionShader is an example of a get_position() implementation (although it is called global_position there) which takes the x,y values from a 2D buffer and the z value from a uniform. There are also example transform shader functions in shaders/transforms.py, but these are a bit more complex because, while defining and overriding a single function prototype is easy, it takes a little more work to implement user-configurable function chains, where you don't know beforehand the number of functions in the chain. Some string manipulation is needed here to prevent collisions between the names of repeated functions and attribute/uniforms.

Luke

[Luke Campagnola]

On Mon, Nov 11, 2013 at 11:11 AM, Cyrille Rossant <cyrille...@gmail.com> wrote:

I think the system I described can work with either strings or
functions. In the first case, you have a system that concatenates
strings with code snippets. In the second case, each snippet is
encapsulated in automatically-generated functions (like Luke described
a few months ago IIRC). The problem with functions it that I'm not
sure how you can return more than 2 variables? Maybe a solution is to
use GLSL "out" or "inout" keywords? I don't know how well this is
supported. 

The big problem with strings is that local variables can have
conflicting names between snippets. So functions should be fine if we
can find how to return several values.

So far I have not run into a situation where I wanted to return more than one variable, but it looks like you can pass variables by reference using "inout":

http://stackoverflow.com/questions/13633586/can-you-pass-a-matrix-by-reference-in-a-glsl-shader

..although as usual I wonder whether this is supported in ES.

[Almar Klein]

The problem with functions it that I'm not
sure how you can return more than 2 variables? Maybe a solution is to
use GLSL "out" or "inout" keywords? I don't know how well this is
supported.

in, out, and inout are supported in ES 2.0 . Anyone knows what version of desktop glsl is needed for this?