using camera classes with gloo

[nbbb]

i'm still following the molecular_viewer example. i've got it working and understand the model, view and perspective matrices now (although they are transposed with respect to what is on most online tutorials)

anyway, is there a way to use the prefab camera classes (PanZoomCamera etc) in conjunction with a shader program written with gloo? I see that one can extract the .transform from a camera instance. what do i need to set as the view and projection matrix in the shader then? how do i plug it together?

[nbbb]

ok, i got a little further.

i am now creating a scene.SceneCanvas object, say, mvc.  i then set mvc.program to the program conaining vertex and fragment shaders, as in molecular_viewer. i further create a cam = TurntableCamera(parent=mvc.scene); finally i manually assign the shader's view matrix as cam.transform.matrix in the method mvc.on_mouse_wheel.

this works in principle but seems clunky since the camera does not seem to receive any events.

reading the docs, i am thinking maybe i should do this instead:

mvc = scene.SceneCanvas(..)
view = mvc.central_widget.add_view(..)
cam = TurntableCamera(..., parent=view)
view.camera = cam

will cam then receive mouse events? more importantly, how do i need to modify the shader programs so that they take care of the proper view matrix coming from the camera cam?

[nbbb]

so the tl;dr is basically:

i want to use a TurntableCamera, which should receive the mouse events coming from some Canvas. drawing in the canvas is handled by gloo directly, and i'm fine with extracting the current transform from the camera and setting for the shader, in some appropriate callback function.

is this feasible?

[Eric Larson]

I think you should build a new Visual. Take a look at the line_prototype.py example, have you seen it? It describes what needs to be added to make a functional Visual. Then you'd use the create_visual_node  (I think that's the name) to turn it into a Node that can be used in a scenegraph. And then you'd set the parent of the Visual to the scene of the ViewBox in use, which you'll see in many Scene examples.

Eric

--
You received this message because you are subscribed to the Google Groups "vispy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vispy+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[nbbb]

> I think you should build a new Visual

wow, this really is a rabbithole. so the Visual basically wraps the gl shader code using template variables and makes it compatible with the scene graph machinery? so what i would be creating is a sort of a point cloud Visual with built-in resizing of the point sprites to the viewer distance. sounds like the right thing, but also somewhat daunting. postponed for now. anyway, thanks for your reply!

 

[Eric Larson]

All you should really need to change in existing vertex shader GLSL code is the addition of a `$transform()` template variable, plus boilerplate _prepare_transforms and _prepare_draw methods to get something workable. Think of it being more like a golf divot than a rabbit hole :)

Eric

On Thu, Aug 13, 2015 at 10:17 AM, nbbb <bck...@gmail.com> wrote:

> I think you should build a new Visual

wow, this really is a rabbithole. so the Visual basically wraps the gl shader code using template variables and makes it compatible with the scene graph machinery? so what i would be creating is a sort of a point cloud Visual with built-in resizing of the point sprites to the viewer distance. sounds like the right thing, but also somewhat daunting. postponed for now. anyway, thanks for your reply!

 

--

[Luke Campagnola]

On Thu, Aug 13, 2015 at 10:17 AM, nbbb <bck...@gmail.com> wrote:

> I think you should build a new Visual

wow, this really is a rabbithole. so the Visual basically wraps the gl shader code using template variables and makes it compatible with the scene graph machinery? so what i would be creating is a sort of a point cloud Visual with built-in resizing of the point sprites to the viewer distance. sounds like the right thing, but also somewhat daunting. postponed for now. anyway, thanks for your reply!

 

It is a rabbithole. The cameras are not meant to be used outside the scenegraph, and there's a steep learning curve for developing new visuals.  My approach would be to modify Markers to support the features you need.

[nbbb]

It is a rabbithole. The cameras are not meant to be used outside the scenegraph, and there's a steep learning curve for developing new visuals.  My approach would be to modify Markers to support the features you need.

ok, that may be more palatable. i did try briefly to adapt the line_prototype to my problem. i quickly hit an obstacle: in order to do the marker-size determination for correct resizing of physical objects as in molecular_viewer, it's necessary to know the distance to the camera. this can be gotten from the product of the model * view matrices. however, i did not see how to get that when using a camera: afaics these only expose the final, finished transform, view * projection; not view alone.

so even for the Markers extension idea i'm not totally sure it can be done easily. basically the difficulty is that to set the gl_PointSize of a marker, one has to know the camera position, which is not a property of the visual itself but has to be taken from upstream in the scene graph. can it be done?

[Eric Larson]

ok, that may be more palatable. i did try briefly to adapt the line_prototype to my problem. i quickly hit an obstacle: in order to do the marker-size determination for correct resizing of physical objects as in molecular_viewer, it's necessary to know the distance to the camera

After the $transform() has done its job, the position is in normalized device coordinates. So the effective distance to the camera is related to the `.z` value, so you might be able to make use of this in some creative way. This might not be a general solution, though, and there might be some more principled way to do it...?

You could also go the route of using a set of spheres, combined with a perspective camera. Then you get the shrinkage/growth based on distance automatically.

Eric

[Luke Campagnola]

The position of the camera can be used with some workarounds, but generally this is the wrong approach since for orthographic cameras the distance is meaningless. A more direct question is: how many pixels should each point occupy on-screen? Here's how I'd do it:

1. Map the center position of the point (call this c) to document coordinates (call this c').

2. Add to that `vec2(1, 0)` (call this px') and map the resulting point back to the visual's coordinates (call this px). Now you know the length of a pixel (at least in one direction) in the coordinate system of your data. 

3. Divide the diameter of the point by the length of the mapped pixel vector; this gives you the point size you need in pixels: GL_PointSize = diameter / length(px - c)

[nbbb]

You could also go the route of using a set of spheres, combined with a perspective camera. Then you get the shrinkage/growth based on distance automatically.

yes, that's how i started out; but it was not clear to me how to make a whole collection of spheres into a single visual which is updated by an array of center positions and radii. (see my previous question here) the suggestion i got was that point markers are a more efficient / clever way of doing it, when combined with resize logic...

[nbbb]

 

1. Map the center position of the point (call this c) to document coordinates (call this c').

2. Add to that `vec2(1, 0)` (call this px') and map the resulting point back to the visual's coordinates (call this px). Now you know the length of a pixel (at least in one direction) in the coordinate system of your data. 

3. Divide the diameter of the point by the length of the mapped pixel vector; this gives you the point size you need in pixels: GL_PointSize = diameter / length(px - c)

that sounds clever. i'd have to do this for every point separately i guess, including the last translation to the point location in the model coordinate system. i guess i'll try it.

[Eric Larson]

The idea with that is that it should be done in GLSL / the vertex shader, so doing it for every point is automatically parallelized.

Eric

--

[nbbb]

The idea with that is that it should be done in GLSL / the vertex shader, so doing it for every point is automatically parallelized.

sure, i understand. so i tried out the rescaling trick using the inverse map, and using a new Visual subclass. This is starting to work, however I now see a kind of a mosquito-net effect on the output: the scene is shown, I can turn the camera etc but the background is sometime drawn in front of the scene, giving a sort of a rasterized random grid in front of the scene. something must be wrong with the drawing order/screen update?

I basically have a new Visual, from which I make a Node by create_visual_node, which is then shown in a SceneGraph.
I could post a gist if anyone cares to have a look...

[Eric Larson]

I wonder if it is related to drawing order / depth testing, like this might be:

https://github.com/vispy/vispy/issues/1060

Eric

--

[nbbb]

I also see the behavior described in #1060 in the code example from #1060, but in my case the effect looks different - garbled graphics instead of the wrong occlusion.

[nbbb]

Update: the initial display of the scene looks ok; only when the camera is moved, things go wrong. resizing the window alone is also ok

also, the rescaling of the marker size does not yet work: the point radius just stays the same on camera movement. anyway here some relevant parts of the code:

vertex = """
#version 120
//
uniform vec3 u_light_position;
uniform vec3 u_light_spec_position;
uniform float u_radius;
//
attribute vec3  a_position;
attribute vec3  a_color;
attribute float a_radius;
//
varying vec3  v_color;
varying float v_radius;
varying vec3  v_light_direction;
//
// template variables:
// $transform
// $inv_transform
//
void main (void) {
    v_radius = a_radius * u_radius;
    v_color = a_color;
    //
    v_light_direction = normalize(u_light_position);
    //
    vec4 pos = vec4(a_position, 1.0);
    gl_Position = $transform(pos);
    // now the inverse trick
    vec4 up_shift = gl_Position + vec4(0,1,0,0);
    vec4 pre_vec = $inv_transform(up_shift) - pos;
    float scale_factor = sqrt(dot(pre_vec, pre_vec));
    gl_PointSize = 10.0/ scale_factor;
}
"""

# fragment shader for drawing something on screen; given the 2D position.

fragment = """
#version 120
//
uniform vec3 u_light_position;
uniform vec3 u_light_spec_position;
//
varying vec3  v_color;
varying float v_radius;
varying vec3  v_light_direction;
void main()
{
    // r^2 = (x - x0)^2 + (y - y0)^2 + (z - z0)^2
    vec2 texcoord = gl_PointCoord* 2.0 - vec2(1.0);
    float x = texcoord.x;
    float y = texcoord.y;
    float d = 1.0 - x*x - y*y;
    if (d <= 0.0)
        discard;
    // no more specular lighting
    float z = sqrt(d);
//    gl_FragDepth = 0.5*(pos.z / pos.w)+0.5;
    vec3 normal = vec3(x,y,z);
    float diffuse = clamp(dot(normal, v_light_direction), 0.0, 1.0);
    //
    vec3 v_light = vec3(1., 1., 1.);
    gl_FragColor.rgb = (.15*v_color + .55*diffuse * v_color);
}
"""


class SpheresVisual(visuals.Visual):

    """A collection of spheres rendered as sprites, with correct perspective
    size"""

    def __init__(self, config_array):
        """Set position and size

        :pos: TODO
        :color: TODO

        """
        visuals.Visual.__init__(self, vcode=vertex, fcode=fragment)

        self.config = config_array

        self.nmax, self.dims, self.data = self.load_molecules()
        self.vbuffer = gloo.VertexBuffer(self.data)

        self.c_inds, self.colors = self.get_colors()

        self.shared_program.bind(self.vbuffer)
        self.shared_program['u_light_position'] = 0., 0., 2.
        self.shared_program['u_radius'] = 1.0
        self._need_upload  = False

        self._draw_mode = 'points'
        self.set_gl_state('opaque', depth_test=False)
       
        self.update_data()

    def get_colors(self):
        ma = self.config.ma
        # invalid value Runtime Warning here;
        # comes from different nan representations; no problem.
        uids = np.unique(ma['id'][~np.isnan(ma['id'])])
        allcols = np.array(color.get_color_names())
        #make this all white
        cnames = allcols[[int(id_) % len(allcols) for id_ in uids]]
        # cnames = ['white' for c in cnames]
        ca = color.ColorArray(list(cnames))
        # hsvs = np.c_[np.modf(uids * 1000 * np.pi)[0],
                # [.8 for _ in uids], [0.5 for _ in uids]]
        # ca = color.ColorArray(color_space="hsv", color=hsvs)
        inds = dict([(id_, i) for (id_, i) in zip(uids, range(len(uids)))])
        return inds, ca

    def n_mols(self):
        return np.searchsorted(self.config.ma['id'], np.nan)

    def load_molecules(self):
        ma = self.config.ma
        nmax = ma.shape[0]
        dims = ma['center'][0].shape[0]

        #pre-allocate
        data = np.zeros(nmax,
                [('a_position', np.float32, 4),
                    ('a_color', np.float32, 4),
                    ('a_radius', np.float32, 1)])
        return nmax, dims, data

    def update_data(self):
        ma = self.config.ma
        n = self.n = self.n_mols()  # needed since it can change

        if self.dims == 2:
            self.data['a_position'][:n] = np.c_[ma['center'][:n],
                    np.zeros(n, dtype=np.float32),
                    np.ones(n, dtype=np.float32)]
        if self.dims == 3:
            self.data['a_position'][:n] = np.c_[ma['center'][:n],
                    np.ones(n, dtype=np.float32)]

        # this is potentially slowest. a python loop!
        for i, id_ in enumerate(ma['id'][:n]):
            self.data['a_color'][i] = self.colors[self.c_inds[id_]].rgba
        self.data['a_radius'][:n] = 5 * ma['radius'][:n]
       
        self._need_upload = True
        self.vbuffer.set_data(self.data[:n])

    def _prepare_transforms(self, view=None):
        view.view_program.vert['transform'] = view.transforms.get_transform()
        view.view_program.vert[
                'inv_transform'] = view.transforms.get_transform().inverse

    def _prepare_draw(self, view=None):
        if self._need_upload:
            self.vbuffer.set_data(self.data[:self.n])
            self._need_upload = False


SpheresNode = create_visual_node(SpheresVisual)


class Spheres(SpheresNode):

    def __init__(self, geom, *args, **kwargs):

        SpheresNode.__init__(self, *args, **kwargs)
        self.unfreeze()
        self.geom = geom

        self.l, self.u, self.ue = self.geom.ma['center'][:]
        self.center = v3(0.5 * (self.l + self.u))
        self.size = self.u - self.l
        self.segs = (self.size / self.ue).astype(int)

        self.transform = STTransform(translate=-self.center)
        self.freeze()