Sorry for the delay. I've reproduced this issue, and it seems that the
segfault is occurring here:
https://github.com/Kitware/VTK/blob/v7.1.1/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx#L386-L387
The cause of the crash seems to be the following sequence of events:
(1) VTK obtains a GLXFBConfig with glXChooseFBConfig(..., {
GLX_RED_SIZE=8, GLX_GREEN_SIZE=8, GLX_BLUE_SIZE=8, GLX_DEPTH_SIZE=16,
GLX_DOUBLEBUFFER=1 }). In the case of my particular machine, the first
FB config returned is:
0x259 24 tc 0 24 0 r y . 8 8 8 0 . 4 24 0 16 16 16 16 0 0
None PRW
which is the most commonly-used FB config for TrueColor rendering.
(2) VTK uses glXGetVisualFromFBConfig() to obtain the corresponding X
visual. VirtualGL returns 0x21, which is the only TrueColor 24-bit
visual provided by TurboVNC.
(3) VTK creates a window using this visual.
(4) For reasons I can't understand, VTK doesn't store the FB config it
obtained earlier. It instead tries to back-engineer it using this code:
https://github.com/Kitware/VTK/blob/feb05ce07089c5c1027818d5c878de19d229a69d/Rendering/OpenGL2/vtkXOpenGLRenderWindow.cxx#L506-L523
Since there is not a 1:1 correspondence between Visuals and GLXFBConfigs
in VirtualGL, this code iterates through all of the GLXFBConfigs and
ultimately chooses the last one, which is an esoteric floating point
pixel format:
0x4c3 0 ?? 0 64 0 y . 32 32 0 0 f 4 24 8 16 16 16 16 0 0
None P..
(5) VTK calls glXCreateContextAttribsARB() with this FB config and
attributes of GLX_CONTEXT_MAJOR_VERSION_ARB=4,
GLX_CONTEXT_MINOR_VERSION_ARB=5, i.e. requesting an OpenGL 4.5 context.
(6) VTK calls glXMakeCurrent() with the new context. Within the body of
glXMakeCurrent(), VirtualGL creates a Pbuffer and maps it to the OpenGL
window, then calls glXMakeContextCurrent() to swap in the Pbuffer. This
call to glXMakeContextCurrent() fails for an unknown reason (but likely
related to the bizarre FB config that VTK has chosen.)
(7) VGL returns False from glXMakeCurrent(), but VTK ignores the return
value.
(8) There is no valid context when glBlendFuncSeparate() is called,
which presumably leads to the segfault.
I was able to work around this by commenting out:
https://github.com/VirtualGL/virtualgl/blob/master/server/faker-glx.cpp#L185-L190
thus making VirtualGL's implementation of glXGetVisualFromFBConfig()
more strict (i.e. it will no longer return a visual for the esoteric
floating point FB configs.) However, unfortunately that kicks the can
down the road and exposes new issues, which I am unable to diagnose:
ERROR: In
/builddir/build/BUILD/VTK-7.1.1/Rendering/OpenGL2/vtkShaderProgram.cxx,
line 424
vtkShaderProgram (0xff953465f0): Shader object was not initialized,
cannot attach it.
ERROR: In
/builddir/build/BUILD/VTK-7.1.1/Rendering/OpenGL2/vtkOpenGLVertexArrayObject.cxx,
line 271
vtkOpenGLVertexArrayObject (0xff9493e730): attempt to add attribute
without a program for attribute vertexWC
ERROR: In
/builddir/build/BUILD/VTK-7.1.1/Rendering/OpenGL2/vtkOpenGLPolyDataMapper2D.cxx,
line 359
vtkOpenGLPolyDataMapper2D (0xff951551c0): Error setting 'vertexWC' in
shader program.
NOTE: These errors can also be reproduced using the VTK examples and
with Mayavi2 on the local display (without VGL), so the errors are VTK
bugs, not ours.
I will investigate how best to work around the initial error. I need to
figure out why the more liberal visual matching code was added to
glXGetVisualFromFBConfig() in the first place and also try to find a
platform that reproduces the visual matching error but doesn't reproduce
the shader errors.
DRC