Ideas for solving the long-standing Visual-to-FBConfig mapping problem in VGL

[DRC]

Dealing with the mapping of X Visuals and GLXFBConfigs in VGL has always
been a pain, since there often is not a 1:1 correspondence between
GLXFBConfigs on the 3D X server and X visuals on the 2D X server.
Applications that do "visual hunting" (i.e. cycling through all of the
visuals or FB configs looking for one that matches a particular set of
attributes) continue to foil VirtualGL-- including, apparently, recent
versions of Google Chrome
(https://github.com/VirtualGL/virtualgl/issues/73).

Since XVisualInfo is an exposed structure, there is no way for VirtualGL
to replace pointers to that structure with opaque pointers to a
VGL-specific structure. However, it occurred to me that it might be
possible to extend the existing structure in an opaque way. The idea is
that, when an application calls XGetVisualInfo(), VirtualGL would
interpose that function and return a pointer to an extended version of
the XVisualInfo structure. From Xlib's point of view, it would look
like a regular XVisualInfo structure, but it would contain an additional
opaque field at the end for storing the corresponding GLXFBConfig.

The problems:

- XMatchVisualInfo() also returns an XVisualInfo structure, but it uses
memory allocated by the caller. Ugh. So there would need to be some
way for VirtualGL to detect whether an XVisualInfo structure passed to a
GLX function contained the opaque field. Although it would be the
rarest of cases for a 3D application to call glXGetConfig() with a
visual returned by XMatchVisualInfo(), VGL would still have to handle
that case.

- If an application used XGetVisualInfo() to return a list of all
visuals, VirtualGL wouldn't necessarily know how many XVisualInfo
structures would be needed in order to accommodate all of the
GLXFBConfigs, because VGL doesn't open a connection to the 3D X server
until a GLX function is called.

- Not sure whether it would mess anything up if XGetVisualInfo()
returned multiple XVisualInfo structures, some of which had the same
visual ID.

Maybe someone else has some ideas regarding how to improve upon this.
This is really the #1 compatibility problem that VirtualGL has. In
order to support X-server-less GPU access (i.e. a GLX-to-EGL
interposer), VirtualGL's visual matching system is going to have to be
overhauled anyhow (because, since EGL doesn't have double-buffered
Pbuffers, it will be necessary to emulate those using FBOs-- effectively
decoupling visual attributes like double buffering, stereo, stencil
buffering, etc. from the actual EGLConfigs.)

DRC

[DRC]

I ultimately went with a solution similar to the one that OpenText
submitted back in the day
(https://sourceforge.net/p/virtualgl/mailman/message/27118523/), and
that solution is now available for testing in the dev branch. The faker
now separates 2D X server visuals into categories based on visual depth
(and, if depth == 32, bits per component as well), visual class, and
stereo capabilities (i.e. the characteristics of 2D X server visuals
that VirtualGL actually cares about.) For each of those categories, the
faker cycles through all of the permutations of desired OpenGL rendering
attributes (alpha channel, double buffering, stencil buffering, depth
buffer, and multisampling) and assigns as many permutations of those
attributes as possible to the available 2D X server visuals in the
category. This works well with X proxies, such as TurboVNC, that have a
built-in software OpenGL implementation, because that gives those X
proxies a lot of available visuals. X proxies that lack a built-in
software OpenGL implementation (such as TurboVNC < 2.2 or TurboVNC 2.2+
when started with '-extension GLX') will generally only have two or
three visuals, so VirtualGL tries to front-load the rendering attribute
permutations in such a way that the first visual in a given category has
all of the rendering attributes, except multisampling, that an
application would want. Since VGL_SAMPLES now affects "default FB
configs" that are assigned to "unknown" visuals (visuals that aren't
obtained from glXChooseVisual() or glXGetVisualFromFBConfig()),
VGL_SAMPLES can easily be used to attach multisampling properties to the
first visual, if necessary. Thus, while VGL_DEFAULTFBCONFIG still
works, it is now mostly useless, and documentation for it has been
removed. The only thing that VGL_DEFAULTFBCONFIG can do that
VGL_FORCEALPHA and VGL_SAMPLES can't already do is disable features
(double buffering, alpha channel, stencil buffer) for all default FB
configs, and there shouldn't ever be a need to do that with modern
applications and GPUs.

I also wrote a script that catalogues the output of 'vglrun glxinfo'
with many different permutations of VGL and Xvfb settings. I now
maintain an offline Git repository with expected results from this
script on my three main VirtualGL test platforms, so that should help
prevent regressions in the VirtualGL visual matching algorithms.

I experimented with attaching a matching GLXFBConfig to the X Visual
structure using the XExtData mechanism, but that didn't work very well,
because X Visual structures survive for the entire duration of the X
connection (as opposed to XVisualInfo structures, which are created and
destroyed on demand at the behest of the calling program.) Also, since
the VirtualGL VisualHash class hashes XVisualInfo pointers to
GLXFBConfigs, it already accomplishes what I would have tried to
accomplish below. I was, however, able to use the XExtData mechanism to
clean up the visual attribute table in glxvisual.cpp (the VA table is
now attached to the Screen structure, since that data is per-screen) and
eliminate the display hash (which tracks the X displays that should be
excluded from GLX interposition-- mainly for use by ParaView and such.)

VirtualGL now also maintains its own FB config structures, and pointers
to these are passed back to the 3D application as GLXFBConfigs. That is
about to be necessary as I refactor the back end to use EGL and FBOs.