OpenMAX and other hardware image and video encoders

[falde]

Did anyone look into using OpenMAX to do JPEG encoding on the GPU which is probably faster than doing it on the CPU? Preferably by using EGL to share a buffer with GL so that there are no extra data copy between these two. This would mean that the data transferred from the GPU would be in JPEG already meaning that less bandwidth are needed for readback.

OpenMAX can also encode to a video stream and having H264/H265 has obvious bandwidth advantages. There are also proprietary api:s like HVENC that may not be able to do zero-copy of video data. However it may be worth it when running a multi-user VNC server over an internet connection as HVENC outputs video that looks better than what JPEG or x265 produces. Most specifically it is worth it client side were hardware decode on such video is very efficient. Sending video instead of a JPEG stream obviously breaks the VNC specification a bit. But that is where other proprietary solutions are going. Such as for example Shadow Inc.

[DRC]

The advantages of H.264 are more mixed than obvious. It depends a lot
on the workload: https://turbovnc.org/About/H264. However, assuming
that GPU acceleration can be used to speed it up, there are enough
advantages to H.264 to make it worthwhile as an option at least.

These days, most people use VirtualGL with an X proxy (including our own
X proxy, TurboVNC), so it doesn't make much sense to spend a lot of time
improving the encoding in the VGL Transport unless TurboVNC can somehow
benefit from that as well.

I've looked into NVENC in the past, but more high-priority features like
the TurboVNC Session Manager and the EGL back end for VirtualGL
distracted me from that work. Here's the GitHub issue for it, though:
https://github.com/TurboVNC/turbovnc/issues/19

Referring to the issue above, I had intended to experiment with using
NVENC in a "naive" capacity, sending uncompressed frames from the GPU to
the X proxy via MIT-SHM, then sending the frames back to the GPU for
encoding. That would've been only a proof of concept, though.
Ultimately, taking advantage of NVENC or a similar GPU encoding solution
is going to require one of two things:

1. Modifying the TurboVNC Server so that it can optionally maintain its
framebuffers in GPU memory. It seems at least feasible to do this, now
that Xwayland has EGLstreams support. The idea is that the TurboVNC
Server could be built upon Xwayland, and the host's GPU-attached Wayland
device could be shared among multiple instances of the TurboVNC Server
in much the same way that the 3D X server is currently shared among
multiple instances of VirtualGL. There are probably 100 reasons why
this is problematic, though, and some of those may only reveal
themselves once I dig into the project. Whereas I think it's possible
to support OpenGL ES with an interposer approach
(https://github.com/VirtualGL/virtualgl/issues/66), supporting Vulkan
with a VirtualGL-style solution may not be possible
(https://github.com/VirtualGL/virtualgl/issues/37.) So if Vulkan is the
direction in which the industry is moving, then the writing is on the
wall vis-a-vis VirtualGL. Either some way can be found to support
Vulkan within the VirtualGL interposer, or VGL will be relegated only to
supporting OpenGL applications. The latter may not be such a huge
limitation for technical applications at the moment, but I could easily
imagine some framework like Qt moving from OpenGL to Vulkan on the back
end and thus forcing our hand.

2. Implementing a PBO handoff mechanism between VGL and the X proxy.
Step 1 of this would be implementing deferred readback in VirtualGL
(https://github.com/VirtualGL/virtualgl/issues/9), which would cause VGL
to maintain its buffer pool in GPU memory (using PBOs) rather than in
main memory. VirtualGL currently transfers the rendered OpenGL frames
from GPU memory to main memory within the body of a "frame
synchronization function" (glXSwapBuffers() when rendering to the back
buffer; glFinish() and glXWaitGL(), and sometimes glFlush(), when
rendering to the front buffer.) With deferred readback, VGL would
instead transfer the rendered OpenGL frames from the framebuffer to a
PBO within the body of a frame synchronization function, then it would
hand off the PBO to the transport thread, which could do one of two
things with it:

a. Hand off the PBO to an X proxy, using an as-yet-to-be-defined X
extension that works basically like MIT-SHM for PBOs. nVidia informed
me that doing this would require handing off the PBO using CUDA memory,
which is less desirable since it would be vendor-specific.

b. Performing GPU-accelerated encoding within VGL's transport thread
and handing off the encoded image to the X proxy using some
as-yet-to-be-defined compressed PutImage extension. Such an extension
is problematic, though, because if the rendered OpenGL frames are being
compressed outside of the X proxy, then the X proxy would have to
decompress them in order to mix OpenGL and X rendering (for instance, if
the application tries to call XGetImage() to fetch some of the
OpenGL-rendered pixels.) This would be difficult to implement without
introducing performance or conformance issues.

Note that deferred readback has benefits irrespective of GPU encoding.
It will conceivably eliminate the need for the VGL_SPOILLAST setting,
which is currently used to implement a sort of deferred readback only
for frames triggered by glFlush().

Honestly, (1) starts to look like a less problematic and more
future-proof path, assuming it's even possible. The primary thing
preventing any of the above from happening, however, is lack of money.
For instance, the technology was largely in place to begin work on the
EGL back end in 2015, but it is such a complicated project that I wasn't
able to get enough funding ducks in a row to begin work on it until two
months ago. The proposals above are going to be equally as complicated,
if not more, so they're going to require some speculative research.
Unfortunately, I can't afford to speculate on anything right now (2018
was a bad year for this project), so moving forward with this stuff is
going to require a company that is willing to pay me for some labor--
perhaps a lot of labor-- that might not ultimately bear fruit.

[DRC]

Note that, once deferred readback is implemented, it would also enable
an intermediate solution that could use the GPU to do YUV encoding and
then transmit the YUV-encoded pixels to the X proxy using X Video. That
would have the same limitations as a compressed PutImage extension,
though, in terms of the inability to mix X and OpenGL rendering, and it
wouldn't do anything to address the GPU-accelerated encoding problem.
There are basically no options I can think of here that would be generic
across multiple X proxies. All of the options would require the X proxy
to be modified in some way.

[falde]

The advantages of H.265 depends a lot on use case specific. There are low power processors with very weak CPU power but an excellent hardware decoder for H,265. If also supporting H.264 the Raspberry Pi Zero can play HD videos in H.265, but it struggles to run VNC.

Server side there are some very powerful H.265 and H.264 encoders. For example the one embedded in Nvidias GPU:s.

Using x265 could be useful on the server side if there the client is the main bottleneck but generally using a cheap GPU id probably better.

There are of course a situation where neither client and server overhead matters much. This is when bandwidth is the bottleneck. If I want to run remote desktop over an international satellite link I rather quickly spend a lot of money transferring frames. In this case buying high end workstations for both client and servers and doing software compression may be completely sane.

[falde]

Oh and if bandwidth is the only bottleneck I would be concerned with then a simple proxy which takes the current JPG stream and transcode it to H.265 may be sufficient. This is of course quite a lot of server side overhead but it still may be worth it. Another possibility could be to use a HDMI frame grabber and and compress that to a video stream. However that solution requires a separate HDMI port for each user which is rather far from ideal, but the advantage with it is that the software overhead is minimal. A virtual desktop server with one HDMI out per user and a terminal server with one HDMI in per user. HDMI in from a frame grabber can be fed right into a GPU for HDMI coding. For a dedicated remote workstation solution I believe that a HDMI frame grabber would work very well and require zero software on the server except perhaps mouse/keyboard redirection and such.

As money is an issue perhaps the HDMI frame grabber solution is the way to go. It is not multi user friendly but for the high end remote workstation that would work rather well. It allows business users to lock their expensive workstations into a server room and it allows home users to put the workstation/desktop in the garage and run games on any laptop.

[falde]

What about something like this.

1. A headless vnc server that do not capture/redirect video only keyboard and mouse

2. A vnc server that gets keyboard and mouse from the headless server and ads a video stream grabbed and compressed by ffmpeg to that.

3. A vnc client that instead of the standard JPEG stream plays an encoded video stream.

When the client connects it should communication what video and audio formats that are locally supported. Most modern PC's support H.265 while for example RPi only support H.264. And on many Android devices there are mp3 decoders but perhaps not some other codecs. I assume that most server admin will want to disable software encoding and most client's will do he same.

4. A vnc server that acts as a client but transcodes between video formats. Can be useful for sending H.265 over the network and still supporting H.264 devices. Ffmpeg can do zero copy transcoding in the GPU and even a cheap Nvidia GPU should be able to handle more than one stream.