lossless compression?
ario...@gmail.com
However TurboVNC (and probably TigerVNC too) seem to use highly optimized JPEG compressor to transfer modified framebuffer sections.
Wiki says, that comparing to original TightVNC one feature - screen resizing - was dropped, that could result to genuine blurriness of source screen video.
So, it seems, form common sense if nothing else, lossless codecs would be better fit for remote screen type software, rather than natural-life optimized pro-blurriness JPEG
However... Speed. PNG compression speed is comparable to JPEG at best - charts from https://jacksondunstan.com/articles/2117
But, recently there appeared other lossless codecs.
Zstd packing in PNG container is claimed to be 500 lines of code, 6% compression time of PNG and 66% file size - https://github.com/catid/Zpng
LZ4 shows great compression speed, but not so great ratio (granted ,the source data was not screencast but something else) - https://github.com/inikep/lizard
There is also FastLZMA2 library, which is perhaps same for LZMA2 that libjpeg-turbo is to libjpeg
Were there tests about adding loss-less image compression?
The very claim that zstd lib can be used with as little as 500 LoCs seem inspiring.
I wonder how it could rate vs jpeg-turbo in the memory-processor-quality triangle
DRC
Possibilities for lossless compression:
1. Lossless Tight+Zlib
Background:
TurboVNC, LibVNCServer, and TigerVNC all use the TightVNC encoder, but I made numerous tweaks to that encoder to maximize performance and compression ratio for 3D and video workloads while maintaining similar performance and compression ratio to TightVNC for 2D workloads. The initial research (https://turbovnc.org/pmwiki/uploads/About/tighttoturbo.pdf) that resulted in the current TurboVNC encoding methods (Tight+Perceptually Lossless JPEG, Tight+Medium-Quality JPEG, Tight+Low-Quality JPEG, Lossless Tight, and Lossless Tight+Zlib) was conducted in 2008, while I was still working for Sun Microsystems and developing TurboVNC as part of their Shared Visualization product. This research concluded that, with respect to 3D and video workloads, using high lib levels in the TightVNC encoder isn't useful, so TurboVNC only exposes the zlib levels that were found to be useful. I conducted similar research specific to TigerVNC in 2011 (https://turbovnc.org/pmwiki/uploads/About/turbototiger.pdf), which brought the performance of TigerVNC (mostly) in line with that of TurboVNC. (At the time, I was floating the idea of re-basing TurboVNC on the TigerVNC code base, but that didn't happen for several reasons. Also note-- the rough equivalents to Tight+Perceptually Lossless JPEG, Tight+Medium-Quality JPEG, and Tight+Low-Quality JPEG in TigerVNC are Compression Level/Quality Level 1/8, 2/5, and 2/1, respectively.) In 2012, in the context of integrating the TurboVNC encoder enhancements into LibVNCServer, the LibVNCServer developers requested that I perform follow-up research to demonstrate that:
(a) it was possible, using the TurboVNC encoder, to produce similar compression ratios to the highest compression levels in TightVNC. This research resulted in a new encoding method, which is implemented as Compression Level 9 in the TurboVNC Server, but since that new encoding method is not generally beneficial for 3D and video workloads, it is not exposed by default in the TurboVNC Viewer GUI. The research also concluded that any TightVNC compression levels over 5 are basically useless. Going from CL 5 to CL 9 results in only about a 2% improvement in compression ratio at the expense of a 2X decline in performance.
(NOTE: these metrics were all measured using the TurboVNC Benchmark Tools-- https://github.com/TurboVNC/vncbenchtools-- with the same techniques and RFB session captures used in the initial TurboVNC encoder research. The 2D session captures are the same ones that Constantin used when designing TightVNC. The 3D session captures are of my own design. I don't claim that these are representative of all modern applications. A much more thorough analysis, however, would require obtaining numerous session captures from actual users of modern 2D and 3D applications, and that would require a lot of time and money that I don't have. I did the best I could with the resources I had.)
(b) that the TurboVNC encoder could perform at least as well as the unmodified TightVNC encoder using both libjpeg and libjpeg-turbo with several different JPEG quality settings.
You can read all about this follow-up research here: https://sourceforge.net/p/libvncserver/mailman/search/?q=turbovnc.
Technical notes:Lossless Tight+Zlib is similar to the other TurboVNC encoding methods, but it uses zlib for all subrectangles ("subrectangles" are produced from an RFB rectangle by the TightVNC encoder after it separates any significant regions of solid color), whereas most of the other encoding methods use libjpeg-turbo for subrectangles that have a high number of unique colors. Since zlib is slower and generally doesn't compress as well as libjpeg-turbo, that necessitates a different mix of TightVNC encoder settings. (More specifically, Lossless Tight+Zlib has a generally higher "palette threshold"-- the number of unique colors beyond which the encoder switches from indexed-color encoding to full-color encoding for a particular image tile.) The TurboVNC Server takes advantage of the Intel SSE2-accelerated zlib implementation in order to improve the performance of Lossless Tight+Zlib by about 20-40% (https://github.com/TurboVNC/turbovnc/issues/95). However, the performance and compression ratio of Lossless Tight+Zlib still falls somewhat short of the Tight+JPEG encoding methods-- generally about 60% slower with 20% worse compression vs. Tight+Perceptually Lossless JPEG and 65% worse compression vs. Tight+Medium-Quality JPEG (measured across all 20 aforementioned session captures, both 2D and 3D.)
2. PNG
The article you referenced compares PNG and JPEG decompression using Adobe Flash, and Adobe products do not use libjpeg-turbo, to the best of my knowledge (and, even if they did use libjpeg-turbo now, it's unlikely that they did in 2013.) In my testing, libjpeg-turbo is generally always at least several times faster than libpng, so even though there is a Tight+PNG encoding type in the RFB specification, performance has been the primary impediment to using that encoding type. However, it would be worthwhile to test a custom build of libpng that uses the aforementioned SSE2-accelerated zlib implementation. That may produce better results than the Lossless Tight+Zlib encoding method, since the latter somewhat naively compresses subrectangles using zlib, whereas PNG has additional pre-compression filters that improve compression ratio and performance. It would also be worthwhile to test whether these pre-compression filters work better than the filters that are part of the Tight/Turbo encoder-- the filters that split rectangles into subrectangles and apply different subencoding types based on the number of unique colors in a given subrectangle. The TurboVNC Benchmark Tools can be modified to test this and to determine, for the set of 20 RFB session captures, the appropriate PNG settings that will balance performance and compression ratio. That analysis takes a lot of time, though, and since I'm an independent developer, time is money. I am very happy to conduct that research on behalf of any organization that is willing to pay for my labor, but otherwise, the research is unlikely to happen anytime soon.
3. LZ4
I looked into LZ4, but the problem is, as you indicate, poor compression ratio. I never got as far as testing LZ4 with the TurboVNC Benchmark Tools, but based on Google's analysis with the Silesia Corpus, LZ4 is either 7x faster than zlib level 1 with 23% worse compression, or it is 2.5x slower than zlib level 1 with similar compression. Thus, while a Tight+LZ4 encoding method might be a suitable replacement for the Lossless Tight encoding method in TurboVNC, which is designed only for high-speed LAN use, Tight+LZ4 wouldn't be very useful on a WAN. Also, Lossless Tight itself is not very useful anymore. That encoding method was originally designed for slow SPARC servers, on which JPEG compression-- even with Sun's VIS-accelerated mediaLib codec-- was prohibitively expensive. These days, CPUs are fast enough that there is generally no reason not to use Tight+Perceptually Lossless JPEG on a LAN, unless you are paranoid about image quality (in which case the Automatic Lossless Refresh feature in TurboVNC provides a way to at least partially mitigate that paranoia without using Lossless Tight, although I freely admit that there are limitations to that feature.)
4. Zpng and FastLZMA2
I have not investigated these, but the author's assertion that
Zpng is "somehow more effective" than PNG does not inspire
confidence. :| This would be a good thing to test in the context
of the Tight+PNG research I proposed above, but again, since I'm
an independent developer, that research will likely never happen
without funding.
5. Lossless JPEG
Lossless encoding is part of the original JPEG specification (ITU T.81 | ISO/IEC 10918), and it should be possible to implement this form of encoding in libjpeg-turbo, but it is unknown how well it would perform. libjpeg-turbo is currently an official ISO reference implementation of the JPEG standard, but ISO maintains another reference implementation solely to demonstrate lossless JPEG, since libjpeg-turbo doesn't support that mode. So, from the point of view of libjpeg-turbo, it would be beneficial to include support for lossless JPEG, but my own-- admittedly non-exhaustive-- research (https://libjpeg-turbo.org/About/SmartScale-Lossless) demonstrated that PNG always compresses better than lossless JPEG, so from the point of view of VNC solutions, lossless JPEG is not likely to be very useful.
As far as image quality, the reason why JPEG has historically produced "blurry" images is because many applications that use the libjpeg API do not change the default 4:2:0 chroma subsampling setting in that API. Thus, many applications that compress JPEG images with libjpeg[-turbo] discard chroma components for every other pixel along both the X and Y image axes. This produces some perceptual quality loss even at the highest JPEG quality levels-- particularly for lines and text and other sharp, non-photographic features. However, turning off chroma subsampling, which TurboVNC and TigerVNC do when you select the highest-quality Tight encoding presets (Quality Level 6 or above in TigerVNC, or Tight+Perceptually Lossless JPEG in TurboVNC), produces perceptually lossless JPEG images if you use a high enough JPEG quality (generally 90 or above for most viewing conditions, but both TurboVNC and TigerVNC use a higher quality than this for their perceptually lossless presets in order to be sure.) Because it is already possible to produce perceptually lossless images using TurboVNC and TigerVNC, there is little need for a true mathematically lossless encoding type, except for use with medical visualization and other image-quality-critical applications. Automatic Lossless Refresh provides a way to address that need somewhat, but medical viz people are ultra-paranoid and generally want some assurance that every single image rendered on the server will reach the client in a mathematically lossless form. This is because any loss introduced into, for instance, an MRI animation may be incorrectly diagnosed as an anomaly.
DRC
For usual "desktop" software, at least "old school", before Windows Aero blurry styles, PNG usually made better screenshots than JPEG: files were both better quality (crispy, rather than blurred edges full with artifacts) and smaller.
However TurboVNC (and probably TigerVNC too) seem to use highly optimized JPEG compressor to transfer modified framebuffer sections.Wiki says, that comparing to original TightVNC one feature - screen resizing - was dropped, that could result to genuine blurriness of source screen video.
So, it seems, form common sense if nothing else, lossless codecs would be better fit for remote screen type software, rather than natural-life optimized pro-blurriness JPEG
However... Speed. PNG compression speed is comparable to JPEG at best - charts from https://jacksondunstan.com/articles/2117
But, recently there appeared other lossless codecs.
Zstd packing in PNG container is claimed to be 500 lines of code, 6% compression time of PNG and 66% file size - https://github.com/catid/ZpngLZ4 shows great compression speed, but not so great ratio (granted ,the source data was not screencast but something else) - https://github.com/inikep/lizard
There is also FastLZMA2 library, which is perhaps same for LZMA2 that libjpeg-trubo is to libjpeg
DRC
"high lib levels" should be "high zlib levels".
Oops.