What is “CCIR601_sampling” in libjpeg?

[Greg Kennedy]

As the title states: Both the jpeg_compress_struct and the jpeg_decompress_struct in libjpeg have a field defined like this:

boolean CCIR601_sampling; /* TRUE=first samples are cosited */

I am having a hard time figuring out what this means, or how it's supposed to be used. If you try to set this flag to true, either for compression or decompression, libjpeg will simply trigger a fatal error with this message:

JMESSAGE(JERR_CCIR601_NOTIMPL, "CCIR601 sampling not implemented yet")

The "yet" is amusing because it's been this way for 20+ years now, at least back to libjpeg62.

So, what is CCIR601_sampling supposed to do? Is it meant as a user-settable parameter for compression, decompression, or both? Is it stored as part of the file format? And why has it never actually been implemented?

-Greg

[DRC]

To the best of my understanding, the libjpeg API and algorithms adhere to the RGB-to-YCbCr conversion formulae specified in CCIR 601 (now ITU-R Recommendation BT.601).  The "CCIR601_sampling" field in the libjpeg API is meant to allow for future support of co-sited Cb and Cr samples-- that is, to allow for the sample arrangement used in MPEG-2.  That sample arrangement is non-planar and specifies a row of Y samples, then a row of packed Cb/Cr samples, then another row of Y samples, etc.

Tom Lane's original goal for libjpeg was to encourage convergence around a common subset of JPEG format features, by providing a FOSS library that implements that subset of features.  He moved on to other things (libpng and PostgreSQL, most notably) after libjpeg v6b, so certain features were left for future expansion.  It's remarkable that libjpeg v6b is so stable that, to this day, only one vulnerability (to my knowledge) has been discovered in that code base.  That stability, as well as a mass migration to Linux in the server room, led to libjpeg v6b becoming the de facto industry standard JPEG library in the 2000s.  It also led to libjpeg v6b becoming a sort of pseudo-standard for JPEG format compatibility.  Thus, the fact that Rec. 601 sampling isn't implemented in libjpeg v6b means that JPEG files with that sampling arrangement are basically non-existent "in the wild."  The JPEG specification supports other features, including a lossless mode, but ultimately, the de facto definition of the "JPEG format" converged to the subset of features implemented by libjpeg v6b (per Tom Lane's original goal.)  To this day, that same chicken-and-egg phenomenon means that web browsers don't support arithmetic-coded JPEG files, even though the patent on arithmetic coding expired long ago and libjpeg-turbo supports those files.

Beginning in 2009, Guido Vollbeding chose to evolve libjpeg in a backward-incompatible direction, as a demonstration of his ideas regarding expanding the JPEG format to handle DCT sizes other than 8x8 (ideas that were rejected by the standards committees, incidentally.)  A year later, I chose to evolve libjpeg in a backward-compatible direction that emphasized high performance.  libjpeg-turbo was originally just intended as a drop-in replacement for libjpeg v6b (hence the name "libjpeg-turbo"), to allow applications (video and remote display applications were early adopters) to take advantage of JPEG acceleration without modifying the applications.  Every other major feature in libjpeg-turbo has been funded by an organization that had a need for it.  The fact that Rec. 601 sampling is still unimplemented is a reflection of the fact that few people need it, and few people need it because libjpeg v6b never supported it.  The "CCIR601_sampling" field remains in the API because the API structures are exposed.  Thus, removing the field would break backward ABI compatibility, and backward ABI compatibility is one of the primary reasons (performance is the other) why libjpeg-turbo became the preferred open source JPEG library.

The format convergence facilitated by libjpeg benefited libjpeg-turbo, because it meant that we only had to focus on accelerating the most common JPEG algorithms and features-- those supported by libjpeg v6b.  At this point, supporting Rec. 601 sampling with full acceleration would require new SIMD color conversion algorithms on the various CPU architectures we support, so it would be a big project.

As far as the use of the word "yet", that appears in several places throughout the libjpeg code base.  "Yet" doesn't guarantee that something will ever happen.  It just means that it hasn't happened prior to this point in time.  It's a bit more optimistic to say "this feature isn't implemented yet" than to say "this feature isn't implemented", which carries an implication that it will never be implemented.  Truth is, I'll implement anything that someone pays me to implement, as long as it's part of the JPEG specification.  That includes Rec. 601 sampling.

DRC