Photometric Interpretation Mismatch with JPEG Baseline (0028,0004) vs. Pixel Data

[Giacomo Petronio]

We encountered a dicom image with the following characteristics:

• Transfer Syntax 1.2.840.10008.1.2.4.50 JPEG Baseline (Process 1)
• Photometric Interpretation "RGB"
• the compressed image has a colorspace of YBR

When we try to display the image, it appears "blueysh" and this seems related to the fact that we rely on the Photometric Interpretation to interpret the color space of the decoded bytestream.

If we interpret the decoded bytestream as YBR instead of RGB, the image colors appear to be correct.

Can we assume that this DICOM image has been produced with the wrong value of Photometric Interpretation?

[Victor Derks]

Hi Giacomo,

If it displays correctly when interpreting it as YBR, you can assume that the photometric interpretation for this image was wrong ;-)

The actual challenge is how to detect this in software reliable. There are systems that convert uncompressed RGB images to JPEG lossy and forget to update the photometric interpretation tag, which makes it more difficult for the receiver to do the right thing.
Table 8.2.1.-1 in PS.3.5 2025d allows usage of both YBR_FULL_422 and RGB as values for the photometric interpretation. But JPEG lossy images are almost always stored in YBR_FULL_422 as common jpeg codecs will do this automaticly when encoding to and from RGB. DICOM correction proposal  CP-1262 has a good discussion about it.
A helpfull check is to also always take into account which IOD your system is processing. For example the Multi-frame True Color SC Image IOD only allows Photometric Interpretation (0028,0004) to have the value YBR_FULL_422 when using JPEG lossy. This info can be used to decide if the Photometric Interpretation tag can be trusted or not.

Regards, Victor

P.S.1 There are also bad DICOM jpeg lossy images that are actually encoded in YBR_FULL instead in YBR_FULL_422, but this information is also present in the encoded bitstream and will be handled by the codec.
P.S.2 Color images encoded with JPEG lossless or JPEG-LS are almost always stored in RGB. Converting to RGB to YBR and back is not a lossless process for these transfer syntaxes and would defeat the purpose of using a lossless TS.

[David Clunie]

Don't forget that sometimes JPEG compressed images really are RGB components (have not been transformed to YCbCr as is the norm), e.g., Leica Aperio whole slide images. See also CP 1841.

To make sure that the codec handles this correctly (does not apply an inverse YCbCr transform automatically) even without the benefit of the DICOM Photometric Interpretation value, a creator can insert an Adobe APP14 marker with ColorTransform 0 = Unknown (RGB or CMYK) into the JPEG stream, and many decoders pay attention to this (e.g., [2]).

You will find this pattern in many of the DICOM Whole Slide images in IDC [3] that you can retrieve and test with:

SELECT DISTINCT PhotometricInterpretation

FROM `bigquery-public-data.idc_current.dicom_all`

WHERE Modality = 'SM' AND TransferSyntaxUID = '1.2.840.10008.1.2.4.50'

LIMIT 100

PhotometricInterpretation
RGB
YBR_FULL_422

David

1. https://exiftool.org/TagNames/JPEG.html#Adobe

2. https://docs.oracle.com/javase/8/docs/api/javax/imageio/metadata/doc-files/jpeg_metadata.html#color

3. https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=idc_current

[Marco Eichelberg]

There is another hint in the JPEG bitstream that indicates whether the color model YCbCr or RGB is used: The channels in the JPEG bitstream have numerical identifiers. For RGB images, these are typically 82, 71 and 66, i.e. the ASCII codes for 'R', 'G' and 'B' (I think this is even documented somewhere in the JPEG standard), in YCbCr images you will typically find other values such as 0, 1, 2.

Marco

[David Clunie]

Hi Marco

How common is the neat trick that you describe? I don't think I have ever seen it.

Normally the components are numbered 1, 2 and 3 (which JFIF requires, albeit for YBR), or 0, 1 and 2, regardless of whether the components are RGB or YBR.

Are you aware of any codec that writes 'R', 'G' and 'B' as the component identifiers, or checks them?

AFAIK, the JPEG standard text [1] says nothing more about component identifiers other than that they need to be unique; Section B.2.2 "Frame header syntax" says:

"Ci: Component identifier – Assigns a unique label to the ith component in the sequence of frame component specification parameters. These values shall be used in the scan headers to identify the components in the scan. The value of Ci shall be different from the values of C1 through Ci - 1."

I do not think I have ever encountered what you suggest documented in any of the JPEG-1 standard documents or descriptions of them.

David

1. https://www.w3.org/Graphics/JPEG/itu-t81.pdf

[Marco Eichelberg]

I cannot say how common such images are, but I have seen examples. The output of your jpegdump tool on one such example will look like this:

Offset 0x0197 Marker 0xffc0 SOF0 Huffman Baseline DCT length variable 0x11
        JPEG_SOF_Parameters:
                 SamplePrecision = 8
                 nLines = 960
                 nSamplesPerLine = 1280
                 nComponentsInFrame = 3
                 component 0
                         ComponentIdentifier = 82
                         HorizontalSamplingFactor = 1
                         VerticalSamplingFactor = 1
                         QuantizationTableDestinationSelector = 0
                 component 1
                         ComponentIdentifier = 71
                         HorizontalSamplingFactor = 1
                         VerticalSamplingFactor = 1
                         QuantizationTableDestinationSelector = 0
                 component 2
                         ComponentIdentifier = 66
                         HorizontalSamplingFactor = 1
                         VerticalSamplingFactor = 1
                         QuantizationTableDestinationSelector = 0

Apparently these images are generated by some Adobe JPEG encoder. I cannot find the official documentation for this hack (it's not in ITU T.81), but here is a web page documenting how the IJG JPEG implementation tries to determine the color space when decompressing a JPEG baseline image - the component IDs are one of the factors taken into account: https://entropymine.wordpress.com/2018/10/22/how-is-a-jpeg-images-color-type-determined/

Marco