ImageAnalysis YUV_420_888 underlying format

[Javier Gerardo Martinez Salomon]

Hi everyone,

I've been working with the ImageAnalysis use case and CameraX version "1.0.0-rc03" I'm becoming familiar with the YUV_420_888 format, I've used YuvToRgbConverter(as well as my own versions) to convert the ImageProxy planes into Nv21 and then to a bitmap, however I've found that for some devices the conversion works fine and for others the first row of the image is in the wrong position.

Samsung S8
This device works fine, when converting the ImageProxy planes to Nv21 I get the correct image and the planes byte buffers look good, here is a sample image captured.

Xiaomi Mi A2

This is one of the devices I'm having troubles with, when I convert this YUV planes into Nv21 the first "row" of the image appears to be misplaced and seems to fit rather to the end, see the attached image, if you pay attention to the first column from left to right, you'll notice that they do not fit and should be on the far right.

For the Mi A2 the result is the same no matter which method I try, and it's the same result if I use an image size where the planes include padding or not. This is not the only device with this problem, I've also found it on a Samsung Tab E and a Kyocera E6560, and it even happens if I just narrow it down to the Y plane and ignore the U and V planes.

The YUV planes buffer bytes
When using the deprecated Camera API the Nv21 byte arrays come just fine and populated with the correct data for all the devices, it's just CameraX where I'm having this problem.

Also what I was able to find, is that there is a strange pattern on the conflicting devices vs the S8 working device.
For the S8 the YUV planes come with the byte data just fine, while for the conflicting devices the Y plane first index is always 0 and the U/V planes first index is always -128, If I take the Y plane and ignore the first index on the conflicting devices, the image comes as it should, without the first row shifted, on the other hand if I do the opposite and take the working S8 device and make the first Y plane byte 0 on purpose, the output image comes shifted as the conflicting devices.

Of course, I have a few questions about this,

1. What's the reason behind these values at the 0 index for the conflicting devices?
   1. Should that byte be skipped? or consumed differently? how can I workaround this conflict?
2. Is there any way that I can query the YUV_420_888 underlying format to know if it's Nv21, Nv12, YV12, etc?
3. On image sizes that add padding to the planes, the planes are not padded on all rows, the last row has no padding, why is that?
4. Are these planes buffers individual pointers to a big single data structure in the C layer? or really 3 different ones?

I'd appreciate your help on this matter, since I've read almost every entry related to YUV format on this group and the CameraX issue tracker.

[James Fung]

2. APIs in Image.java can help with usage, though finding specific formats may need pointer comparisons yet. 

3.  My *rough* understanding is it has to do with if the vertical resolution is some odd number, its ambiguous what to do with the last row of subsampled/interleaved data and so I suppose its difficult to guarantee in the format.  The strided rows are memory aligned and sized for efficient transfers, except the last row.

4. I think of it as storing pointers to memory, offsets and block sizes - not sure if the specific structs holding that data is helpful.

For the main question (1) I'm not sure this is a familiar issue and so we'll probably need to dig into the details.  To confirm - this is reproducible using  YuvToRgbConverter and saving to a bitmap on the listed devices? 

[Javier Gerardo Martinez Salomon]

Hi James,

Thanks for your help, yes, this is reproducible using the YuvToRgbConverter as well as my own implementations to turn YUV_420_888 to Nv21, so far I've only found the issue on the mentioned devices, the rest of devices seems to be fine, for the conflicting devices the following buffer values are always true in every received frame:

yPlane.buffer[0] == 0
uPlane.buffer[0] == -128
vPlane.buffer[0] == -128

Things I've tried:

• Different target sizes
  • Same issue
• Compare it to the Nv21 of the deprecated Camera API
  • The one returned by Camera API it's clean and works fine
    
• Use Camera2
  • Same issue
    
• Skip/Stub the first byte of the buffers
  
  • Works but I lose one pixel
• Stub the first byte of the buffers on devices that do not have the issue
  • The generated bitmap has the same issue as conflicting devices

It's worth to say that this is hard to notice on high resolution bitmaps, hence the reason I attached one captured with a low resolution. I'm willing to try any ideas you might have, if you need more image samples, YUV encoded files, or anything I can provide to expand on details just let me know.

Best Regards

[James Fung]

The deprecated Camera API working is interesting.  Either the data is different, or there's an issue interpreting the data in the Camera2/CameraX path.  Do the affected devices (Samsung Tab E/Kyocera) have the "legacy" support: 

check the camera hw level:

https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics?hl=en#INFO_SUPPORTED_HARDWARE_LEVEL

if it has: 

https://developer.android.com/reference/android/hardware/camera2/CameraMetadata#INFO_SUPPORTED_HARDWARE_LEVEL_LEGACY

[Javier Gerardo Martinez Salomon]

It's quite interesting, I can't say for sure that it is different data for all buffers, but narrowing it down to the Y plane, the first byte of the Nv21 byte array returned by the deprecated Camera API should be in theory the first byte returned by the Y plane buffer, in which case the deprecated API Nv21 byte array has a non zero value while the first byte of the Y plane buffer is always 0, this seems really strange.

As for the hardware levels, according to this snippet:
val supportedHardwareLevel = camera2CameraInfo.getCameraCharacteristic(CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL)

Xiaomi Mi A2 = LEGACY

Kyocera E6560 = LEGACY

Samsung Tab E = LEGACY

Let me know if I can do anything else

[Charcoal Chen]

Hi,

 

After investigating, the issue is caused by the source RGBA pixel buffer having a one-pixel shift issue. The produceFrame() function in camera framework converts an uint8_t* pixelBuffer to YUV_420 format. I can reproduce the issue on a LEGACY-level device and the first four pixelBuffer data of each frame are always 0 which are mapping to R/G/B/A data. This should be unexpected extra pixel data which will affect the converted YUV_420 result. Due to the unexpected extra pixel data, the RGB image converted from the problematic YUV_420 frame data will result in that the last pixel of each row will be displayed as the first pixel in the next row. The result will just look the same as the problematic image reported in this forum thread.

 

I can only reproduce this issue on LEGACY-level devices, but not all LEGACY-level devices have this issue. API level is not a condition to have this issue. Samsung Galaxy J2 Pro LEGACY/API 25 device has the issue but OPPA A75 LEGACY/API 25 device doesn't have this issue. There is no clear rule to indicate which kind of device might have this issue.

 

For more details about the RGB<->YUV_420 conversion, the Y data is mapped to the source data pixels, so it is also shifted one pixel. But the  U/V data are calculated by 2x2 block pixels of the source RGB data. When the source RGB pixel data is shifted one pixel, the U/V data will be calculated by incorrect 2x2 block pixels data. The color information of the last pixel data in each row will be incorrectly brought to the first pixel of the next row. Therefore, even if we shift the converted RGB buffer one pixel back, taking the reported problematic image as example, we will still see some color which belongs to the last pixel in the previous row occurs in the first pixel. If I do not misunderstand, RGB<->YUV_420 should not be a lossless conversion. It means that it is difficult to calculate out the original source RGB data when we only have the converted incorrect YUV_420 buffer data. There is still no solution/workaround found for this issue on the CameraX side.