Mirrored green images in ImageAnalyzer

[roman.r...@jumio.com]

Hello,

we are facing issues with CameraX 1.2.1 on at least one device (LG Leon LG-H320 Android 5.0.1) where we get a mirrored yuv frame with wrong u and v information in the ImageAnalysis UseCase while the captured image in the ImageCapture UseCase is perfectly fine. 

We used the CameraXBasic Demo that we updated to version 1.2.1.

Why is the ImageAnalysis frame bad while the captured image is perfectly fine? 

Is there a way for us to detect phones that produce wrong frames in the ImageAnalysis UseCase?

Attached you'll find the following files:

• ImageCapture.jpg - the jpg from the ImageCapture UseCase
•  all 3 planes in the format planeProxy<index>_<imageWidth>_<imageHeight>_<pixelStride>_<rowStride>
• nv21_640_480 - all planes merged together to a nv21 buffer
• ImageAnalysis.png - nv21_640_480 converted to png

Thanks, Roman

[Kailiang Chen]

Hi Roman,

1. Does it happen on other devices?

2. How do you merge all the planes together to a nv21 buffer?

3. What happen if you merge planes to a nv12 buffer?

IIUC, the YUV_420_888 format could be NV21 or NV12 or YV12 or I420 (YU12).

Best,

Kailiang

[roman.r...@jumio.com]

1. At the moment we see this on only this device - but we are not very confident to roll this out to a broader audience as long as we are not sure whats exactly happening here.

2. I tried multiple approaches - including using the YuvByteBuffer from the samples - the result is the same on every conversion. The code works fine on all the other tested devices. Also, if you check the planeProxy files (which are the raw buffers that we get from the ImageProxy in ImageAnalysis.Analyzer) then they seem to be wrong already - planeProxy_0 (the y part) is mirrored and the planeProxy_1 and 2 (u and v) seem to be incomplete (as they are almost entirely 0x00) 
3. Is the underlying yuv format (NV21, NV12, YV12, ..) defined by pixelstride? And as I understand the YuvByteBuffer code it should take this into account. Nevertheless this would not correct the mirrored y buffer.

[Kailiang Chen]

Hi Roman,

1. I think the YuvByteBuffer sample is problematic. It's not handling YUV_420_888 format correctly. 

To differentiate underlying yuv format correctly, you need to check pixelstride and also the uv plane's memory address

https://chromium.googlesource.com/libyuv/libyuv/+/HEAD/source/convert_argb.cc#4317

2. What do you mean by "mirrored" for y buffer? Not rotated correctly to appear upright? Have you called setTargetRotation for ImageAnalysis?

3. Regarding u and v are almost 0x00, have you tried other scenes? u, v plane each should have 1/4 data bytes of y plane, otherwise, it seems incorrect.

We will check to see if we could reproduce this issue on old LG devices. 

Let us know if you found other problematic devices.

Best,

Kaiilang

[roman.r...@jumio.com]

Hey Kaiilang,

1. Do you maybe also have a Kotlin example for that?

2. Please have a look at the ImageAnalysis.png or planeProxy_0 (y-plane) - the image of the keyboard is mirrored on the y axis (see keyboard characters N,J,Z, R) - the planeProxy_0 is the direct dump of the ByteBuffer in imageProxy.planes, therefore this does not happen because of a wrong conversion method. setTargetRotation is set and it also works on other Devices.

3. Only the first 0x140 (=320) Bytes are set in the u and v buffer, this is exactly the rowStride. Therefore I only get u and v for 1 row instead of 240 - the image is 640x480 therefore the size of u and v should be 320x240 (1/4) - please see the buffer dumps planeProxy_1 and planeProxy_2

I also tried the rgba 8888 output format by adding .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) to the ImageAnalysis.Builder which also returns a green mirrored image. I attached it below if you want to also have a look.

Thanks for your help!

Regards, Roman

[Kailiang Chen]

Hi Roman,

1. Sorry, we don't have java/kotlin version of that. You might need to use JNI to wire up to the native code.

2. Yes, the planeProxy_0 is mirrored.

3. Yes, the planeProxy_1/2 has only 1 row of data.

I will try to see if I can reproduce on old LG devices.

This could be a device specific issue.