Issue 419 in zxing: Image format not supported (rgb565)

[zx...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 419 by johnvill...@gmail.com: Image format not supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

When BarcodeScanner is opened zxing throws the following error:

E/AndroidRuntime( 2056): Uncaught handler: thread Thread-10 exiting due to
uncaught exception
E/AndroidRuntime( 2056): java.lang.IllegalArgumentException: Unsupported
picture format: 4/rgb565
E/AndroidRuntime( 2056): at
com.google.zxing.client.android.camera.CameraManager.buildLuminanceSource(CameraManager.java:302)
E/AndroidRuntime( 2056): at
com.google.zxing.client.android.DecodeHandler.handleMessage(DecodeHandler.java:53)
E/AndroidRuntime( 2056): at
android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime( 2056): at android.os.Looper.loop(Looper.java:123)
E/AndroidRuntime( 2056): at
com.google.zxing.client.android.DecodeThread.run(DecodeThread.java:84)

And it stops barcode recognition.

[zx...@googlecode.com]

Comment #1 on issue 419 by johnvill...@gmail.com: Image format not
supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

Ooh... sorry to not include the following information:
Android Version: 1.5
Device: Sciphone N19
Architecture: ARMv4t (arm9)

[zx...@googlecode.com]

Updates:
Status: NotABug
Labels: -Priority-Medium Priority-Low

Comment #2 on issue 419 by sro...@gmail.com: Image format not supported
(rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

Android devices are required to support YUV preview format. If yours
doesn't it's a platform problem. This
indicates the device was asked for YUV and didn't return it. It's a device
bug.

[kslater]

Not to be argumentative, but where is it stated that all Android
devices are required to support YUV preview format? I found a comment
in some of the Android documentation, but it didn't seem to be hard
requirement, more of an expectation?

As a side note, I've been working on a couple of devices that also
only seem to support rgb565 previews. I've been working on some
refactoring of the ZXing Android source to that end, but I would
appreciate any advice or comments from the development team on whether
they think such a thing is possible (and would perform well enough).

[Sean Owen]

Erm... mostly because Daniel said so I guess. Yes it's mentioned in
the docs somewhere too though I had the impression it was supposed to
be required. Certainly I have only seen one rogue-looking device that
ever reported this as the default format.

The nice thing about YUV is that the luminance is already extracted in
a Y-plane upfront. RGB565 is going to take a little more work to
support, since you have to unpack a pixel from every successive pair
of bytes and convert it to luminance. It's going to slow down
performance.

I would copy the skeleton of YUVPlanarLuminanceSource if you go this
way. And use the efficient RGB-luminance conversion in javame/'s
LCDUILuminanceSource. If it works and you're interested in
contributing it we can add it to the project.

I should have really put it that way. It's not hard to support but I
haven't been interested in writing it since it had seemed like only
something weird devices would need.

[kslater]

Sean,

Thanks for taking the time to get back to me.

I'll look into what you suggested. I also found an RGBLuminanceSource
class in the test app and I began working to see if that could be used
as a starting point. Do you know where I might find some detailed
documentation on the image formats? I knew that RGB images needed to
have 16 bits to support each pixel, but I didn't realize that the file
format laid those out as two bytes / pixel (makes sense though). Do
you know which comes first in the file format? High order byte or low
order byte?

[Sean Owen]

I am pretty sure each two bytes are: RRRRRGGG GGGBBBBB
This code that does this conversion seems to confirm it:

http://android.git.kernel.org/?p=platform/build.git;a=blob_plain;f=tools/rgb2565/to565.c;hb=HEAD

Off the top of my head, here's how you convert some RGB565-encoded
bytes to luminances. You probably can't use it directly but this is
the gist. I bet you can optimize it a bit more, and probably should
bother.

byte[] rgb565 = ...
int[] luminances = new int[rgb565.length >> 1];
for (int i = 0; i < rgb565.length; i += 2) {
int b1 = rgb565[i];
int b2 = rgb565[i+1];
int r = (b1 & 0xF8) >> 3;
int g = ((b1 & 0x07) << 3) | ((b1 & 0xE0) >> 3);
int b = (b2 & 0x1F);
luminances[i >> 1] = (r + (g << 1) + b) >> 2;

[Daniel Switkin]

To follow up, yuv420sp is the standard color space for camera preview
frames. Every device must support it, and also use it as the default
if the app doesn't specific a different one. This is part of the
Android Compatibility Suite although it may not appear in the SDK
docs.

Daniel

[zx...@googlecode.com]

Comment #3 on issue 419 by aricblumer: Image format not supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

Could you please give a reference where "YUV" preview is required for
Android devices? The Android 2.1 Compatibility Definition document [1]
doesn't even include the string "YUV", and it's pretty clear that YCbCr is
not required in section 8.9.

[1]
http://static.googleusercontent.com/external_content/untrusted_dlcp/source.android.com/en/us/compatibility/android-2.1-cdd.pdf

[Daniel Switkin]

Dan, can you comment on where this requirement is documented please?

Thanks,
Daniel

[zx...@googlecode.com]

Comment #4 on issue 419 by morrildl: Image format not supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

@aricblumer - NV21 is a specific encoding of YUV. For some details, see:
http://www.fourcc.org/yuv.php YUV alone is not sufficient since it doesn't
specify byte ordering within each field, nor whether it's a planar
representation, etc.

So, by specifying NV21 as the default preview format, we are specifying a
byte-specific representation of YUV for the preview format.

Section 8.9, item #2 in the bulleted list.

[zx...@googlecode.com]

Comment #5 on issue 419 by aricblumer: Image format not supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

There is a huge difference between a default and a requirement. Section
8.9 #2 is quite clear that it only applies when the default preview format
is used. That default preview format is not required.

So back to my original question: Could you please give a reference
where "YUV" preview is required for Android devices? If it's not required,
then zxing should not require it.

[zx...@googlecode.com]

Comment #6 on issue 419 by sro...@gmail.com: Image format not supported
(rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

Are we reading the same text?

Device implementations MUST implement the following behaviors for the
camera-related APIs [Resources, 27]:

1. If an application has never called
android.hardware.Camera.Parameters.setPreviewFormat(int), then the device
MUST use android.hardware.PixelFormat.YCbCr_420_SP for preview data
provided to application callbacks.

2. If an application registers an android.hardware.Camera.PreviewCallback
instance and the
system calls the onPreviewFrame() method when the preview format is
YCbCr_420_SP, the
data in the byte[] passed into onPreviewFrame() must further be in the NV21
encoding format.
(This is the format used natively by the 7k hardware family.) That is, NV21
MUST be the default.

It's pretty black and white there: The device must use YCbCr_420_SP / NV21
(nee "YUV") as the default, supported preview format. Whether the app has
set a preview format is irrelevant. This requirement cannot be fulfilled
unless the format is supported -- it happens to be a moot point for an app
that *does* set a preview format, but, naturally an app can exist that does
not. How does a device behave correctly for that app without supporting
YUV, in light of the text above?

Backing up, what's the problem you're facing? It's not desirable to use RGB
here if you don't have to, since it will involve luminance conversion
overhead. If you have a device that can't support YUV, then the CDD doc
seems to be pretty clear that's the device problem, but, you can still
write RGB support, per above. You're welcome to do it, and even contribute
if you like.

[zx...@googlecode.com]

Comment #7 on issue 419 by aricblumer: Image format not supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

Yes, we're reading the same text, but I believe the confusion lies with the
use of the term "application" in this context. From the Linux side of
things, I did not take that as the end-user code, but as the Video4Linux
CameraHardwareInterface code. Our camera produces Bayer format images
(converted to RGB565), not YCbCr, so we call setPreviewFormat() to let the
system know that and satisfy requirement 1. Works fine.

There is also a comment in the Android code that indicates YCbCr is not
required: From
frameworks/base/graphics/java/android/graphics/PixelFormat.java:

/**
* YCbCr formats, used for video. These are not necessarily supported
* by the hardware.
*/
public static final int YCbCr_422_SP= 0x10;

If they meant only end-user code, then I understand your viewpoint. But,
IMHO, if they wanted to require that all cameras produce YCbCr_420_SP, they
should have listed that at the beginning of section 8.9 where it does list
the camera's requirements, rather than imply it in the context of software
API behavior (and then say the opposite in the code).

Regardless, it was fairly straightforward to get an RGB video source
working, although RGB to grayscale in Java is pretty slow.

[zx...@googlecode.com]

Comment #8 on issue 419 by sro...@gmail.com: Image format not supported
(rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

I'm certain the CDD is specifying requirements on the API exposed to
applications here. It's differentiating between "implementation"
and "application". The method it mentions is an application API method. And
at least, there's the confirmation of two people actually working on this
bit of Android.

I read that comment as it reads: the hardware itself may do as it likes.
The implementation must however expose particular behavior to applications.
This implies the implementation must convert to YUV if the hardware does
not. It is not required that the hardware produce YUV, so it doesn't say
that. This all looks consistent to me.

[zx...@googlecode.com]

Comment #9 on issue 419 by sro...@gmail.com: Image format not supported
(rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

I guess a soft corollary is this, which is I think what you're getting at:
if the upstream Android source code does not have a sort of 'driver' for
hardware that's not already producing this format, then it can't be used
without modification with cameras that don't output YUV. (I don't know the
source well enough to know that it doesn't have such support, but for the
sake of argument...)

I don't know if that means Android would like to have a patch with such
support, or whether the idea is you'd want to patch and maintain this
difference in your downstream source to go with your device. I think it's
the latter. You'd want to be able to support this translation inside the
implementation anyway, rather than patching one application.

[zx...@googlecode.com]

Comment #10 on issue 419 by aricblumer: Image format not supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

It has been a while since I've been able to work on this. I've converted
one of our camera interfaces from RGB to YCbCr for a camera that supports
YCbCr. To my surprise, Android shows the preview in grayscale rather than
color in the camera application (at least as of Android 2.1 r2).
Investigating, I found code like this
(frameworks/base/libs/surfaceflinger/Layer.cpp in this case):

case GGL_PIXEL_FORMAT_YCbCr_422_SP:
case GGL_PIXEL_FORMAT_YCbCr_420_SP:
// just show the Y plane of YUV buffers
bpp = 1;
break;

Similar comments are in frameworks/base/libs/surfaceflinger/LayerBase.cpp.
So it appears that Android requires preview in a format that it does not
fully support. I'm not sure what direction we'll go, but grayscale preview
is out of the question.

[zx...@googlecode.com]

Comment #11 on issue 419 by sro...@gmail.com: Image format not supported
(rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

Bleh, you may be on to something there.
But I think most (?) Android device cameras output YUV already, and of
course preview frames are available in color. What's the difference here
again?

[zx...@googlecode.com]

Comment #12 on issue 419 by aricblumer: Image format not supported (rgb565)
http://code.google.com/p/zxing/issues/detail?id=419

There are 3 differences I can think of:

1. The phone vendors have already fixed Android to handle YCbCr properly in
their previews.

2. They have hardware overlays or accelerators that convert the YCbCr of
the camera preview back to RGB for LCD display without any processor
intervention. This option is not available on our platform.

3. The publicly available Android source is not what the phone vendors are
using.