MediaCodec and device compatibility

[Martin Storsjö]

Hi,


I've tried taking the new, long desired, MediaCodec API into use in a few
different use cases, and while it looks promising, there's still a few
issues that feels like they need documenting before the new API can be
relied upon properly.


First off - pixel formats. The scenario is that I'm using the MediaCodec
API to decode video, and I want to render the decoded YUV data myself, I'm
not passing any Surface to the system for rendering. Or I'm encoding raw
video data, captured from the camera via the preview image callback.


Since I want to use the decoded YUV data myself, I need to be able to
handle the pixel formats that the decoder produces. The
MediaCodecInfo.CodecCapabilities class
(http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html)
lists a vast array of pixel formats (about 40 or so), with only a
reference to the OpenMAX IL spec. Is there any guarantee about which pixel
formats the decoders on Android 4.1 compliant devices will provide? Or do
I have to be able to handle all those 40 different pixel formats, if some
unknown device with Android 4.1 that I don't have access to would choose
to use some of them? This is of course a bit exaggerated, I know, in
practice I probably would only have to support the relatively few YUV 420
formats, perhaps in addition to some of the YUV 422 versions. That's only
about 10 different formats.

But this list also contains the vendor specific
COLOR_QCOM_FormatYUV420SemiPlanar and
COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find information about
how these formats differ from the normal COLOR_FormatYUV420SemiPlanar and
COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL spec
doesn't know about these vendor extensions, obviously.) Are these the only
vendor specific formats that vendors are allowed to use, or is it ok if
qcom devices return e.g. the
QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (0x7FA30C03) pixel
format? Many current qcom hw decoders return data in this format, and it
is very nontrivial to use.

Will the CTS require device manufacturers to return data in some of the
"generally known/supported" pixel formats? If not, in order to use the raw
decoded data from the MediaCodec API, one would need to test things on one
device per chipset family/generation, more or less.

For video capture with the low level Camera class, this kind of issue is
already documented, see e.g.
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12.
That section says "This format is guaranteed to be supported for camera
preview images since API level 12; for earlier API versions, check
getSupportedPreviewFormats()."

For encoding, this at least is manageable where the codec can support more
than one format, one can request which one to use. For decoding, the API
might indicate that the codec supports a few different pixel formats, but
one does not know which one it actually will use until the decoder is
started.


I haven't had a chance to test this API on the newly released Nexus
devices, but I have played a little with it on the emulator and on a
Pandaboard. On the latter, the video decoder indicates that it supports
two pixel formats, 0x7f000100 which is
OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which I don't
even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.


Once we get past the issue of pixel formats, then we get to the issue of
how to use each pixel format. The MediaFormat class documents that the
KEY_COLOR_FORMAT key is encoder only, while one obviously need to read it
in order to interpret the decoded data as well.

The MediaFormat object when using OMX.google.h264.decoder contains data
like this:

{height=176, what=1869968451, color-format=19, slice-height=176,
crop-left=0, width=320, crop-bottom=169, crop-top=0, mime=video/raw,
stride=320, crop-right=319}

The attributes slice-height, stride, crop-* are not documented in
MediaFormat at all, but since the data seems to be tightly packed, one
could ignore them altogether and get the right result.

However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a pandaboard, the
MediaFormat for the decoded data from same video file contains this:

{height=272, what=1869968451, color-format=2130706688, slice-height=272,
crop-left=32, width=384, crop-bottom=193, crop-top=24, mime=video/raw,
stride=384, crop-right=351}

This is the pixel format 2130706688 aka 0x7f000100,
COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one absolutely need
to use all the extra attributes in order to interpret the data properly.
But can one count on them existing since they are not documented?


With all this in mind, it seems to me that this API is really hard to use
over all Android 4.1 devices without actually testing on almost all of
them.

I hope that the CTS will force the vendors to conforming to some sane
behaviour at least, to make it even remotely possible for third party
application developers to use it across all devices. (If a device can't
provide what is needed, that's also totally acceptable, as long as it is
signaled in a clear way.)



There's also a few minor issues with the API that aren't as critical for
using it, but that still feel slightly awkward:

If I call MediaCodec.createDecoderByType or
MediaCodec.createEncoderByType, the process will crash in native code if
there is no such matching codec. Thus one cannot use these functions at
all, one has to use the MediaCodecList class to iterate all the existing
codecs in order to safely call MediaCodec.createByCodecName instead. I
have submitted a fix for this crash in
https://android-review.googlesource.com/39080, but since it isn't part of
the first 4.1 devices released, one can't trust it unless one only targets
devices above the first 4.1 release. (And since one needs to write the few
lines of code for iterating the MediaCodecList, there's no point in using
the shortcut convenience functions at all.)

If I call MediaCodec.getOutputFormat immediately after calling
MediaCodec.start, as the example at
http://developer.android.com/reference/android/media/MediaCodec.html
shows, the process once again crashes in native code. This isn't too much
of an issue, "just don't do that then". I have provided a fix for the
crash in https://android-review.googlesource.com/39090 and a fix for
updating the example in https://android-review.googlesource.com/39100 at
least.


If I were able to call MediaCodec.createEncoderByType safely, which I
can't right now, I would still need to know the pixel formats that this
MediaCodec instance supports. Since the MediaCodec neither allows me to
get the name of the actual codec or the corresponding MediaCodecInfo, I
can't get a hold of this information, and thus there's little point in
using the convenience function createEncoderByType, since one still needs
to iterate the MediaCodecList.

It would also be useful to know whether a codec is a HW or SW codec (other
than checking for the OMX.google prefix), since an app might be shipping
SW fallback codecs of its own, but would prefer to use the MediaCodec API
only in the case where it actually is hardware accelerated.


So, a method in MediaCodec for getting the codec name and the
corresponding MediaCodecInfo, and a method in MediaCodecInfo for knowing
whether it is a HW codec would be useful.

// Martin

[James Dong]

Yes, but you may know there is some performance penalty associated with this particular chosen color format on some devices.

Recommending vendors to support a specific color format is doable, but that may have same impact.

We should document it.

Thanks for sharing this. We have looked at your suggested patch.

 

// Martin

[Martin Storsjö]

Hi James,

Thanks for your reply - some further questions below.

On Tue, 17 Jul 2012, James Dong wrote:

Yes, recommending a single one might not be the best way forward, but I
hope you agree and understand that interpreting something like a
manufacturer specific tiled format (which might be more or less
undocumented) might not be an option for a third party application vendor.
If a vendor chooses to only expose such pixel formats through this new
API, it's essentially useless to anything else than just direct rendering
to a Surface.

This actually is what most decoders on the qcom MSM 8xxx chipsets do, as
far as I've seen - the decoder returns data in an almost incomprehensible
tiled format. Unless somebody actually does something to change it, this
will be the pixel format that will be provided if decoding via the new
MediaCodec API as well. And very few callers will be able to use that
data.

So what I'm hoping for is not one single pixel format supported by
everybody, but some sane (documented!) set of pixel formats that most
vendors would be able to provide without too much performance loss.

I would be ok with dealing with vendor specific 420 semiplanar (such as
TI, having only some extra additions for adjusting the slice height, iirc)
as long as it isn't one of the totally proprietary tiled ones.

The problem just lies in how to build something that will work on a
reasonable number of devices. If each vendor has their own private pixel
format variant, one can't support them (except with clean SW fallbacks)
unless one actually have access to testing on them.

Good, thanks.

It would also be good to know if one has to set any of these while
encoding, to describe the incoming data buffers. Or is it ok to just pass
tightly packed buffers without padding to them? My fear here is that while
I have something working on one device with one pixel format, another
device might fail when running the same code, since it might expect a
certain amount of padding between the planes or a certain stride between
lines, etc.

This is even harder to handle in practice, since one doesn't know if it
will work or if one should fall back to a bundled SW codec in advance. If
an encoder has hidden assumptions about padding/stride, it will
(code-wise) seem to work but just output garbage data, even if the pixel
format is one that is known and tested from before.

Yes, thanks for the quick review of the patches! (Now only waiting for
someone to verify them so they can be merged.)

// Martin

[Vineeth Upadhya]

Hi ,

     I am looking to test this new api ..Could you guys share a elementary src code . I have an x86 android and can share my learnings as well. What I am trying  to achieve is here  http://dpsm.wordpress.com/2012/07/28/android-mediacodec-decoded/#comment-24 

[Vineeth Upadhya]

Hi martin ,

[Martin Storsjö]

Hi,

After getting access to a few actual commercial devices running JellyBean,
I now can concretize my questions/concerns.

On Wed, 18 Jul 2012, Martin Storsj� wrote:

> It would also be good to know if one has to set any of these while encoding,
> to describe the incoming data buffers. Or is it ok to just pass tightly
> packed buffers without padding to them? My fear here is that while I have
> something working on one device with one pixel format, another device might
> fail when running the same code, since it might expect a certain amount of
> padding between the planes or a certain stride between lines, etc.
>
> This is even harder to handle in practice, since one doesn't know if it will
> work or if one should fall back to a bundled SW codec in advance. If an
> encoder has hidden assumptions about padding/stride, it will (code-wise) seem
> to work but just output garbage data, even if the pixel format is one that is
> known and tested from before.

This is exactly the issue I'm running into on Nexus 7 and Nexus S.

The Nexus S supports encoding in both COLOR_FormatYUV420Planar and
COLOR_FormatYUV420SemiPlanar, while the Nexus 7 only supports encoding in
COLOR_FormatYUV420Planar.

(As a side note, both of them also claim supporting encoding the pixel
format 2130708361, 0x7f000789, that is OMX_COLOR_FormatAndroidOpaque - I
assume this isn't intended to be used via the MediaCodec API at all -
would it make sense to filter it out at this API level as well so it isn't
exposed to third party java apps?)

When encoding using the COLOR_FormatYUV420Planar pixel format that both
devices support, the same code work fine on both devices, as long as I
encode dimensions that are multiples of 16. If I encode video having a
dimension not a multiple of 16 (but still a multiple of 8), things behave
differently.

The Nexus S wants to have data tightly packed (stride equal to width, and
chroma stride equal to width/2), while Nexus 7 requires the stride to be
aligned to 16. When reading the source code of ACodec::setupVideoEncoder,
I realize that I could try setting the undocumented parameter "stride" in
the MediaFormat object, but that does not help either. Both devices ignore
the value I set there.

Thus, if I want to encode video on these two devices, my calling code
needs to know the device specific quirk whether the data should be padded
to a certain alignment or not. And if I want to run this code on devices
other than these two (that also claim to support
COLOR_FormatYUV420Planar), how will I know which behaviour to expect?

Also, it's hard to write a bug report about this, since there is no
documentation saying what the expected behaviour is, so I can't say which
one of them is wrong, if any.

Currently this means that apps using MediaCodec to encode video needs to
either be very careful with odd resolutions, or test the apps on every
single device. And if not even two google devices can do this
consistently, what can one expect from the devices from other
manufacturers?

Furthermore, this leads to the following question - what are the "limits"
of what one can expect from devices I haven't yet had a chance to test?
Dimensions not a multiple of 16 is apparently an issue. Are there other
issues I can expect to run into on other devices with this pixel format,
that still are ok according to the CTS and similar tests? (Do those tests
even require anything at all from this API?)

// Martin