Hi,
I've tried taking the new, long desired, MediaCodec API into use in a few
different use cases, and while it looks promising, there's still a few
issues that feels like they need documenting before the new API can be
relied upon properly.
First off - pixel formats. The scenario is that I'm using the MediaCodec
API to decode video, and I want to render the decoded YUV data myself, I'm
not passing any Surface to the system for rendering. Or I'm encoding raw
video data, captured from the camera via the preview image callback.
Since I want to use the decoded YUV data myself, I need to be able to
handle the pixel formats that the decoder produces. The
MediaCodecInfo.CodecCapabilities class
(
http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html)
lists a vast array of pixel formats (about 40 or so), with only a
reference to the OpenMAX IL spec. Is there any guarantee about which pixel
formats the decoders on Android 4.1 compliant devices will provide? Or do
I have to be able to handle all those 40 different pixel formats, if some
unknown device with Android 4.1 that I don't have access to would choose
to use some of them? This is of course a bit exaggerated, I know, in
practice I probably would only have to support the relatively few YUV 420
formats, perhaps in addition to some of the YUV 422 versions. That's only
about 10 different formats.
But this list also contains the vendor specific
COLOR_QCOM_FormatYUV420SemiPlanar and
COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find information about
how these formats differ from the normal COLOR_FormatYUV420SemiPlanar and
COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL spec
doesn't know about these vendor extensions, obviously.) Are these the only
vendor specific formats that vendors are allowed to use, or is it ok if
qcom devices return e.g. the
QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (0x7FA30C03) pixel
format? Many current qcom hw decoders return data in this format, and it
is very nontrivial to use.
Will the CTS require device manufacturers to return data in some of the
"generally known/supported" pixel formats? If not, in order to use the raw
decoded data from the MediaCodec API, one would need to test things on one
device per chipset family/generation, more or less.
For video capture with the low level Camera class, this kind of issue is
already documented, see e.g.
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12.
That section says "This format is guaranteed to be supported for camera
preview images since API level 12; for earlier API versions, check
getSupportedPreviewFormats()."
For encoding, this at least is manageable where the codec can support more
than one format, one can request which one to use. For decoding, the API
might indicate that the codec supports a few different pixel formats, but
one does not know which one it actually will use until the decoder is
started.
I haven't had a chance to test this API on the newly released Nexus
devices, but I have played a little with it on the emulator and on a
Pandaboard. On the latter, the video decoder indicates that it supports
two pixel formats, 0x7f000100 which is
OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which I don't
even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.
Once we get past the issue of pixel formats, then we get to the issue of
how to use each pixel format. The MediaFormat class documents that the
KEY_COLOR_FORMAT key is encoder only, while one obviously need to read it
in order to interpret the decoded data as well.
The MediaFormat object when using OMX.google.h264.decoder contains data
like this:
{height=176, what=1869968451, color-format=19, slice-height=176,
crop-left=0, width=320, crop-bottom=169, crop-top=0, mime=video/raw,
stride=320, crop-right=319}
The attributes slice-height, stride, crop-* are not documented in
MediaFormat at all, but since the data seems to be tightly packed, one
could ignore them altogether and get the right result.
However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a pandaboard, the
MediaFormat for the decoded data from same video file contains this:
{height=272, what=1869968451, color-format=2130706688, slice-height=272,
crop-left=32, width=384, crop-bottom=193, crop-top=24, mime=video/raw,
stride=384, crop-right=351}
This is the pixel format 2130706688 aka 0x7f000100,
COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one absolutely need
to use all the extra attributes in order to interpret the data properly.
But can one count on them existing since they are not documented?
With all this in mind, it seems to me that this API is really hard to use
over all Android 4.1 devices without actually testing on almost all of
them.
I hope that the CTS will force the vendors to conforming to some sane
behaviour at least, to make it even remotely possible for third party
application developers to use it across all devices. (If a device can't
provide what is needed, that's also totally acceptable, as long as it is
signaled in a clear way.)
There's also a few minor issues with the API that aren't as critical for
using it, but that still feel slightly awkward:
If I call MediaCodec.createDecoderByType or
MediaCodec.createEncoderByType, the process will crash in native code if
there is no such matching codec. Thus one cannot use these functions at
all, one has to use the MediaCodecList class to iterate all the existing
codecs in order to safely call MediaCodec.createByCodecName instead. I
have submitted a fix for this crash in
https://android-review.googlesource.com/39080, but since it isn't part of
the first 4.1 devices released, one can't trust it unless one only targets
devices above the first 4.1 release. (And since one needs to write the few
lines of code for iterating the MediaCodecList, there's no point in using
the shortcut convenience functions at all.)
If I call MediaCodec.getOutputFormat immediately after calling
MediaCodec.start, as the example at
http://developer.android.com/reference/android/media/MediaCodec.html
shows, the process once again crashes in native code. This isn't too much
of an issue, "just don't do that then". I have provided a fix for the
crash in
https://android-review.googlesource.com/39090 and a fix for
updating the example in
https://android-review.googlesource.com/39100 at
least.
If I were able to call MediaCodec.createEncoderByType safely, which I
can't right now, I would still need to know the pixel formats that this
MediaCodec instance supports. Since the MediaCodec neither allows me to
get the name of the actual codec or the corresponding MediaCodecInfo, I
can't get a hold of this information, and thus there's little point in
using the convenience function createEncoderByType, since one still needs
to iterate the MediaCodecList.
It would also be useful to know whether a codec is a HW or SW codec (other
than checking for the OMX.google prefix), since an app might be shipping
SW fallback codecs of its own, but would prefer to use the MediaCodec API
only in the case where it actually is hardware accelerated.
So, a method in MediaCodec for getting the codec name and the
corresponding MediaCodecInfo, and a method in MediaCodecInfo for knowing
whether it is a HW codec would be useful.
// Martin