I've tried taking the new, long desired, MediaCodec API into use in a few different use cases, and while it looks promising, there's still a few issues that feels like they need documenting before the new API can be relied upon properly.
First off - pixel formats. The scenario is that I'm using the MediaCodec API to decode video, and I want to render the decoded YUV data myself, I'm not passing any Surface to the system for rendering. Or I'm encoding raw video data, captured from the camera via the preview image callback.
Since I want to use the decoded YUV data myself, I need to be able to handle the pixel formats that the decoder produces. The MediaCodecInfo.CodecCapabilities class (http://developer.android.com/reference/android/media/MediaCodecInfo.C...) lists a vast array of pixel formats (about 40 or so), with only a reference to the OpenMAX IL spec. Is there any guarantee about which pixel formats the decoders on Android 4.1 compliant devices will provide? Or do I have to be able to handle all those 40 different pixel formats, if some unknown device with Android 4.1 that I don't have access to would choose to use some of them? This is of course a bit exaggerated, I know, in practice I probably would only have to support the relatively few YUV 420 formats, perhaps in addition to some of the YUV 422 versions. That's only about 10 different formats.
But this list also contains the vendor specific COLOR_QCOM_FormatYUV420SemiPlanar and COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find information about how these formats differ from the normal COLOR_FormatYUV420SemiPlanar and COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL spec doesn't know about these vendor extensions, obviously.) Are these the only vendor specific formats that vendors are allowed to use, or is it ok if qcom devices return e.g. the QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (0x7FA30C03) pixel format? Many current qcom hw decoders return data in this format, and it is very nontrivial to use.
Will the CTS require device manufacturers to return data in some of the "generally known/supported" pixel formats? If not, in order to use the raw decoded data from the MediaCodec API, one would need to test things on one device per chipset family/generation, more or less.
For video capture with the low level Camera class, this kind of issue is already documented, see e.g. http://developer.android.com/reference/android/graphics/ImageFormat.h.... That section says "This format is guaranteed to be supported for camera preview images since API level 12; for earlier API versions, check getSupportedPreviewFormats()."
For encoding, this at least is manageable where the codec can support more than one format, one can request which one to use. For decoding, the API might indicate that the codec supports a few different pixel formats, but one does not know which one it actually will use until the decoder is started.
I haven't had a chance to test this API on the newly released Nexus devices, but I have played a little with it on the emulator and on a Pandaboard. On the latter, the video decoder indicates that it supports two pixel formats, 0x7f000100 which is OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which I don't even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.
Once we get past the issue of pixel formats, then we get to the issue of how to use each pixel format. The MediaFormat class documents that the KEY_COLOR_FORMAT key is encoder only, while one obviously need to read it in order to interpret the decoded data as well.
The MediaFormat object when using OMX.google.h264.decoder contains data like this:
The attributes slice-height, stride, crop-* are not documented in MediaFormat at all, but since the data seems to be tightly packed, one could ignore them altogether and get the right result.
However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a pandaboard, the MediaFormat for the decoded data from same video file contains this:
This is the pixel format 2130706688 aka 0x7f000100, COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one absolutely need to use all the extra attributes in order to interpret the data properly. But can one count on them existing since they are not documented?
With all this in mind, it seems to me that this API is really hard to use over all Android 4.1 devices without actually testing on almost all of them.
I hope that the CTS will force the vendors to conforming to some sane behaviour at least, to make it even remotely possible for third party application developers to use it across all devices. (If a device can't provide what is needed, that's also totally acceptable, as long as it is signaled in a clear way.)
There's also a few minor issues with the API that aren't as critical for using it, but that still feel slightly awkward:
If I call MediaCodec.createDecoderByType or MediaCodec.createEncoderByType, the process will crash in native code if there is no such matching codec. Thus one cannot use these functions at all, one has to use the MediaCodecList class to iterate all the existing codecs in order to safely call MediaCodec.createByCodecName instead. I have submitted a fix for this crash in https://android-review.googlesource.com/39080, but since it isn't part of the first 4.1 devices released, one can't trust it unless one only targets devices above the first 4.1 release. (And since one needs to write the few lines of code for iterating the MediaCodecList, there's no point in using the shortcut convenience functions at all.)
If I were able to call MediaCodec.createEncoderByType safely, which I can't right now, I would still need to know the pixel formats that this MediaCodec instance supports. Since the MediaCodec neither allows me to get the name of the actual codec or the corresponding MediaCodecInfo, I can't get a hold of this information, and thus there's little point in using the convenience function createEncoderByType, since one still needs to iterate the MediaCodecList.
It would also be useful to know whether a codec is a HW or SW codec (other than checking for the OMX.google prefix), since an app might be shipping SW fallback codecs of its own, but would prefer to use the MediaCodec API only in the case where it actually is hardware accelerated.
So, a method in MediaCodec for getting the codec name and the corresponding MediaCodecInfo, and a method in MediaCodecInfo for knowing whether it is a HW codec would be useful.
On Friday, July 13, 2012 6:00:35 AM UTC-7, Martin Storsjö wrote:
> Hi,
> I've tried taking the new, long desired, MediaCodec API into use in a few > different use cases, and while it looks promising, there's still a few > issues that feels like they need documenting before the new API can be > relied upon properly.
> First off - pixel formats. The scenario is that I'm using the MediaCodec > API to decode video, and I want to render the decoded YUV data myself, I'm > not passing any Surface to the system for rendering. Or I'm encoding raw > video data, captured from the camera via the preview image callback.
> lists a vast array of pixel formats (about 40 or so), with only a > reference to the OpenMAX IL spec. Is there any guarantee about which pixel > formats the decoders on Android 4.1 compliant devices will provide? Or do > I have to be able to handle all those 40 different pixel formats, if some > unknown device with Android 4.1 that I don't have access to would choose > to use some of them? This is of course a bit exaggerated, I know, in > practice I probably would only have to support the relatively few YUV 420 > formats, perhaps in addition to some of the YUV 422 versions. That's only > about 10 different formats.
> But this list also contains the vendor specific > COLOR_QCOM_FormatYUV420SemiPlanar and > COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find information about > how these formats differ from the normal COLOR_FormatYUV420SemiPlanar and > COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL spec > doesn't know about these vendor extensions, obviously.) Are these the only > vendor specific formats that vendors are allowed to use, or is it ok if > qcom devices return e.g. the > QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (0x7FA30C03) pixel > format? Many current qcom hw decoders return data in this format, and it > is very nontrivial to use.
> Will the CTS require device manufacturers to return data in some of the > "generally known/supported" pixel formats? If not, in order to use the raw > decoded data from the MediaCodec API, one would need to test things on one > device per chipset family/generation, more or less.
> For video capture with the low level Camera class, this kind of issue is > already documented, see e.g.
> That section says "This format is guaranteed to be supported for camera > preview images since API level 12; for earlier API versions, check > getSupportedPreviewFormats()."
Yes, but you may know there is some performance penalty associated with this particular chosen color format on some devices.
Recommending vendors to support a specific color format is doable, but that may have same impact.
> For encoding, this at least is manageable where the codec can support more > than one format, one can request which one to use. For decoding, the API > might indicate that the codec supports a few different pixel formats, but > one does not know which one it actually will use until the decoder is > started.
> I haven't had a chance to test this API on the newly released Nexus > devices, but I have played a little with it on the emulator and on a > Pandaboard. On the latter, the video decoder indicates that it supports > two pixel formats, 0x7f000100 which is > OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which I don't > even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.
> Once we get past the issue of pixel formats, then we get to the issue of > how to use each pixel format. The MediaFormat class documents that the > KEY_COLOR_FORMAT key is encoder only, while one obviously need to read it > in order to interpret the decoded data as well.
> The MediaFormat object when using OMX.google.h264.decoder contains data > like this:
> The attributes slice-height, stride, crop-* are not documented in > MediaFormat at all, but since the data seems to be tightly packed, one > could ignore them altogether and get the right result.
> However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a pandaboard, the > MediaFormat for the decoded data from same video file contains this:
> This is the pixel format 2130706688 aka 0x7f000100, > COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one absolutely need > to use all the extra attributes in order to interpret the data properly. > But can one count on them existing since they are not documented?
> With all this in mind, it seems to me that this API is really hard to use > over all Android 4.1 devices without actually testing on almost all of > them.
> I hope that the CTS will force the vendors to conforming to some sane > behaviour at least, to make it even remotely possible for third party > application developers to use it across all devices. (If a device can't > provide what is needed, that's also totally acceptable, as long as it is > signaled in a clear way.)
> There's also a few minor issues with the API that aren't as critical for > using it, but that still feel slightly awkward:
> If I call MediaCodec.createDecoderByType or > MediaCodec.createEncoderByType, the process will crash in native code if > there is no such matching codec. Thus one cannot use these functions at > all, one has to use the MediaCodecList class to iterate all the existing > codecs in order to safely call MediaCodec.createByCodecName instead. I > have submitted a fix for this crash in > https://android-review.googlesource.com/39080, but since it isn't part of > the first 4.1 devices released, one can't trust it unless one only targets > devices above the first 4.1 release. (And since one needs to write the few > lines of code for iterating the MediaCodecList, there's no point in using > the shortcut convenience functions at all.)
> If I were able to call MediaCodec.createEncoderByType safely, which I > can't right now, I would still need to know the pixel formats that this > MediaCodec instance supports. Since the MediaCodec neither allows me to > get the name of the actual codec or the corresponding MediaCodecInfo, I > can't get a hold of this information, and thus there's little point in > using the convenience function createEncoderByType, since one still needs > to iterate the MediaCodecList.
> It would also be useful to know whether a codec is a HW or SW codec (other > than checking for the OMX.google prefix), since an app might be shipping > SW fallback codecs of its own, but would prefer to use the MediaCodec API > only in the case where it actually is hardware accelerated.
> So, a method in MediaCodec for getting the codec name and the > corresponding MediaCodecInfo, and a method in MediaCodecInfo for knowing > whether it is a HW codec would be useful.
Thanks for sharing this. We have looked at your suggested patch.
> On Friday, July 13, 2012 6:00:35 AM UTC-7, Martin Storsj� wrote:
> First off - pixel formats. The scenario is that I'm using the
> MediaCodec
> API to decode video, and I want to render the decoded YUV data
> myself, I'm
> not passing any Surface to the system for rendering. Or I'm
> encoding raw
> video data, captured from the camera via the preview image
> callback.
> Since I want to use the decoded YUV data myself, I need to be
> able to
> handle the pixel formats that the decoder produces. The
> MediaCodecInfo.CodecCapabilities class
> (http://developer.android.com/reference/android/media/MediaCodecInfo.C... > apabilities.html)
> lists a vast array of pixel formats (about 40 or so), with only
> a
> reference to the OpenMAX IL spec. Is there any guarantee about
> which pixel
> formats the decoders on Android 4.1 compliant devices will
> provide? Or do
> I have to be able to handle all those 40 different pixel
> formats, if some
> unknown device with Android 4.1 that I don't have access to
> would choose
> to use some of them? This is of course a bit exaggerated, I
> know, in
> practice I probably would only have to support the relatively
> few YUV 420
> formats, perhaps in addition to some of the YUV 422 versions.
> That's only
> about 10 different formats.
> But this list also contains the vendor specific
> COLOR_QCOM_FormatYUV420SemiPlanar and
> COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find
> information about
> how these formats differ from the normal
> COLOR_FormatYUV420SemiPlanar and
> COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL
> spec
> doesn't know about these vendor extensions, obviously.) Are
> these the only
> vendor specific formats that vendors are allowed to use, or is
> it ok if
> qcom devices return e.g. the
> QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka
> (0x7FA30C03) pixel
> format? Many current qcom hw decoders return data in this
> format, and it
> is very nontrivial to use.
> Will the CTS require device manufacturers to return data in some
> of the
> "generally known/supported" pixel formats? If not, in order to
> use the raw
> decoded data from the MediaCodec API, one would need to test
> things on one
> device per chipset family/generation, more or less.
> For video capture with the low level Camera class, this kind of
> issue is
> already documented, see e.g.
> http://developer.android.com/reference/android/graphics/ImageFormat.h... > 12.
> That section says "This format is guaranteed to be supported for
> camera
> preview images since API level 12; for earlier API versions,
> check
> getSupportedPreviewFormats()."
> Yes, but you may know there is some performance penalty associated with this
> particular chosen color format on some devices.
> Recommending vendors to support a specific color format is doable, but that
> may have same impact.
Yes, recommending a single one might not be the best way forward, but I hope you agree and understand that interpreting something like a manufacturer specific tiled format (which might be more or less undocumented) might not be an option for a third party application vendor. If a vendor chooses to only expose such pixel formats through this new API, it's essentially useless to anything else than just direct rendering to a Surface.
This actually is what most decoders on the qcom MSM 8xxx chipsets do, as far as I've seen - the decoder returns data in an almost incomprehensible tiled format. Unless somebody actually does something to change it, this will be the pixel format that will be provided if decoding via the new MediaCodec API as well. And very few callers will be able to use that data.
So what I'm hoping for is not one single pixel format supported by everybody, but some sane (documented!) set of pixel formats that most vendors would be able to provide without too much performance loss.
I would be ok with dealing with vendor specific 420 semiplanar (such as TI, having only some extra additions for adjusting the slice height, iirc) as long as it isn't one of the totally proprietary tiled ones.
The problem just lies in how to build something that will work on a reasonable number of devices. If each vendor has their own private pixel format variant, one can't support them (except with clean SW fallbacks) unless one actually have access to testing on them.
> For encoding, this at least is manageable where the codec can
> support more
> than one format, one can request which one to use. For decoding,
> the API
> might indicate that the codec supports a few different pixel
> formats, but
> one does not know which one it actually will use until the
> decoder is
> started.
> I haven't had a chance to test this API on the newly released
> Nexus
> devices, but I have played a little with it on the emulator and
> on a
> Pandaboard. On the latter, the video decoder indicates that it
> supports
> two pixel formats, 0x7f000100 which is
> OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which
> I don't
> even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.
> Once we get past the issue of pixel formats, then we get to the
> issue of
> how to use each pixel format. The MediaFormat class documents
> that the
> KEY_COLOR_FORMAT key is encoder only, while one obviously need
> to read it
> in order to interpret the decoded data as well.
> The MediaFormat object when using OMX.google.h264.decoder
> contains data
> like this:
> The attributes slice-height, stride, crop-* are not documented
> in
> MediaFormat at all, but since the data seems to be tightly
> packed, one
> could ignore them altogether and get the right result.
> However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a
> pandaboard, the
> MediaFormat for the decoded data from same video file contains
> this:
> This is the pixel format 2130706688 aka 0x7f000100,
> COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one
> absolutely need
> to use all the extra attributes in order to interpret the data
> properly.
> But can one count on them existing since they are not
> documented?
> We should document it.
Good, thanks.
It would also be good to know if one has to set any of these while encoding, to describe the incoming data buffers. Or is it ok to just pass tightly packed buffers without padding to them? My fear here is that while I have something working on one device with one pixel format, another device might fail when running the same code, since it might expect a certain amount of padding between the planes or a certain stride between lines, etc.
This is even harder to handle in practice, since one doesn't know if it will work or if one should fall back to a bundled SW codec in advance. If an encoder has hidden assumptions about padding/stride, it will (code-wise) seem to work but just output garbage data, even if the pixel format is one that is known and tested from before.
> With all this in mind, it seems to me that this API is really
> hard to use
> over all Android 4.1 devices without actually testing on almost
> all of
> them.
> I hope that the CTS will force the vendors to conforming to some
> sane
> behaviour at least, to make it even remotely possible for third
> party
> application developers to use it across all devices. (If a
> device can't
> provide what is needed, that's also totally acceptable, as long
> as it is
> signaled in a clear way.) �
> There's also a few minor issues with the API that aren't as
> critical for
> using it, but that still feel slightly awkward:
> If I call MediaCodec.createDecoderByType or
> MediaCodec.createEncoderByType, the process will crash in native
> code if
> there is no such matching codec. Thus one cannot use these
> functions at
> all, one has to use the MediaCodecList class to iterate all the
> existing
> codecs in order to safely call MediaCodec.createByCodecName
> instead. I
> have submitted a fix for this crash in
> https://android-review.googlesource.com/39080, but since it
> isn't part of
> the first 4.1 devices released, one can't trust it unless one
> only targets
> devices above the first 4.1 release. (And since one needs to
> write the few
> lines of code for iterating the MediaCodecList, there's no point
> in using
> the shortcut convenience functions at all.)
After getting access to a few actual commercial devices running JellyBean, I now can concretize my questions/concerns.
On Wed, 18 Jul 2012, Martin Storsj� wrote:
> It would also be good to know if one has to set any of these while encoding, > to describe the incoming data buffers. Or is it ok to just pass tightly > packed buffers without padding to them? My fear here is that while I have > something working on one device with one pixel format, another device might > fail when running the same code, since it might expect a certain amount of > padding between the planes or a certain stride between lines, etc.
> This is even harder to handle in practice, since one doesn't know if it will > work or if one should fall back to a bundled SW codec in advance. If an > encoder has hidden assumptions about padding/stride, it will (code-wise) seem > to work but just output garbage data, even if the pixel format is one that is > known and tested from before.
This is exactly the issue I'm running into on Nexus 7 and Nexus S.
The Nexus S supports encoding in both COLOR_FormatYUV420Planar and COLOR_FormatYUV420SemiPlanar, while the Nexus 7 only supports encoding in COLOR_FormatYUV420Planar.
(As a side note, both of them also claim supporting encoding the pixel format 2130708361, 0x7f000789, that is OMX_COLOR_FormatAndroidOpaque - I assume this isn't intended to be used via the MediaCodec API at all - would it make sense to filter it out at this API level as well so it isn't exposed to third party java apps?)
When encoding using the COLOR_FormatYUV420Planar pixel format that both devices support, the same code work fine on both devices, as long as I encode dimensions that are multiples of 16. If I encode video having a dimension not a multiple of 16 (but still a multiple of 8), things behave differently.
The Nexus S wants to have data tightly packed (stride equal to width, and chroma stride equal to width/2), while Nexus 7 requires the stride to be aligned to 16. When reading the source code of ACodec::setupVideoEncoder, I realize that I could try setting the undocumented parameter "stride" in the MediaFormat object, but that does not help either. Both devices ignore the value I set there.
Thus, if I want to encode video on these two devices, my calling code needs to know the device specific quirk whether the data should be padded to a certain alignment or not. And if I want to run this code on devices other than these two (that also claim to support COLOR_FormatYUV420Planar), how will I know which behaviour to expect?
Also, it's hard to write a bug report about this, since there is no documentation saying what the expected behaviour is, so I can't say which one of them is wrong, if any.
Currently this means that apps using MediaCodec to encode video needs to either be very careful with odd resolutions, or test the apps on every single device. And if not even two google devices can do this consistently, what can one expect from the devices from other manufacturers?
Furthermore, this leads to the following question - what are the "limits" of what one can expect from devices I haven't yet had a chance to test? Dimensions not a multiple of 16 is apparently an issue. Are there other issues I can expect to run into on other devices with this pixel format, that still are ok according to the CTS and similar tests? (Do those tests even require anything at all from this API?)
Dear Martin, I am able to Decode a video mp4 file using mediacodec api and dump that decoded file as decoderoutput.bin. I am using the software OMX Encoder and OMX Decoder (OMX.google.h264.encoder) on Emulator. Now I want to encode the decoded output i,e decoderouput.bin again using mediacodec Encoder api. can I give on fly the decoded bytebuffers(which will be copied to other bytebuffer and used as input bytebuffer to encoder) as input to the encoder? If I try giving like that what should be my encoder configuring parameters such as bitrate, framerate etc? I tried configuring with 30fps,512kbps and 40 I-frameinterval,color as yuv420planar(19)and with above on fly copying the decoder outputbytebuffer to another bytebuffer which will be input to encoder. Now I am getting illegal state exception with OMX Error 0x80001001, OMX_ErrorUndefined after configure/starting the encoder. How can I solve this error?
Another approach is giving the input file as decoderoutput.bin to Encoder. How will I configure inputformat to the Encoder. I have seen your example code DecoderTest.java in that you are creating a input frame. but here I want to give Decoderoutput.bin as input.
On Friday, July 13, 2012 9:00:35 PM UTC+8, Martin Storsjö wrote:
> Hi,
> I've tried taking the new, long desired, MediaCodec API into use in a few > different use cases, and while it looks promising, there's still a few > issues that feels like they need documenting before the new API can be > relied upon properly.
> First off - pixel formats. The scenario is that I'm using the MediaCodec > API to decode video, and I want to render the decoded YUV data myself, I'm > not passing any Surface to the system for rendering. Or I'm encoding raw > video data, captured from the camera via the preview image callback.
> lists a vast array of pixel formats (about 40 or so), with only a > reference to the OpenMAX IL spec. Is there any guarantee about which pixel > formats the decoders on Android 4.1 compliant devices will provide? Or do > I have to be able to handle all those 40 different pixel formats, if some > unknown device with Android 4.1 that I don't have access to would choose > to use some of them? This is of course a bit exaggerated, I know, in > practice I probably would only have to support the relatively few YUV 420 > formats, perhaps in addition to some of the YUV 422 versions. That's only > about 10 different formats.
> But this list also contains the vendor specific > COLOR_QCOM_FormatYUV420SemiPlanar and > COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find information about > how these formats differ from the normal COLOR_FormatYUV420SemiPlanar and > COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL spec > doesn't know about these vendor extensions, obviously.) Are these the only > vendor specific formats that vendors are allowed to use, or is it ok if > qcom devices return e.g. the > QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (0x7FA30C03) pixel > format? Many current qcom hw decoders return data in this format, and it > is very nontrivial to use.
> Will the CTS require device manufacturers to return data in some of the > "generally known/supported" pixel formats? If not, in order to use the raw > decoded data from the MediaCodec API, one would need to test things on one > device per chipset family/generation, more or less.
> For video capture with the low level Camera class, this kind of issue is > already documented, see e.g.
> That section says "This format is guaranteed to be supported for camera > preview images since API level 12; for earlier API versions, check > getSupportedPreviewFormats()."
> For encoding, this at least is manageable where the codec can support more > than one format, one can request which one to use. For decoding, the API > might indicate that the codec supports a few different pixel formats, but > one does not know which one it actually will use until the decoder is > started.
> I haven't had a chance to test this API on the newly released Nexus > devices, but I have played a little with it on the emulator and on a > Pandaboard. On the latter, the video decoder indicates that it supports > two pixel formats, 0x7f000100 which is > OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which I don't > even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.
> Once we get past the issue of pixel formats, then we get to the issue of > how to use each pixel format. The MediaFormat class documents that the > KEY_COLOR_FORMAT key is encoder only, while one obviously need to read it > in order to interpret the decoded data as well.
> The MediaFormat object when using OMX.google.h264.decoder contains data > like this:
> The attributes slice-height, stride, crop-* are not documented in > MediaFormat at all, but since the data seems to be tightly packed, one > could ignore them altogether and get the right result.
> However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a pandaboard, the > MediaFormat for the decoded data from same video file contains this:
> This is the pixel format 2130706688 aka 0x7f000100, > COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one absolutely need > to use all the extra attributes in order to interpret the data properly. > But can one count on them existing since they are not documented?
> With all this in mind, it seems to me that this API is really hard to use > over all Android 4.1 devices without actually testing on almost all of > them.
> I hope that the CTS will force the vendors to conforming to some sane > behaviour at least, to make it even remotely possible for third party > application developers to use it across all devices. (If a device can't > provide what is needed, that's also totally acceptable, as long as it is > signaled in a clear way.)
> There's also a few minor issues with the API that aren't as critical for > using it, but that still feel slightly awkward:
> If I call MediaCodec.createDecoderByType or > MediaCodec.createEncoderByType, the process will crash in native code if > there is no such matching codec. Thus one cannot use these functions at > all, one has to use the MediaCodecList class to iterate all the existing > codecs in order to safely call MediaCodec.createByCodecName instead. I > have submitted a fix for this crash in > https://android-review.googlesource.com/39080, but since it isn't part of > the first 4.1 devices released, one can't trust it unless one only targets > devices above the first 4.1 release. (And since one needs to write the few > lines of code for iterating the MediaCodecList, there's no point in using > the shortcut convenience functions at all.)
> If I were able to call MediaCodec.createEncoderByType safely, which I > can't right now, I would still need to know the pixel formats that this > MediaCodec instance supports. Since the MediaCodec neither allows me to > get the name of the actual codec or the corresponding MediaCodecInfo, I > can't get a hold of this information, and thus there's little point in > using the convenience function createEncoderByType, since one still needs > to iterate the MediaCodecList.
> It would also be useful to know whether a codec is a HW or SW codec (other > than checking for the OMX.google prefix), since an app might be shipping > SW fallback codecs of its own, but would prefer to use the MediaCodec API > only in the case where it actually is hardware accelerated.
> So, a method in MediaCodec for getting the codec name and the > corresponding MediaCodecInfo, and a method in MediaCodecInfo for knowing > whether it is a HW codec would be useful.
> // Martin
Hi Martin,
Is there any document saying how to use the code which download from https://android-review.googlesource.com/#/c/43410/, I really can not find the exact path to test the code, please kindly give some hint. Thank you.
On Friday, July 13, 2012 9:00:35 PM UTC+8, Martin Storsjö wrote:
> Hi,
> I've tried taking the new, long desired, MediaCodec API into use in a few > different use cases, and while it looks promising, there's still a few > issues that feels like they need documenting before the new API can be > relied upon properly.
> First off - pixel formats. The scenario is that I'm using the MediaCodec > API to decode video, and I want to render the decoded YUV data myself, I'm > not passing any Surface to the system for rendering. Or I'm encoding raw > video data, captured from the camera via the preview image callback.
> lists a vast array of pixel formats (about 40 or so), with only a > reference to the OpenMAX IL spec. Is there any guarantee about which pixel > formats the decoders on Android 4.1 compliant devices will provide? Or do > I have to be able to handle all those 40 different pixel formats, if some > unknown device with Android 4.1 that I don't have access to would choose > to use some of them? This is of course a bit exaggerated, I know, in > practice I probably would only have to support the relatively few YUV 420 > formats, perhaps in addition to some of the YUV 422 versions. That's only > about 10 different formats.
> But this list also contains the vendor specific > COLOR_QCOM_FormatYUV420SemiPlanar and > COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find information about > how these formats differ from the normal COLOR_FormatYUV420SemiPlanar and > COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL spec > doesn't know about these vendor extensions, obviously.) Are these the only > vendor specific formats that vendors are allowed to use, or is it ok if > qcom devices return e.g. the > QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (0x7FA30C03) pixel > format? Many current qcom hw decoders return data in this format, and it > is very nontrivial to use.
> Will the CTS require device manufacturers to return data in some of the > "generally known/supported" pixel formats? If not, in order to use the raw > decoded data from the MediaCodec API, one would need to test things on one > device per chipset family/generation, more or less.
> For video capture with the low level Camera class, this kind of issue is > already documented, see e.g.
> That section says "This format is guaranteed to be supported for camera > preview images since API level 12; for earlier API versions, check > getSupportedPreviewFormats()."
> For encoding, this at least is manageable where the codec can support more > than one format, one can request which one to use. For decoding, the API > might indicate that the codec supports a few different pixel formats, but > one does not know which one it actually will use until the decoder is > started.
> I haven't had a chance to test this API on the newly released Nexus > devices, but I have played a little with it on the emulator and on a > Pandaboard. On the latter, the video decoder indicates that it supports > two pixel formats, 0x7f000100 which is > OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which I don't > even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.
> Once we get past the issue of pixel formats, then we get to the issue of > how to use each pixel format. The MediaFormat class documents that the > KEY_COLOR_FORMAT key is encoder only, while one obviously need to read it > in order to interpret the decoded data as well.
> The MediaFormat object when using OMX.google.h264.decoder contains data > like this:
> The attributes slice-height, stride, crop-* are not documented in > MediaFormat at all, but since the data seems to be tightly packed, one > could ignore them altogether and get the right result.
> However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a pandaboard, the > MediaFormat for the decoded data from same video file contains this:
> This is the pixel format 2130706688 aka 0x7f000100, > COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one absolutely need > to use all the extra attributes in order to interpret the data properly. > But can one count on them existing since they are not documented?
> With all this in mind, it seems to me that this API is really hard to use > over all Android 4.1 devices without actually testing on almost all of > them.
> I hope that the CTS will force the vendors to conforming to some sane > behaviour at least, to make it even remotely possible for third party > application developers to use it across all devices. (If a device can't > provide what is needed, that's also totally acceptable, as long as it is > signaled in a clear way.)
> There's also a few minor issues with the API that aren't as critical for > using it, but that still feel slightly awkward:
> If I call MediaCodec.createDecoderByType or > MediaCodec.createEncoderByType, the process will crash in native code if > there is no such matching codec. Thus one cannot use these functions at > all, one has to use the MediaCodecList class to iterate all the existing > codecs in order to safely call MediaCodec.createByCodecName instead. I > have submitted a fix for this crash in > https://android-review.googlesource.com/39080, but since it isn't part of > the first 4.1 devices released, one can't trust it unless one only targets > devices above the first 4.1 release. (And since one needs to write the few > lines of code for iterating the MediaCodecList, there's no point in using > the shortcut convenience functions at all.)
> If I were able to call MediaCodec.createEncoderByType safely, which I > can't right now, I would still need to know the pixel formats that this > MediaCodec instance supports. Since the MediaCodec neither allows me to > get the name of the actual codec or the corresponding MediaCodecInfo, I > can't get a hold of this information, and thus there's little point in > using the convenience function createEncoderByType, since one still needs > to iterate the MediaCodecList.
> It would also be useful to know whether a codec is a HW or SW codec (other > than checking for the OMX.google prefix), since an app might be shipping > SW fallback codecs of its own, but would prefer to use the MediaCodec API > only in the case where it actually is hardware accelerated.
> So, a method in MediaCodec for getting the codec name and the > corresponding MediaCodecInfo, and a method in MediaCodecInfo for knowing > whether it is a HW codec would be useful.
> // Martin
Hi Martin,
Is there any document saying how to run the test code which download form https://android-review.googlesource.com/#/c/43410/. I really can not find the right path to run the code, please kindly give some hint. Thank you.
I am having mp4 video. I am extracting this using MediaExtractor, sending it to decoder. Capturing output buffer of decoder and send it to Encoder to encode to different format say mpeg. But decoded output buffer is always giving size as 0 or same as capacity. It is not containing decoded raw data. Please let me know if I am doing anything wrong. My code is similar to below one:
private MediaExtractor extractor;private MediaCodec decoder;private Surface surface;
public void run() {
extractor = new MediaExtractor();
extractor.setDataSource(SAMPLE);
for (int i = 0; i < extractor.getTrackCount(); i++) {
MediaFormat format = extractor.getTrackFormat(i);
String mime = format.getString(MediaFormat.KEY_MIME);
if (mime.startsWith("video/")) {
extractor.selectTrack(i);
decoder = MediaCodec.createDecoderByType(mime);
decoder.configure(format, surface, null, 0);
break;
}
}
if (decoder == null) {
Log.e("DecodeActivity", "Can't find video info!");
return;
}
decoder.start();
ByteBuffer[] inputBuffers = decoder.getInputBuffers();
ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
BufferInfo info = new BufferInfo();
boolean isEOS = false;
long startMs = System.currentTimeMillis();
while (!Thread.interrupted()) {
if (!isEOS) {
int inIndex = decoder.dequeueInputBuffer(10000);
if (inIndex >= 0) {
ByteBuffer buffer = inputBuffers[inIndex];
int sampleSize = extractor.readSampleData(buffer, 0);
if (sampleSize < 0) {
Log.d("DecodeActivity", "InputBuffer BUFFER_FLAG_END_OF_STREAM");
decoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
isEOS = true;
} else {
decoder.queueInputBuffer(inIndex, 0, sampleSize, extractor.getSampleTime(), 0);
extractor.advance();
}
}
}
int outIndex = decoder.dequeueOutputBuffer(info, 10000);
switch (outIndex) {
case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED");
outputBuffers = decoder.getOutputBuffers();
break;
case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
Log.d("DecodeActivity", "New format " + decoder.getOutputFormat());
break;
case MediaCodec.INFO_TRY_AGAIN_LATER:
Log.d("DecodeActivity", "dequeueOutputBuffer timed out!");
break;
default:
ByteBuffer buffer = outputBuffers[outIndex];
Log.v("DecodeActivity", "We can't use this buffer but render it due to the API limit, " + buffer);
// We use a very simple clock to keep the video FPS, or the video
// playback will be too fast
while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {
try {
sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
break;
}
}
decoder.releaseOutputBuffer(outIndex, true);
break;
}
info.size is set to either 0 or same as capacity of the buffer.
On Friday, 13 July 2012 18:30:35 UTC+5:30, Martin Storsjö wrote:
> Hi,
> I've tried taking the new, long desired, MediaCodec API into use in a few > different use cases, and while it looks promising, there's still a few > issues that feels like they need documenting before the new API can be > relied upon properly.
> First off - pixel formats. The scenario is that I'm using the MediaCodec > API to decode video, and I want to render the decoded YUV data myself, I'm > not passing any Surface to the system for rendering. Or I'm encoding raw > video data, captured from the camera via the preview image callback.
> lists a vast array of pixel formats (about 40 or so), with only a > reference to the OpenMAX IL spec. Is there any guarantee about which pixel > formats the decoders on Android 4.1 compliant devices will provide? Or do > I have to be able to handle all those 40 different pixel formats, if some > unknown device with Android 4.1 that I don't have access to would choose > to use some of them? This is of course a bit exaggerated, I know, in > practice I probably would only have to support the relatively few YUV 420 > formats, perhaps in addition to some of the YUV 422 versions. That's only > about 10 different formats.
> But this list also contains the vendor specific > COLOR_QCOM_FormatYUV420SemiPlanar and > COLOR_TI_FormatYUV420PackedSemiPlanar. Where do I find information about > how these formats differ from the normal COLOR_FormatYUV420SemiPlanar and > COLOR_FormatYUV420PackedSemiPlanar respectively? (The OpenMAX IL spec > doesn't know about these vendor extensions, obviously.) Are these the only > vendor specific formats that vendors are allowed to use, or is it ok if > qcom devices return e.g. the > QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (0x7FA30C03) pixel > format? Many current qcom hw decoders return data in this format, and it > is very nontrivial to use.
> Will the CTS require device manufacturers to return data in some of the > "generally known/supported" pixel formats? If not, in order to use the raw > decoded data from the MediaCodec API, one would need to test things on one > device per chipset family/generation, more or less.
> For video capture with the low level Camera class, this kind of issue is > already documented, see e.g.
> That section says "This format is guaranteed to be supported for camera > preview images since API level 12; for earlier API versions, check > getSupportedPreviewFormats()."
> For encoding, this at least is manageable where the codec can support more > than one format, one can request which one to use. For decoding, the API > might indicate that the codec supports a few different pixel formats, but > one does not know which one it actually will use until the decoder is > started.
> I haven't had a chance to test this API on the newly released Nexus > devices, but I have played a little with it on the emulator and on a > Pandaboard. On the latter, the video decoder indicates that it supports > two pixel formats, 0x7f000100 which is > OMX_TI_COLOR_FormatYUV420PackedSemiPlanar, and 0x7f000001 which I don't > even know what it is, it's OMX_COLOR_FormatVendorStartUnused+1.
> Once we get past the issue of pixel formats, then we get to the issue of > how to use each pixel format. The MediaFormat class documents that the > KEY_COLOR_FORMAT key is encoder only, while one obviously need to read it > in order to interpret the decoded data as well.
> The MediaFormat object when using OMX.google.h264.decoder contains data > like this:
> The attributes slice-height, stride, crop-* are not documented in > MediaFormat at all, but since the data seems to be tightly packed, one > could ignore them altogether and get the right result.
> However, when using OMX.TI.DUCATI1.VIDEO.DECODER on a pandaboard, the > MediaFormat for the decoded data from same video file contains this:
> This is the pixel format 2130706688 aka 0x7f000100, > COLOR_TI_FormatYUV420PackedSemiPlanar. With this one, one absolutely need > to use all the extra attributes in order to interpret the data properly. > But can one count on them existing since they are not documented?
> With all this in mind, it seems to me that this API is really hard to use > over all Android 4.1 devices without actually testing on almost all of > them.
> I hope that the CTS will force the vendors to conforming to some sane > behaviour at least, to make it even remotely possible for third party > application developers to use it across all devices. (If a device can't > provide what is needed, that's also totally acceptable, as long as it is > signaled in a clear way.)
> There's also a few minor issues with the API that aren't as critical for > using it, but that still feel slightly awkward:
> If I call MediaCodec.createDecoderByType or > MediaCodec.createEncoderByType, the process will crash in native code if > there is no such matching codec. Thus one cannot use these functions at > all, one has to use the MediaCodecList class to iterate all the existing > codecs in order to safely call MediaCodec.createByCodecName instead. I > have submitted a fix for this crash in > https://android-review.googlesource.com/39080, but since it isn't part of > the first 4.1 devices released, one can't trust it unless one only targets > devices above the first 4.1 release. (And since one needs to write the few > lines of code for iterating the MediaCodecList, there's no point in using > the shortcut convenience functions at all.)
> If I call MediaCodec.getOutputFormat immediately after calling > MediaCodec.start, as the example at > http://developer.android.com/reference/android/media/MediaCodec.html > shows, the process once again crashes in native code. This isn't too much > of an issue, "just don't do that then". I have provided a fix for the > crash in
