IOMX related documentation

[ABS]

Hi

I am a media applications developer who keeps navigating to the native code very often. I use codecs on the android system through IOMX APIs. Although I have been able to work with the codecs quite well, the lack of documentation of the IOMX layer kind of surprises me, although, there are few related files with very good documentation, like, the HardwareAPI.h. 

• Are there any plans to supplement the code with documentation in upcoming releases? 
  
• Can other developers contribute to this? 
  
• Are there existing documents that you point me to?
• Are there plans to release some API documents instead for the IOMX layer and its usage? 
  
• Is there a potential for these codec APIs to be released as part of the next NDK release?
  

My query basically comes in from a point of view of writing custom image / video processing applications can sometimes become tricky unless there is good documentation for reference, esp., considering the buffer copies that get are encountered in certain modes. 

Regards

ABS

[Glenn Kasten]

In Android, the OpenMAX IL codecs are intended for implementing

the platform, not applications.  For applications use MediaCodec API.

If you're implementing the platform for a specific device rather than an app, then

yes you can use OpenMAX IL codecs (via Stagefright),

but you'll need to be prepared to study and trace the code quite a bit,

and be aware the interfaces will not be stable across devices or platform versions.

[ABS]

Glenn

Thanks for your reply.

We have ported an existing linux media player onto Android and customized it for Galaxy Nexus which runs TI's Mobile Application Processor. This player in some sense is a parallel stack to the Android Stagefright stack and uses Stagefright for only Codec access. It more or less works straight away on other devices running on the same chip but cannot run on devices using a different application processor - Qualcomm / Nvidia.

We are looking at an upgrade to the player to make it more portable and MediaCodec does look like a viable option. But, if I were to look at this API from a perspective of certain latency critical components of our player, having access to them at native layer definitely helps.

• I was wondering if there is any plan for adding NDK support any time soon for accessing media codecs at the native layer? 
  
• When the support does come by, will this be the counterpart APIs of MediaCodec which is already present at Java?
• Has any analysis been done for
• power consumption and latency overheads noticed for intensive use-cases like multi-slice video decoding?
  

Regards

ABS

[Glenn Kasten]

Thanks for your suggestions, I am noting your request for native media codec APIs.  

I think that some performance analysis was done of android.media.MediaCodec, but I don't recall the results.

Do you have any performance data on the relative performance of MediaCodec vs. bypassing it?

If you have some results that show the performance of up-call from native to MediaCodec via JNI, vs. bypassing MediaCodec, it would be helpful.

Unfortunately I can't commit to any future APIs.

[ABS]

Glenn

Native Media Codec APIs would really be great. I'm hoping that the next NDK releases have it!

I think we have some data on performance of an in-house decoder when using through MediaCodec v/s using it at native through IOMX. I will look into these numbers and get back to you.

Regards

ABS

[ABS]

Hi

In the AOSP based platform specific solution, we used ANativeWindow APIs (dequeue-queue) available in android source for video display. 

But when we move to NDK, I do not find any suitable API for display.

If we want to use our custom soft codec implementation, the only native API available is lock()-unlockandpost() pair, which is not suited for use as decoder buffers because the design seems to be more suited for displaying graphics/UI kind of stuff. Decoders typically need to hold multiple reference frames and since lock()-unlockandpost() allows lock-ing only a single buffer at a time, this cannot be used straight away by the decoders. 

We will need to have a different set of buffers for decoding data and then, copy from this set to the "lock"ed buffer for displaying the content. (Refer to the attached doc to get an idea of this design)

Some of my queries regarding such a use-case.

• I have provided a potential design - do you notice any flaw in this design? 
• This design is not optimal on performance because of the additional copies involved. Any alternatives that I could explore?
• Can I instead use the dequeue-queue APIs, include necessary libraries and headers from android-source for building and be sure that the solution will work on all Android devices that run the Android version used for building?
• Is there any way for me to plugin my custom codec into stagefright without depending on android-source, so that I can let stage-fright take care of the buffer handling for me?

Regards

ABS

[Glenn Kasten]

ABS, thanks for your notes.  We're tracking your request internally, 

but I can't make any further comments or commitment right now.

[ABS]

Glenn

It would be good if you can provide your thoughts on the design approach I have listed. 

Thanks. 

Regards

ABS