Use ffmpeg as an Android stagefright plugin

[Chih-Wei Huang]

Hello ffmpeg list,
I'm planning to integrate the latest ffmpeg as
a stagefright OMXPlugin with Android-x86 4.3.
I found the stagefright-plugins developed by
Michael Chen is a good start.
(repo: https://github.com/omxcodec/stagefright-plugins.git)
But it's based on android 4.0 (ICS).

My plan is:
* Update stagefright-plugins to work with
stagefright of android 4.3.
* Make it to be a libstagefrighthw.so plugin.
* Create Android.mk to build ffmpeg libs.

Before I go ahead, I'd like to know
if anyone is doing a similar project
to avoid duplicate work.
Thanks a lot!


--
Chih-Wei
Android-x86 project
http://www.android-x86.org

[Chih-Wei Huang]

2013/8/13 Michael Chen <omxc...@gmail.com>:
> I was very glad to hear the news.
> it not a difficult thing to update stagefright-plugins to JB. but you
> should cleanup these code, if feel a bit chaotic.

>
> On Sun, Aug 11, 2013 at 4:25 PM, Chih-Wei Huang <cwh...@android-x86.org>
> wrote:
>>
>> Hello ffmpeg list,
>> I'm planning to integrate the latest ffmpeg as
>> a stagefright OMXPlugin with Android-x86 4.3.
>> I found the stagefright-plugins developed by
>> Michael Chen is a good start.
>> (repo: https://github.com/omxcodec/stagefright-plugins.git)
>> But it's based on android 4.0 (ICS).
>>
>> My plan is:
>> * Update stagefright-plugins to work with
>> stagefright of android 4.3.
>> * Make it to be a libstagefrighthw.so plugin.
>> * Create Android.mk to build ffmpeg libs.

Hi ffmpeg-dev,
I've finished item 3 so now I'm able to build ffmpeg libs
with the AOSP 4.3 tree. I've succeeded to build different
archs including x86, mips, armv7-a and armv7-a-neon.

http://git.android-x86.org/?p=platform/external/ffmpeg.git;a=shortlog;h=refs/heads/jb-x86

Is it possible to merge it in the ffmpeg upstream?


About the stagefright-plugins, I've also succeeded to
update it to build with android 4.3.
With some hacks to frameworks/av,
now I'm able to load the ffmpeg plugins and
play full hd video by the built-in Gallery2 and Youtube.
But the performance is not so good as expected.
If anyone has experience to tune the android multimedia
framework, please give me a hand.
Anyway, will publish it once I clean it up.


Regards,

[Chih-Wei Huang]

Michael Niedermayer said:

> Is it possible to merge it in the ffmpeg upstream?

i see lots of hardcoded config.mak/h files
Consider that new options could be added to configure and would then
be missing from this, also a user might want to build with different
codecs / (de)muxers than the ones hardcoded in these files

That's right.

This is the easier way to create a compatible

makefile for android build system.

Considering the target (android) is well-known,

there is no real benefit to configure codecs/(de)muxers

dynamically.

I also considered to generate the android config.*

from configure script on the fly.

However, the output config.mak of configure is not friendly

to android build system. It exports some variables

like CC, CFLAGS that will conflict with

the android build system.

(Note Android doesn't use sub-make --

all sub-makefiles are included in from the main makefile

and all vars in sub-makefiles are exported)

If you can modify the output of configure, say,

splitting it into two files to let ARCH_*,

HAVE_*, CONFIG_* vars in one file and

others in another file, that would be more

friendly to android build system.

Then I could generate the android config.* on the fly.

[Chih-Wei Huang]

2013/8/21 Michael Niedermayer <mich...@gmx.at>:
> On Tue, Aug 20, 2013 at 11:51:51AM +0800, Chih-Wei Huang wrote:
>
> You should be able to change CFLAGS / CC in the "# OS specific"
> section of configure if they have incorrect values

Android build system has already defined them.
(in its way)
What I need in config.mak is just the
HAVE_*, CONFIG_* stuff.

> configure is using the set cc/cflags to test various things like
> compiler features so they should be set correctly.
>
> But if i misunderstand and theres a real need to omit fields from
> the config files that can be done as well or they could be placed
> under ifdef

Yes, I want to omit them.
So ifdef is a good approach.

[Fabio Fumi]

Hi,

I'd like to ask for your generic advice in back-porting your work to my build, which is a slightly customized 4.1.2 AOSP (found here: https://github.com/Renesas-EMEV2).

I think the pieces are all there already, but I'm wondering whether there's something preventing that, or making it too complex.

Thanks in advance

Fabio
Rome - Italy

[Michael Chen]

Hi,

    Oh, NO, it is easy that Compatible with older Android versions， like 4.1.2 or 4.0.4 AOSP 

[Fabio Fumi]

Thanks for confirming that.

Could you please explain a bit this step, how it could be applied to the AOSP case:

"you should merge my android_frameworks_native(branch: cm_maguro-10.1) and android_frameworks_av(branch: cm_maguro-10.1) code"

thanks
Fabio

[Fabio Fumi]

Hi Chih,

I've completed the merge of frameworks/av and framewroks/native and added the stagefright-plugins and ffmpeg to my project. I enabld the USES_NAM and built new system.

Startup wnet fine, but on YouTube test playback (which went just fine before), I get a crash (below).

Note how I haven't enabled he new ffmpeg codecs in media_codecs.xml and I'm using Google SW codec for h264:

        <MediaCodec name="OMX.google.h264.decoder" type="video/avc"/>

Suggestions?

thanks
Fabio


...
D/H264Dec (  707): H264SwDecInit#
D/H264Dec (  707): H264SwDecInit# decInst 0x2a055c08 noOutputReordering 0
D/H264Dec (  707): H264SwDecInit# OK: return 0x2a0b1238
I/OMXCodec(  707): [OMX.google.h264.decoder] AVC profile = 66 (Baseline), level = 30
I/OMXCodec(  707): [OMX.google.h264.decoder] video dimensions are 320 x 240
I/OMXCodec(  707): [OMX.google.h264.decoder] Crop rect is 320 x 240 @ (0, 0)
...
D/H264Dec (  707): H264SwDecDecode#
D/H264Dec (  707): H264SwDecDecode# decInst 0x2a0b1238  pInput 0x4201ed88  pOutput 0x4201ed7c
D/H264_decoder(  707): Access unit boundary
D/H264_decoder(  707): SEQ PARAM SET
D/H264Dec (  707): H264SwDecDecode# OK: DecResult 1
D/H264Dec (  707): H264SwDecDecode#
D/H264Dec (  707): H264SwDecDecode# decInst 0x2a0b1238  pInput 0x4201ed88  pOutput 0x4201ed7c
D/H264_decoder(  707): Access unit boundary
D/H264_decoder(  707): PIC PARAM SET
D/H264Dec (  707): H264SwDecDecode# OK: DecResult 1
D/H264Dec (  707): H264SwDecDecode#
D/H264Dec (  707): H264SwDecDecode# decInst 0x2a0b1238  pInput 0x4201ed88  pOutput 0x4201ed7c
D/H264_decoder(  707): Access unit boundary
D/H264_decoder(  707): IDR
D/H264_decoder(  707): SLICE HEADER
D/H264Dec (  707): H264SwDecDecode# OK: DecResult 4
D/H264Dec (  707): H264SwDecGetInfo#
D/H264Dec (  707): H264SwDecGetInfo# decInst 0x2a0b1238  pDecInfo 0x4201eda8
D/H264Dec (  707): H264SwDecGetInfo# OK
D/H264Dec (  707): H264SwDecDecode#
D/H264Dec (  707): H264SwDecDecode# decInst 0x2a0b1238  pInput 0x4201ed88  pOutput 0x4201ed7c
D/H264_decoder(  707): IDR
D/H264_decoder(  707): SLICE HEADER
D/H264_decoder(  707): (null)
D/H264Dec (  707): H264SwDecDecode# OK: DecResult 2
D/H264Dec (  707): H264SwDecNextPicture#
D/H264Dec (  707): H264SwDecNextPicture# decInst 0x2a0b1238 pOutput 0x4201ed98 flushBuffer 0
D/H264Dec (  707): H264SwDecNextPicture# OK: return H264SWDEC_PIC_RDY
F/libc    (  707): Fatal signal 11 (SIGSEGV) at 0x00000008 (code=1), thread 1818 (le.h264.decoder)
...
I/DEBUG   (   70):     #00  pc 00011ed0  /system/lib/libc.so
I/DEBUG   (   70):     #01  pc 0000ad6b  /system/lib/libstagefright_omx.so (android::OMX::CallbackDispatcher::post(android::omx_message const&)+14)
I/DEBUG   (   70):     #02  pc 0000b5e5  /system/lib/libstagefright_omx.so (android::OMX::OnEmptyBufferDone(void*, OMX_BUFFERHEADERTYPE*)+30)
I/DEBUG   (   70):     #03  pc 0000ebd9  /system/lib/libstagefright_omx.so (android::SoftOMXComponent::notifyEmptyBufferDone(OMX_BUFFERHEADERTYPE*)+16)
I/DEBUG   (   70):     #04  pc 00009f60  /system/lib/libstagefright_soft_h264dec.so (android::SoftAVC::onQueueFilled(unsigned long)+508)

[Fabio Fumi]

Also after adding new ffmpeg codecs to I still get a crash on H264 (avc) playback.

Added ffmpeg codecs to device/renesas/emev/media_codecs.xml

Rebuilt... test... still crashing:

V/AwesomePlayer(  758): track of type 'video/avc' does not publish bitrate
V/AwesomePlayer(  758): mBitrate = -1 bits/sec
V/AwesomePlayer(  758): haveAudio:1, haveVideo:1
V/AwesomePlayer(  758): initVideoDecoder flags=0x0
I/EV2OMXPlugin(  758): ***run OMFPlugin ------- makeComponentInstance !!!!!!!!
I/OMXCodec(  758): [OMX.RENESAS.VIDEO.DECODER.H264] AVC profile = 66 (Baseline), level = 30
I/OMXCodec(  758): [OMX.RENESAS.VIDEO.DECODER.H264] video dimensions are 640 x 344
E/OMXNodeInstance(  758): OMX_GetExtensionIndex (index:'7fffffff') failed
I/EV2OMXPlugin(  758): ***run OMFPlugin ------- destroyComponentInstance !!!!!!!!
E/OMXMaster(  758): Invalid OMX component name 'OMX.ffmpeg.h264.decoder'
E/MediaPlayer(  996): error (1, -2147483648)
E/MediaPlayer(  996): Error (1,-2147483648)
W/YouTube (  996): (unknown) MediaPlayer error during prepare [what=1, extra=-2147483648]
W/YouTube (  996): (unknown) Retrying MediaPlayer error [retry=1, max=3]
F/libc    (  758): @@@ ABORTING: INVALID HEAP ADDRESS IN dlfree addr=0x2a023ad0
F/libc    (  758): Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1), thread 794 (Binder_2)
...
I/DEBUG   (   70):     #00  pc 000137b8  /system/lib/libc.so
I/DEBUG   (   70):     #01  pc 00015b95  /system/lib/libc.so (dlfree+1628)
I/DEBUG   (   70):     #02  pc 00016d43  /system/lib/libc.so (free+10)
I/DEBUG   (   70):     #03  pc 0007a853  /system/lib/libstagefright.so
I/DEBUG   (   70):     #04  pc 0007ac0d  /system/lib/libstagefright.so (android::TimedEventQueue::~TimedEventQueue()+36)
I/DEBUG   (   70):     #05  pc 000538ad  /system/lib/libstagefright.so (android::AwesomePlayer::~AwesomePlayer()+288)

[Fabio Fumi]

I later noticed some libs weren't built... not sure why, but I had to touch the merged file to trigger the build... Anyway, after touching all merged and new files and rebuilding, te crash has disappeared, but I still get this error:

V/SoftOMXPlugin(   75): makeComponentInstance 'OMX.ffmpeg.h264.decoder'
E/SoftOMXPlugin(   75): unable to dlopen libstagefright_soft_ffmpegvdec.so: Cannot load library: load_library[1093]: Library 'libstagefright_soft_ffmpegvdec.so' not found
E/OMXMaster(   75): Failed making plugin component instance (index 16)
E/MediaPlayer( 1610): error (1, -2147483648)
E/MediaPlayer( 1610): Error (1,-2147483648)

What I noticed is that all libstagefright_soft_* libs are rebuilt and found in /system/lib, while libstagefright_soft_ffmpegvdec.so is not...

It is built, as I find it in the AOSP build out, but not moved to target /system/lib. Why?

ffxx68@ubuntu:/media/u02/RenesasEV2/jb/out/target/product/emev$ find . -name libstagefright_soft_ffmpegvdec.so
./obj/lib/libstagefright_soft_ffmpegvdec.so
./obj/SHARED_LIBRARIES/libstagefright_soft_ffmpegvdec_intermediates/LINKED/libstagefright_soft_ffmpegvdec.so
./symbols/system/lib/libstagefright_soft_ffmpegvdec.so
ffxx68@ubuntu:/media/u02/RenesasEV2/jb/out/target/product/emev$ find . -name libstagefright_soft_mpeg4dec.so
./system/lib/libstagefright_soft_mpeg4dec.so
./obj/lib/libstagefright_soft_mpeg4dec.so
./obj/SHARED_LIBRARIES/libstagefright_soft_mpeg4dec_intermediates/LINKED/libstagefright_soft_mpeg4dec.so
./symbols/system/lib/libstagefright_soft_mpeg4dec.so

On Sunday, 11 August 2013 01:25:25 UTC-7, Chih-Wei Huang wrote:

[Chih-Wei Huang]

You need to add it to PRODUCT_PACKAGES in some mk.

--
You received this message because you are subscribed to the Google Groups "Android-x86" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-x86...@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at http://groups.google.com/group/android-x86.
For more options, visit https://groups.google.com/d/optout.

[Fabio Fumi]

Yes! I had to add below lines to my device.mk

PRODUCT_PACKAGES += \
        libstagefright_soft_ffmpegvdec

Now, playback happens, using the soft_ffmpeg decoder, but video rendering is heavily distorted.
I'm attaching a couple video examples (both captured from YouTube app):

Example 1) 20141106_163957.jpg - adb logcat saved in http://pastebin.com/5CiDJ3vh

Example 2) 20141106_163957.jpg - adb logcat saved in http://pastebin.com/dbqG2qfA

How can I debug that? I understand it's hard to help me remotely about that, but if you could give a hint about what to start debugging...
What modules should I enable debug, to start with?

thanks
Fabio

[Fabio Fumi]

second example attachement 20141106_163805.jpg was missing ...

On Sunday, 11 August 2013 10:25:25 UTC+2, Chih-Wei Huang wrote:

[Michael Chen]

hints

audio:

1. E/OMXCodec( 2556): [OMX.RENESAS.AUDIO.DECODER.AAC] setParameter('OMX_IndexParamAudioAac') failed (err = -1010)
2. E/OMXCodec( 2556): [OMX.RENESAS.AUDIO.DECODER.AAC] setAACFormat() failed (err = -1010)

video:

1. E/SoftFFmpegVideo( 2556): Deinterlacing failed

[Chih-Wei Huang]

2014-11-07 0:09 GMT+08:00 Fabio Fumi <ffu...@googlemail.com>:
> Yes! I had to add below lines to my device.mk
>
> PRODUCT_PACKAGES += \
> libstagefright_soft_ffmpegvdec

Have you added libstagefright_soft_ffmpegadec
and libFFmpegExtractor ?

[Fabio Fumi]

I've added that now, but it didn't change the outcome...

I'm aware of audio error, that HW codecs don't work, but not a big issue as far as google SW ones does.

Video is the actual issue. Seen the two pictures? First one, with scrambled colors only, give no error messages in logs...

The "Deinterlacing failed" message gets from second example, which is completely off.

Anyway, maybe debugging this I might solve both cases... "Deinterlacing failed" comes from this failing check (in external/ffmpeg/libavcodec/imgconvert.c):

int avpicture_deinterlace(AVPicture *dst, const AVPicture *src,
                          enum AVPixelFormat pix_fmt, int width, int height)
{
    int i;

    if (pix_fmt != AV_PIX_FMT_YUV420P &&
        pix_fmt != AV_PIX_FMT_YUVJ420P &&
        pix_fmt != AV_PIX_FMT_YUV422P &&
        pix_fmt != AV_PIX_FMT_YUVJ422P &&
        pix_fmt != AV_PIX_FMT_YUV444P &&
        pix_fmt != AV_PIX_FMT_YUV411P &&
        pix_fmt != AV_PIX_FMT_GRAY8)
        return -1;
    if ((width & 3) != 0 || (height & 3) != 0)
        return -1;

Any hint?

[Fabio Fumi]

Btw - testing downloaded files (as suggested in https://github.com/omxcodec/stagefright-plugins), playback is fine, at east with respect to image quality. Similar for both the test .mov and .mp4 files.

Playback hangs and locks once in a while, for a few seconds, with logcat showing lots of:

V/AwesomePlayer( 2090): we're late by 125631 us (0.13 secs)
V/AwesomePlayer( 2090): we're late by 125631 us (0.13 secs) dropping one after 0 frames
V/AwesomePlayer( 2090): we're late by 125382 us (0.13 secs)
V/AwesomePlayer( 2090): we're late by 125382 us (0.13 secs) dropping one after 0 frames
V/AwesomePlayer( 2090): we're late by 120819 us (0.12 secs)
V/AwesomePlayer( 2090): we're late by 120819 us (0.12 secs) dropping one after 0 frames
...

This happens in particular when there's extesive image changing. Less when video is more static.

Seems like a lack of CPU bandwidth... Mediaserver sets CPU above 50% (though never 100%).

  PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
 2090  0  52% S    16  64160K  17060K  fg media    /system/bin/mediaserver
 3175  0   1% R     1   1072K    476K     root     top
  257  0   0% S    73 549956K  38280K  fg system   system_server
 3009  0   0% S    15 492776K  37872K  fg u0_a69   com.mxtech.videoplayer.ad
...

Anyway, with respect to image quelity I don't see any significant difference in the video attribute, with repsect to those streamed though YouTube. See also logcat in http://pastebin.com/tnrt9NB6

On Sunday, 11 August 2013 01:25:25 UTC-7, Chih-Wei Huang wrote:

[Fabio Fumi]

Forgetting for the moment the image issues above (as I've restored playback
through the HW H264 codec), I'm now hitting another issue...

Testing on a "Live" (m3u8) streaming (.m3u8 URLs; which is the reason I'm
trying to use ffmpeg SW codecs at all, as HW one failed to playback these),
I got this failure:

V/ACodec ( 75): onAllocateComponent
V/ACodec ( 75): findMatching OMX codecs 'video/avc' (SW only)
V/OMXCodec( 75): matchComponentName (null)
V/OMXCodec( 75): matchComponentName (null)
V/ACodec ( 75): Found matching codec: 'OMX.ffmpeg.h264.decoder'
...
V/ACodec ( 75): ComponentName: OMX.ffmpeg.h264.decoder
V/ACodec ( 75): nativeWindow creation
V/ACodec ( 75): initNativeWindow
E/OMXNodeInstance( 75): OMX_GetExtensionIndex (index:'4083345d') failed

Probably ACodec need being patched for ffmpeg too? For one, I don't expect
it to use native window...

Added below check, in ACodec::LoadedState::onConfigureComponent:

if (msg->findObject("native-window", &obj)
&& strncmp("OMX.google.", mCodec->mComponentName.c_str(), 11)
&& strncmp("OMX.RENESAS.", mCodec->mComponentName.c_str(), 12)
#ifdef USES_NAM
&& strncmp("OMX.ffmpeg.", mCodec->mComponentName.c_str(), 11)
#endif

Now ACodec doesn't crash anymore and (just like the HW codec!) it goes on
in logcat but no image is shown on screen:

...
V/ACodec ( 75): [OMX.google.aac.decoder] calling fillBuffer 0x2a0671b8
V/ACodec ( 75): [OMX.google.aac.decoder] Now Executing
V/ACodec ( 75): [OMX.google.aac.decoder] onOMXFillBufferDone 0x2a06a998
time 0 us, flags = 0x00000000, dataPtr = 0x42cef000
V/ACodec ( 75): [OMX.google.aac.decoder] calling fillBuffer 0x2a06a998
V/ACodec ( 75): [OMX.google.aac.decoder] onOMXEmptyBufferDone 0x2a0df9c0
V/ACodec ( 75): [OMX.google.aac.decoder] calling emptyBuffer 0x2a0df9c0
w/ time 92877 us
...

Could something still be missing in ACodec?

Thanks once again in advance for any help!
Fabio

[Fabio Fumi]

Sorry posted wrong logcat fragment in my last update.
During playback I keep getting messages like these below, though no image
is shown:

...
V/ACodec ( 75): [OMX.ffmpeg.h264.decoder] calling emptyBuffer 0x2a06c4f0
w/ time 53232200 us
V/ACodec ( 75): [OMX.google.aac.decoder] onOMXFillBufferDone 0x2a06aa48
time 53266566 us, flags = 0x00000000, dataPtr = 0x42cf7000

V/ACodec ( 75): [OMX.google.aac.decoder] onOMXEmptyBufferDone 0x2a0df9c0
V/ACodec ( 75): [OMX.google.aac.decoder] calling emptyBuffer 0x2a0df9c0

w/ time 53336233 us
V/ACodec ( 75): [OMX.google.aac.decoder] calling emptyBuffer 0x2a07a038
w/ time 53359455 us
V/ACodec ( 75): [OMX.ffmpeg.h264.decoder] onOMXFillBufferDone 0x2a067240
time 52912200 us, flags = 0x00000000, dataPtr = 0x4452e000
V/ACodec ( 75): [OMX.ffmpeg.h264.decoder] calling fillBuffer 0x2a067240
V/ACodec ( 75): [OMX.ffmpeg.h264.decoder] onOMXEmptyBufferDone 0x2a06cfb0
V/ACodec ( 75): [OMX.ffmpeg.h264.decoder] calling emptyBuffer 0x2a06cfb0
w/ time 53152200 us
V/ACodec ( 75): [OMX.ffmpeg.h264.decoder] onOMXFillBufferDone 0x2a0d4d48
time
...

On Sunday, 11 August 2013 01:25:25 UTC-7, Chih-Wei Huang wrote:
>