WebRTC Android - H.264 MediaCodec Hardware Video Decoding with FFMpeg

[Ryan Dawson]

I have been attempting to get hardware video decoding to work with WebRTC on Android and would love some help determining what I may be doing wrong. I am hoping to get H.264 hardware video decoding working on Android via the MediaCodec API support in the FFmpeg version included with WebRTC.

The reason I have chosen to venture down this path is that I am seeing poor video decoding performance on my Google Pixel 2 test device using the default WebRTC selected VP8 codec. I also saw poor video decoding performance when using the FFmpeg H.264 codec (which is software from what I can tell).

I am using a Vagrant/Virtual Box Ubuntu 18.04 instance on Windows 10 to cross-compile FFmpeg for Android and compile WebRTC for Android.

I have been cross-compiling FFmpeg in my environment using the following command-line:

./chromium/scripts/build_ffmpeg.py android arm-neon -- --enable-jni --enable-mediacodec --enable-decoder=h264_mediacodec --enable-hwaccel=h264_mediacodec --enable-parser=h264 --enable-demuxer=h264 --enable-demuxer=mov

The configure line generated by the execution of the build_ffmpeg.py script is as follows:

Chrome configure/build:
Running /home/vagrant/Google/webrtc_android/src/third_party/ffmpeg/configure
 --disable-everything
 --disable-all
 --disable-doc
 --disable-htmlpages
 --disable-manpages
 --disable-podpages
 --disable-txtpages
 --disable-static
 --enable-avcodec
 --enable-avformat
 --enable-avutil
 --enable-fft
 --enable-rdft
 --enable-static
 --enable-libopus
 --disable-debug
 --disable-bzlib
 --disable-error-resilience
 --disable-iconv
 --disable-lzo
 --disable-network
 --disable-schannel
 --disable-sdl2
 --disable-symver
 --disable-xlib
 --disable-zlib
 --disable-securetransport
 --disable-faan
 --disable-alsa
 --disable-autodetect
 --enable-decoder=vorbis,libopus,flac
 --enable-decoder=pcm_u8,pcm_s16le,pcm_s24le,pcm_s32le,pcm_f32le,mp3
 --enable-decoder=pcm_s16be,pcm_s24be,pcm_mulaw,pcm_alaw
 --enable-demuxer=ogg,matroska,wav,flac,mp3,mov
 --enable-parser=opus,vorbis,flac,mpegaudio
 --extra-cflags=-I/home/vagrant/Google/webrtc_android/src/third_party/opus/src/include
 --disable-linux-perf
 --x86asmexe=yasm
 --enable-small
 --enable-cross-compile
 --sysroot=/home/vagrant/Google/webrtc_android/src/third_party/android_ndk/platforms/android-16/arch-arm
 --extra-cflags=-I/home/vagrant/Google/webrtc_android/src/third_party/android_ndk/sysroot/usr/include
 --extra-cflags=-I/home/vagrant/Google/webrtc_android/src/third_party/android_ndk/sysroot/usr/include/arm-linux-androideabi
 --extra-cflags=--target=arm-linux-androideabi
 --extra-ldflags=--target=arm-linux-androideabi
 --extra-ldflags=--gcc-toolchain=/home/vagrant/Google/webrtc_android/src/third_party/android_ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/
 --target-os=android
 --arch=arm
 --enable-armv6
 --enable-armv6t2
 --enable-vfp
 --enable-thumb
 --extra-cflags=-march=armv7-a
 --enable-neon
 --extra-cflags=-mtune=generic-armv7-a
 --extra-cflags=-mfloat-abi=softfp
 --extra-cflags=-mfpu=neon
 --enable-pic
 --cc=clang
 --cxx=clang++
 --ld=clang
 --extra-ldflags=-fuse-ld=lld
 --enable-demuxer=aac
 --enable-parser=aac
 --enable-decoder=aac
 --enable-jni
 --enable-mediacodec
 --enable-decoder=h264_mediacodec
 --enable-hwaccel=h264_mediacodec
 --enable-parser=h264
 --enable-demuxer=h264
 --enable-demuxer=mov

I then copy the generated config files and run the gn generation python script as follows:

./chromium/scripts/copy_config.sh
./chromium/scripts/generate_gn.py

Finally, I build WebRTC using:

ninja -C out/DefaultH264

The GN generation line I have been using is as follows:

gn gen out/DefaultH264 --args='target_os="android" target_cpu="arm" rtc_use_h264=true rtc_initialize_ffmpeg=true proprietary_codecs=true ffmpeg_branding="Chrome"'

 

Upon completing these steps and running my test application on device, the call to avcodec_open2 (line 227) in src\modules\video_coding\codecs\h264\h264_decoder_impl.cc fails with a result code of -542398533 (Generic error in an external library). This is unfortunately where I have become stuck. I have not yet found any help via Google research on this issue. An IRC conversation on the #ffmpeg channel seemed to indicate that libavcodec might not be getting the JVM provided to it via an av_jni_set_java_vm call and the thought was that I could make such a call just before the avcodec_open2 call, but I am unsure how to access the JVM via WebRTC at that point and I am not sure if that is something that needs to be done in the WebRTC world. 

I would appreciate any help that can be provided to get me going in to right direction and show the error in my ways so I can finally get this working. :)

My WebRTC repro is currently at commit ac5bbd940ed31f8a58095952f4dcdcbb1b58203c (6/22/2018) and the third_party/ffmpeg repro is currently at commit 90210b5e10d3917567a3025e4853704bfefd8384 (4/30/2018) with the FFmpeg RELEASE file indicating 3.4.git.

Thanks for the help,

Ryan

[mag...@webrtc.org]

Hi Ryan,

Are you aware of the hardware video codecs we already have for Android here https://cs.chromium.org/chromium/src/third_party/webrtc/sdk/android/src/java/org/webrtc/HardwareVideoDecoder.java? You can try them out in the native AppRTCMobile.

Thanks,

Magnus

[Ryan Dawson]

Hi Magnus,

Thank you for the reply.

Yes, I did see the HardwareVideoDecoder implementation on the Java side and I noted that is what the Android example application used via the DefaultVideoDecoderFactory. We are working on the C++ side with our plugin though. 

However, I did make a test modification to our plugin to swap out the C++ InternalDecoderFactory with the Java DefaultVideoDecoderFactory using JNI. That did seem to work to get MediaCodec hardware decoding running, but we are still seeing performance issues with our implementation and are continuing to investigate the cause to determine the proper direction to take.

It seems that a lot of the time the client buffers many seconds of video from the server and falls far behind. Is there a way to reset the video stream, ignore the buffered frames and just have the client pick up at the latest frame from the server?

Thanks,

Ryan

[Wesley]

Hi,

Right now I am facing the same problem, I use H264 HW en/decoder at two android phone. the log like below

09-18 13:34:23.215 32665-528/com.wtk.mobile.WtkRtc E/org.webrtc.Logging: MediaCodecVideoDecoder: Very high decode time: 229ms. Q size: 4. Might be caused by resuming H264 decoding after a pause.

09-18 13:34:23.235 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 49. 360 x 480. Color: 2141391876. TS: 1633. DecTime: 200. DelayTime: 0

09-18 13:34:23.335 32665-528/com.wtk.mobile.WtkRtc E/org.webrtc.Logging: MediaCodecVideoDecoder: Very high decode time: 225ms. Q size: 3. Might be caused by resuming H264 decoding after a pause.

09-18 13:34:23.405 32665-528/com.wtk.mobile.WtkRtc W/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder is too far behind. Try to drain. Received: 61. Decoded: 56

09-18 13:34:23.445 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 56. 360 x 480. Color: 2141391876. TS: 1866. DecTime: 65. DelayTime: 0

    MediaCodecVideoDecoder: Decoder frame in # 61. Type: 4. Buffer # 1. TS: 2033. Size: 290

09-18 13:34:23.445 32665-528/com.wtk.mobile.WtkRtc W/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder is too far behind. Try to drain. Received: 62. Decoded: 57

09-18 13:34:23.465 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 57. 360 x 480. Color: 2141391876. TS: 1900. DecTime: 96. DelayTime: 0

    MediaCodecVideoDecoder: Decoder frame in # 62. Type: 4. Buffer # 0. TS: 2066. Size: 51

09-18 13:34:23.465 32665-528/com.wtk.mobile.WtkRtc W/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder is too far behind. Try to drain. Received: 63. Decoded: 58

09-18 13:34:23.475 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 58. 360 x 480. Color: 2141391876. TS: 1933. DecTime: 75. DelayTime: 0

    MediaCodecVideoDecoder: Decoder frame in # 63. Type: 4. Buffer # 2. TS: 2100. Size: 287

09-18 13:34:23.505 32665-528/com.wtk.mobile.WtkRtc W/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder is too far behind. Try to drain. Received: 64. Decoded: 59

09-18 13:34:23.545 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 59. 360 x 480. Color: 2141391876. TS: 1966. DecTime: 110. DelayTime: 0

    MediaCodecVideoDecoder: Decoder frame in # 64. Type: 4. Buffer # 3. TS: 2133. Size: 185

09-18 13:34:23.585 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 60. 360 x 480. Color: 2141391876. TS: 2000. DecTime: 177. DelayTime: 0

09-18 13:34:23.775 32665-528/com.wtk.mobile.WtkRtc W/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder is too far behind. Try to drain. Received: 70. Decoded: 65

09-18 13:34:23.795 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 65. 360 x 480. Color: 2141391876. TS: 2166. DecTime: 168. DelayTime: 0

09-18 13:34:23.815 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame in # 70. Type: 4. Buffer # 0. TS: 2333. Size: 45

09-18 13:34:23.865 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Decoder frame out # 66. 360 x 480. Color: 2141391876. TS: 2200. DecTime: 165. DelayTime: 0

09-18 13:34:39.815 32665-528/com.wtk.mobile.WtkRtc E/org.webrtc.Logging: MediaCodecVideoDecoder: Very high decode time: 16044ms. Q size: 4. Might be caused by resuming H264 decoding after a pause.

09-18 13:34:39.815 32665-528/com.wtk.mobile.WtkRtc I/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Frames decoded: 70. Received: 74. Bitrate: 7 kbps. Fps: 3. DecTime: 162. DelayTime: 0 for last 18212 ms.

09-18 13:34:39.885 32665-528/com.wtk.mobile.WtkRtc E/org.webrtc.Logging: MediaCodecVideoDecoder: Very high decode time: 16065ms. Q size: 4. Might be caused by resuming H264 decoding after a pause.

09-18 13:34:39.885 32665-533/com.wtk.mobile.WtkRtc W/MediaCodecVideoDecoder: MediaCodecVideoDecoder: Input resolution changed from 360 x 480 to 240 x 320

But when I use looptest on the same phone, the performance much better.

And the same network sutution, if I change HW H264 en/decoder to SW en/decoder, the performance much better than the HW H264 en/decoder.

Has any update for this problem?

在 2018年7月18日星期三 UTC+8下午2:27:29，Ryan Dawson写道：