Hardware acceleration broken in Android 4.4 on newer Chromium versions

[Daniel]

Hello Chromium team!

We really would like to use the newest Chromium browser on our Android-compatible custom tablet hardware.
We are able to build the newest Chromium from source and the app works, but the hardware acceleration for video does not work.
Video HW acceleartion works for older Chromium versions (for example Chrome 30 or 43), but not for the newest versions.

We use a custom build of Android 4.4 (KitKat) on our custom tablet hardware.
Currently we use the standard Android 4.4 AOSP browser (based on Chromium 30).

In short:

• Chrome 30: HW acceleration for Video works (using MediaPlayer)
• Chrome 43: HW acceleration for Video works (using MediaPlayer)
• Chrome 53+: HW acceleration for Video does NOT work (we think because MediaPlayer is no longer used)
  

We think the new Unified Media Pipeline in Chromium could be the reason why HW accelerated video is not possible on our hardware.
We tried the fallback code, but it does not seem to work anymore.

We hope someone can point us in the right direction, what we need to do to activate HW acceleration again.

Regards,
Daniel

[Dale Curtis]

The unified media pipeline requires a working MediaCodec implementation. If this is unavailable on your hardware you will need to launch chrome with --disable-accelerated-video-decode (or flip it in chrome://flags) so we can fall back to MediaPlayer.

Note, any site using MSE (e.g., YouTube will not work in this configuration); though if MediaCodec is broken it shouldn't have worked previously either. If it did, please file a bug and include chrome://gpu, chrome://media-internals, and logcat information for the failure.

- dale

[Chen Zhixiang]

We are planing to update our custom embedder's chromium kernel from M43 to M56+.

The 2 bugs which are found on M43:

(1) M43 uses MediaPlayer, (i guess M46+ switched to MediaCodec), which doesn't support HLS streaming video pause-then-resume.

(2) 2d-canvas's sub-texture copy(gpu to gpu) broken on Android 7.0, which is not related here

So, i'm not familiar with Android MediaPlayer or MediaCodec API. but i thought: there are in fact both "HW acceleration"? Or the "HW acceleration" just means the video decoding on gpu driver? Because where MediaPlayer or MediaCodec, the decodec video frames are always passed to Chromium's compositing code path.

[Chen Zhixiang]

I guess the "HW acceleration" points on "video decoding"? For if use FFmpeg on M53+ (combined with MediaCodec), FFmpeg is not a "HW acceleration" video decoding component, but a software based video decoding lib.

On Tuesday, February 7, 2017 at 12:49:56 PM UTC+8, Chen Zhixiang wrote:

We are planing to update our custom embedder's chromium kernel from M43 to M56+.

The 2 bugs which are found on M43:

(1) M43 uses MediaPlayer, (i guess M46+ switched to MediaCodec), which doesn't support HLS streaming video pause-then-resume.

(2) 2d-canvas's sub-texture copy(gpu to gpu) broken on Android 7.0, which is not related here

So, i'm not familiar with Android MediaPlayer or MediaCodec API. but i thought: they are both "HW acceleration"? Or the "HW acceleration" just means the video decoding on gpu driver? Because where MediaPlayer or MediaCodec, the decodec video frames are always passed to Chromium's compositing code path.

[Dale Curtis]

On Mon, Feb 6, 2017 at 8:49 PM, Chen Zhixiang <cten...@gmail.com> wrote:

We are planing to update our custom embedder's chromium kernel from M43 to M56+.

The 2 bugs which are found on M43:

(1) M43 uses MediaPlayer, (i guess M46+ switched to MediaCodec), which doesn't support HLS streaming video pause-then-resume.

It's not clear what bug you're describing here. For HLS playback we still use MediaPlayer but through a new mechanism instead of the WebMediaPlayerAndroid path. MediaCodec has always been the only way to play MSE content like YouTube. Starting in M52+ we use ffmpeg to demux src= content, MediaCodec for "accelerated" video decoding, and ffmpeg for audio decoding (except in the encrypted content case). 

 

(2) 2d-canvas's sub-texture copy(gpu to gpu) broken on Android 7.0, which is not related here

So, i'm not familiar with Android MediaPlayer or MediaCodec API. but i thought: there are in fact both "HW acceleration"? Or the "HW acceleration" just means the video decoding on gpu driver? Because where MediaPlayer or MediaCodec, the decodec video frames are always passed to Chromium's compositing code path.

Neither is always hardware acceleration, but both generally are; internally either may use software decoders depending on the device's codec support. For common H264 bitstreams this is almost always hardware decoding.

[akim...@gmail.com]

Hi Dale,

I believe we've faced the same issue. We are trying to to upgrade build-in webview in AOSP.
We are using Android 6.0. CPU consumption during playing h264 encoded video became very high after upgrading to latest webview. We are sure that corresponding MediaCodec for h264 exists and it worked fine with old webview version.

Could you please help us with the following:
- Is it possible to check which codec exactly is used (I see nothing related in logcat during youtube playing for instance)? How could we do it?
- I see my 'hardware-accelerated' decoder in MediaCodecInfo.getCodecInfos(). Does it mean that webview also see it and should decode corresponding video using it? How does webview/chromium understand whether decoder is hardware accelerated or not?
- I do understand that this is very device specific question but could you please help us with understanding what is expected load on CPU during video decoding (very roughly)? For instance we want to play 1080p h264 decoded video on 4-core arm64 processor. What CPU load should we expect in case of software/hardware decoding?

Many thanks in advance.

Regards,
Sergey Akimov

вторник, 7 февраля 2017 г., 22:09:48 UTC+3 пользователь Dale Curtis написал:

> On Mon, Feb 6, 2017 at 8:49 PM, Chen Zhixiang <cten...@gmail.com> wrote:
>
>
> We are planing to update our custom embedder's chromium kernel from M43 to M56+.
>
>
> The 2 bugs which are found on M43:
> (1) M43 uses MediaPlayer, (i guess M46+ switched to MediaCodec), which doesn't support HLS streaming video pause-then-resume.
>
>
> It's not clear what bug you're describing here. For HLS playback we still use MediaPlayer but through a new mechanism instead of the WebMediaPlayerAndroid path. MediaCodec has always been the only way to play MSE content like YouTube. Starting in M52+ we use ffmpeg to demux src= content, MediaCodec for "accelerated" video decoding, and ffmpeg for audio decoding (except in the encrypted content case). 
>  
>
>
> (2) 2d-canvas's sub-texture copy(gpu to gpu) broken on Android 7.0, which is not related here
>
>
> So, i'm not familiar with Android MediaPlayer or MediaCodec API. but i thought: there are in fact both "HW acceleration"? Or the "HW acceleration" just means the video decoding on gpu driver? Because where MediaPlayer or MediaCodec, the decodec video frames are always passed to Chromium's compositing code path.
>
>
> Neither is always hardware acceleration, but both generally are; internally either may use software decoders depending on the device's codec support. For common H264 bitstreams this is almost always hardware decoding.
>  
>
>
>
>
> On Tuesday, February 7, 2017 at 1:39:57 AM UTC+8, Dale Curtis wrote:
> The unified media pipeline requires a working MediaCodec implementation. If this is unavailable on your hardware you will need to launch chrome with --disable-accelerated-video-decode (or flip it in chrome://flags) so we can fall back to MediaPlayer.
>
>
> Note, any site using MSE (e.g., YouTube will not work in this configuration); though if MediaCodec is broken it shouldn't have worked previously either. If it did, please file a bug and include chrome://gpu, chrome://media-internals, and logcat information for the failure.
>
>
>
>
>
> - dale
>
>
> On Mon, Feb 6, 2017 at 5:43 AM, Daniel <tde.bewa...@gmail.com> wrote:
>
> Hello Chromium team!
>
> We really would like to use the newest Chromium browser on our Android-compatible custom tablet hardware.
> We are able to build the newest Chromium from source and the app works, but the hardware acceleration for video does not work.
> Video HW acceleartion works for older Chromium versions (for example Chrome 30 or 43), but not for the newest versions.
>
> We use a custom build of Android 4.4 (KitKat) on our custom tablet hardware.
> Currently we use the standard Android 4.4 AOSP browser (based on Chromium 30).
>
> In short:

> Chrome 30: HW acceleration for Video works (using MediaPlayer)Chrome 43: HW acceleration for Video works (using MediaPlayer)Chrome 53+: HW acceleration for Video does NOT work (we think because MediaPlayer is no longer used)

[Dale Curtis]

You can see how we select codecs here:

https://cs.chromium.org/chromium/src/media/base/android/java/src/org/chromium/media/MediaCodecUtil.java

We don't ship software decoders for h264 on Android, we only use MediaCodec. I would take a look at the code I linked and see which codec is being selected. We typically choose the first codec in the list if I recall correctly.

- dale

On Mon, Oct 16, 2017 at 9:53 AM, <akim...@gmail.com> wrote:

Hi Dale,

I believe we've faced the same issue. We are trying to to upgrade build-in webview in AOSP.
We are using Android 6.0. CPU consumption  during playing h264 encoded video became very high after upgrading to latest webview. We are sure that corresponding MediaCodec for h264 exists and it worked fine with old webview version.

Could you please help us with the following:
- Is it possible to check which codec exactly is used (I see nothing related in logcat during youtube playing for instance)? How could we do it?

 If you add some logging in that code you'll be able to see which codec is selected.

 

- I see my 'hardware-accelerated' decoder in MediaCodecInfo.getCodecInfos(). Does it mean that webview also see it and should decode corresponding video using it? How does webview/chromium understand whether decoder is hardware accelerated or not?

For the case of H264 we don't care since we don't have software codecs available to use instead. We just use whatever MediaCodec allows us.

 

- I do understand that this is very device specific question but could you please help us with understanding what is expected load on CPU during video decoding (very roughly)? For instance we want to play 1080p h264 decoded video on 4-core arm64 processor. What CPU load should we expect in case of software/hardware decoding?

I don't know, but if I had to guess I'd expect moderate (10-30%) load for 1080p material using hardware and high (60-80%) for software decoding at that resolution; more if it's high-frame-rate material.

[akim...@gmail.com]

Hi Dale,

Thank you for your reply.

We did some further investigation. We compare video stream playing in webview vs ExoPlayer (https://github.com/google/ExoPlayer). Just one small page opened in webview with js video player. We play exactly the same video stream on both.

ExoPlayer consumes ~10-15% CPU
Webview consumes ~35-40% CPU

I did some CPU sampling on both.
Below are trace files from Android Device Monitor.
Exoplayer trace file: https://drive.google.com/open?id=0B6SW5uU1O4vRcWF5NDBjZDFnTms
Webview trace file: https://drive.google.com/open?id=0B6SW5uU1O4vRWU1LUFJNTEZ4TGc

In ExoPlayer just one thread did all heavy work.
In Webview at least three thread did much work. 'main' thread mostly does rendering work and there are two other threads that does some work with MediaCodec. One of them anonymous thread (Thread-135 in my case) and another is 'Chrome_InProcGpuThread'.
After that I checked and note that exoplayer doesn't use GPU at all during video playing while webview loads GPU very much.

So my questions are:
- Could you confirm that I have an issue and that much difference in CPU load with ExoPlayer is not expected?
- Why do I have two threads doing the same (or at least very similar) tasks? Is this expected?
- Why do I need a GPU at all during video playing? Is there a way to disable it?

I'll appreciate any help with my issue.
Thanks in advance.

Regards,
Sergey Akimov



понедельник, 16 октября 2017 г., 20:03:16 UTC+3 пользователь Dale Curtis написал:

[Dale Curtis]

WebView is limited to using SurfaceTexture due to the compositing model; this means the GPU is required to composite and display frames (even in fullscreen). As such it's expected it's going to use more processing than a pure video player like ExoPlayer. Which has the advantage of being "just" a video player and the ability to use things like SurfaceView without worry about how web things like scrolling and css (and more!) might break.

Your three threads are probably media (demuxing, audio decoding, video decode control channel), gpu (compositing), mediacodec (decoding) and in Chrome proper all exist in different processes (hence the inability to reuse).

You might compare Chrome proper in fullscreen to see what the lowest possible usage looks like. I don't expect we can ever be as efficient as dedicated video player applications, but in fullscreen with a SurfaceView our testing has shown the differences are minor. We are also working to expand the number of cases in which we can use SurfaceView as well as where we can bypass large portions of the compositor even with SurfaceTexture. You can see those projects here:

https://bugs.chromium.org/p/chromium/issues/detail?id=618368

https://bugs.chromium.org/p/chromium/issues/detail?id=746182

- dale

[akim...@gmail.com]

Hi Dale,

Could you please suggest any hints we can apply now to achieve best possible performance. All we need is just play fullscreen video. So may be there is some fine tuning just for this case? As I understand we should use SurfaceView. Maybe something else we should try?

Sergey

вторник, 17 октября 2017 г., 20:00:20 UTC+3 пользователь Dale Curtis написал:

[Dale Curtis]

Using SurfaceView (SV) is the only optimization that I think you could try aside from the two work in progress ones I've mentioned. I'm not sure it's possible to use a SV for media playback with WebView though, so you may need to look at customizing Chrome's content_shell if Chrome proper is not an option for you.

+liberato, any thoughts on how one might use SV for WebView?

- dale

[Bo Liu]

Getting SV to work in webview to get all the benefits is going to be fairly big and complicated project.

SV is never going to work perfectly because webview is not actually in control of its own layout, and that includes fullscreen. Ans since surface positioning and updates cannot be synchronized, there will always be possibility of glitches. This problem isn't solvable in chromium code, and it's always going to be an best effort/hope for the best kind of thing.

The other problem, especially for saving power/cpu time is that the optimization of "skip updating the web contents surface if only video changed" just doesn't work in webview. And the cost of updating 2 surfaces at 60fps (or however frequently video updates) might very well be more than the current thing of a gpu composite of the video, and then update only one surface. I don't have a good solution for this in the current set up, but I hear video might be being pulled out into its own vis::surface? We might be able to do something special for webview to avoid invalidating webview itself in that case, but again, I imagine it's super complicated...

[Frank Liberato]

> any thoughts on how one might use SV for WebView?

Unfortunately, there's no way to enable it.  It's been off since, i think, about ~2 years ago.  I made no attempt to re-activate it in crbug.com/618368. As Bo mentions, it's probably never going to be on by default for lack of robustness.  At best, we might be able to enable it if the WebView client specifically allows it.

+1 comparing to a trace of chrome to make sure that the best case is still as good as expected.  Also, one might try Chrome Custom Tabs.

thanks

-fl

On Wed, Oct 18, 2017 at 10:29 AM, Dale Curtis <dalec...@chromium.org> wrote: