HW accelerated playback of H.264 on Linux through OpenMax, system FFMPEG or plugins

[nicolash...@gmail.com]

I've been looking for a few days on how to get hardware accelerated video to display in Chrome on linux. I'm developing an application which will run inside the browser and sometimes needs to display heavy 1080p videos (I can choose the encoder used). The devices are usually intel Atoms so they are too low-powered for smooth playback through software.

After enquiring on the IRC channel, looking at the bug tracker and searching for terms such as "ffmpeg" and "omx" I'm still not sure what would be the most fruitful thing for me to do at the moment.

Hardware platform: Intel HD 3000 and 4000 chips

Some ideas that I've had:

1) Extend chromium source myself => I know C++ but have no experience with neither Chromium nor ffmpeg nor OMX nor VA-API so I assume that's a very steep learning curve and I don't think I have sufficient time for that allotted to this project

2) Use a bit of the work that has been done for ChromeOS HW-accelerated support. Would it reasonably work if I could get OpenMax working on debian 7?

3) Compile chromium with use_system_ffmpeg. I'm not sure if this will help as the posts I've read seem to hint that whatever ffmpeg (system or internal) is used, it is sandboxed and cannot reach the GPU. This will sound stupid but is there a way I can disable the sandbox? Probably not but I'm running on a closed platform so it doesn't need to be secure in that regard.

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware? Are plugins sandboxed in the same way? Can that be disabled? Is this a good idea?

5) Compile own webkit and try to shoehorn playing via system ffmpeg decoded frames in.

Some are of course far more far flung than others (especially the own webkit browser), but at this point I'm willing to consider everything. What do you guys think would be the most fruitful way to spend my time trying to get this to work?

I know the chromium devs are busy with making the web a better place for all and you probably won't be able to answer every question in my mail, but I would appreciate even a "it's not going to happen, look for something else".

Thanks in advance,

Nicolas

P.S.: I have these mailing list posts bookmarked, descending date of OP:

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/ffmpeg/chromium-dev/o0qIVudjoSI/FIC2bg1IPZsJ (about using system mesa, thought I'd flag it just in case)

https://groups.google.com/a/chromium.org/forum/?fromgroups#!searchin/chromium-dev/ffmpeg/chromium-dev/VL2Hb5W581E/n4q_VDG4i7sJ (about ffvp8 being used for decodes)

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/omx/chromium-dev/W1Z-sAGQin4/m3kqG5TrdV0J

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/omx/chromium-dev/lz0WdNpDxcE/X7gOSL23hWkJ

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/ffmpeg/chromium-dev/Kf7LEQEPu-E/_OJAU6buqwgJ

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/ffmpeg/chromium-dev/C4ZyD8MSE20/LeMn93zrhhcJ

[Stuart Morgan]

I can't speak to the rest, but:

2013/2/19 <nicolash...@gmail.com>

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware?

NPAPI plugins can run arbitrary native code, so if the OS allows hardware decoding, then NPAPI plugins can do it.

 

Are plugins sandboxed in the same way?

NPAPI plugins aren't sandboxed at all.

 

Is this a good idea?

If there's an existing NPAPI plugin that does what you want and that you trust, then it would be easy to use. Whether it's a good idea or not depends a lot on the details of what you want, and how well that matches existing plugins, so you'd have to play with them and see.

-Stuart

[Ami Fischman]

W1Z-sAGQin4 is the thread you should most pay attention to, and bug 137247 is the bug that probably contains the most pointers.

Your best bet is to get VA-API drivers/libs working on your target platform and then enabling VAVDA in your build of chromium.  Make sure you set the GYP_DEFINES variables to enable h264 decoding:

proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome

Note that this is an entirely unsupported/untested configuration and you likely *will* need to use a debugger to figure out what's going wrong until it all goes right (and that even once it works, it may well break in the future as nobody else will be building your config).

 

2) Use a bit of the work that has been done for ChromeOS HW-accelerated support. Would it reasonably work if I could get OpenMax working on debian 7?

I don't believe OMX-IL implementations exist for x86/intel hardware.

Intel's horse is VA-API, which is why I recommend VAVDA above.

 

3) Compile chromium with use_system_ffmpeg.

As you've guessed, this will be useless to you - the system ffmpeg still runs inside the renderer.

You can disable the sandbox, but that's a bad idea, and I don't know if even that would make the system ffmpeg use any HW accel (I've never tried).

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware? Are plugins sandboxed in the same way? Can that be disabled? Is this a good idea?

NPAPI plugins are not sandboxed, and it is a terrible idea to use them for that reason.

 

5) Compile own webkit and try to shoehorn playing via system ffmpeg decoded frames in.

This combines the weaknesses of your #1 & #3 :)

Good luck,

-a

[nicolash...@gmail.com]

Good evening,

First of all sincerely thanks for your replies, I'm glad someone took notice

W1Z-sAGQin4 is the thread you should most pay attention to, and bug 137247 is the bug that probably contains the most pointers.

Yep, I already had that bug starred, very interesting. The "vibe" I got from this is that it is extremely experimental and would be pretty unstable.

 

Your best bet is to get VA-API drivers/libs working on your target platform and then enabling VAVDA in your build of chromium.  Make sure you set the GYP_DEFINES variables to enable h264 decoding:

proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome

Thanks for the pointer, I will try a custom compile on my debian 7 box. The very least I can do is try.

 

Note that this is an entirely unsupported/untested configuration and you likely *will* need to use a debugger to figure out what's going wrong until it all goes right (and that even once it works, it may well break in the future as nobody else will be building your config).

It indeed seemed to me that VAVDA on Chrome for linux is still heavily in testing. Yet it also seems to be exactly what I want. Sandboxed, accelerated display of high-def content.

 

2) Use a bit of the work that has been done for ChromeOS HW-accelerated support. Would it reasonably work if I could get OpenMax working on debian 7?

I don't believe OMX-IL implementations exist for x86/intel hardware.

Intel's horse is VA-API, which is why I recommend VAVDA above.

 

I was also really hoping to be able to use VA-API, the OMX-IL business was more me looking for a way out, listing all the possible paths so to say.

 

3) Compile chromium with use_system_ffmpeg.

As you've guessed, this will be useless to you - the system ffmpeg still runs inside the renderer.

You can disable the sandbox, but that's a bad idea, and I don't know if even that would make the system ffmpeg use any HW accel (I've never tried).

That's actually interesting. In this comment and in a later comment you refer to the badness of running outside of the sandbox. In theory I fully agree with you, but if it is what is necessary to get some HW acceleration for my videos, it doesn't seem like the worst thing in the world. Are you more concerned about loss of security or crash-proneness? Because I can take the lack of security, I will only display local content.

 

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware? Are plugins sandboxed in the same way? Can that be disabled? Is this a good idea?

NPAPI plugins are not sandboxed, and it is a terrible idea to use them for that reason.

 

Security, crashing or both?

 

5) Compile own webkit and try to shoehorn playing via system ffmpeg decoded frames in.

This combines the weaknesses of your #1 & #3 :)

Hehe, that does sound bad :). Ideally I would like to stay with chrome indeed.

 

Good luck,

-a

Thanks! Currently I'm still eyeing up VAVDA, NPAPI plugins and a custom webkit derivate such as luakit (or other minimal browser equivalents, they all seem to rely on webkitgtk).

Kind regards,

Nicolas

[Ami Fischman]

Note that this is an entirely unsupported/untested configuration and you likely *will* need to use a debugger to figure out what's going wrong until it all goes right (and that even once it works, it may well break in the future as nobody else will be building your config).

It indeed seemed to me that VAVDA on Chrome for linux is still heavily in testing.

Not really.  VAVDA on CrOS is shipping / production quality.  VAVDA on Linux is not even a little bit in testing :)

 

 

3) Compile chromium with use_system_ffmpeg.

As you've guessed, this will be useless to you - the system ffmpeg still runs inside the renderer.

You can disable the sandbox, but that's a bad idea, and I don't know if even that would make the system ffmpeg use any HW accel (I've never tried).

That's actually interesting. In this comment and in a later comment you refer to the badness of running outside of the sandbox. In theory I fully agree with you, but if it is what is necessary to get some HW acceleration for my videos, it doesn't seem like the worst thing in the world. Are you more concerned about loss of security or crash-proneness? Because I can take the lack of security, I will only display local content.

I'm concerned about security, and about the odds that the thing you build to only play local content will eventually get applied to untrusted content ;)

If all you need is a local media player, you might be better off using a media player application instead of a web browser...

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware? Are plugins sandboxed in the same way? Can that be disabled? Is this a good idea?

NPAPI plugins are not sandboxed, and it is a terrible idea to use them for that reason.

Security, crashing or both?

Security.

-a 

[nicolash...@gmail.com]

Note that this is an entirely unsupported/untested configuration and you likely *will* need to use a debugger to figure out what's going wrong until it all goes right (and that even once it works, it may well break in the future as nobody else will be building your config).

It indeed seemed to me that VAVDA on Chrome for linux is still heavily in testing.

Not really.  VAVDA on CrOS is shipping / production quality.  VAVDA on Linux is not even a little bit in testing :)

 

The CrOS part of your sentence gives me more hope while the linux part of your sentence gives me less hope. I guess it cancels out :). Not sure how much infrastructure CrOS shares with most linux distributions (does it even use normal mesa? x.org? Does it use the same drivers?). But by the fact that its almost completely untested (someone tried it once) but still worked I deduce that there is a decent overlap. I might strike lucky gold.

 

 

3) Compile chromium with use_system_ffmpeg.

As you've guessed, this will be useless to you - the system ffmpeg still runs inside the renderer.

You can disable the sandbox, but that's a bad idea, and I don't know if even that would make the system ffmpeg use any HW accel (I've never tried).

That's actually interesting. In this comment and in a later comment you refer to the badness of running outside of the sandbox. In theory I fully agree with you, but if it is what is necessary to get some HW acceleration for my videos, it doesn't seem like the worst thing in the world. Are you more concerned about loss of security or crash-proneness? Because I can take the lack of security, I will only display local content.

I'm concerned about security, and about the odds that the thing you build to only play local content will eventually get applied to untrusted content ;)

If all you need is a local media player, you might be better off using a media player application instead of a web browser...

I can basically guarantee that only local content will be played the first year. And I would almost put my hand in the fire for the following years but in the case non-local content gets played by then, I hope there will be HW acceleration via VAVDA on linux at that time. Lord knows I would love to switch over to such a vastly simpler solution (less moving parts = good). That said, I can't really use a media player (I think) because it is some kind of loop in which many things can appear: hi-def video, newstickers displayed with HTML/CSS, presentations that make heavy use of modern JS, ... Switching between these items in a list needs to be smooth as butter, there can be no flickering. Truthfully, I have had crazy thoughts of configuring a window manager to overlay a media player over the browser when hi-def video passes, but every time I think of it I throw it in the bin because of its inelegance and maintenance nightmaryness. That said, I might have to do it.

However I don't much see the difference between a VLC plugin and a VLC overlay in security except that the overlay is even more of a hackjob.

 

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware? Are plugins sandboxed in the same way? Can that be disabled? Is this a good idea?

NPAPI plugins are not sandboxed, and it is a terrible idea to use them for that reason.

Security, crashing or both?

Security.

-a 

Kind of good to know. Currently only vetted and/or self-produced content gets on there. I know the risks and luckily make the calls about that. Hopefully sandboxing can soon be a reality though.

One more thing, if you happen to know this, in which version did libvpx decoding get switched out for ffvp8 decoding? I'm having a hard time tracking it down, probably because of my weak search-fu. I'm asking because we're currently staying on an older version of chrome for stability reasons but maybe we should switch to webm and see if maybe a dual-core atom can in fact play 1080p without stuttering.

Already thanks for the great answers, nothing better and more assuring than words from the powers that be. :)

Kind regards,

Nicolas

[Ami Fischman]

It indeed seemed to me that VAVDA on Chrome for linux is still heavily in testing.

Not really.  VAVDA on CrOS is shipping / production quality.  VAVDA on Linux is not even a little bit in testing :)

The CrOS part of your sentence gives me more hope while the linux part of your sentence gives me less hope.  I guess it cancels out :).

I just wanted to emphasize that VAVDA is the more attended-to VDA, compared to the OMX one (which you had mentioned earlier, and which would be a poorer fit for an intel platform).

 

One more thing, if you happen to know this, in which version did libvpx decoding get switched out for ffvp8 decoding?

I don't know off the top of my head, but it's been at least a year.

I'm asking because we're currently staying on an older version of chrome for stability reasons

You'll have to do some crbug/logs spelunking to figure out when VAVDA launched, but if it was me I'd be working off ToT - it's likely that bugfixes have only been merged back as far as necessary and no farther.

 

-a

[Frank Barchard]

My 2 cents...

On Tue, Feb 19, 2013 at 7:02 AM, <nicolash...@gmail.com> wrote:

I've been looking for a few days on how to get hardware accelerated video to display in Chrome on linux.

Preferred approach is OpenMac for linux.

 

I'm developing an application which will run inside the browser and sometimes needs to display heavy 1080p videos (I can choose the encoder used). The devices are usually intel Atoms so they are too low-powered for smooth playback through software.

After enquiring on the IRC channel, looking at the bug tracker and searching for terms such as "ffmpeg" and "omx" I'm still not sure what would be the most fruitful thing for me to do at the moment.

Hardware platform: Intel HD 3000 and 4000 chips

Atom has GMA, not Intel HD.

You're better off with external nVidia Ion for Atom. 

Some ideas that I've had:

1) Extend chromium source myself => I know C++ but have no experience with neither Chromium nor ffmpeg nor OMX nor VA-API so I assume that's a very steep learning curve and I don't think I have sufficient time for that allotted to this project

None of these work within the sandbox.

 

2) Use a bit of the work that has been done for ChromeOS HW-accelerated support. Would it reasonably work if I could get OpenMax working on debian 7?

Yes

 

3) Compile chromium with use_system_ffmpeg. I'm not sure if this will help as the posts I've read seem to hint that whatever ffmpeg (system or internal) is used, it is sandboxed and cannot reach the GPU. This will sound stupid but is there a way I can disable the sandbox? Probably not but I'm running on a closed platform so it doesn't need to be secure in that regard.

Sandbox will prevent this.   Same as (1). Historically we tested outside the sandbox, but you couldnt ship code like this. 

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware? Are plugins sandboxed in the same way? Can that be disabled? Is this a good idea?

Short term will work with NPAPI or ActiveX, but not Pepper.

 

5) Compile own webkit and try to shoehorn playing via system ffmpeg decoded frames in.

Safari/Quicktime and Netflix work this way.

Some are of course far more far flung than others (especially the own webkit browser), but at this point I'm willing to consider everything. What do you guys think would be the most fruitful way to spend my time trying to get this to work?

OpenMax or CPU.

 

I know the chromium devs are busy with making the web a better place for all and you probably won't be able to answer every question in my mail, but I would appreciate even a "it's not going to happen, look for something else".

Chrome has the internals to do HW decode.

Its enabled for Windows publicly.

Making it work for Linux would be the next step.

Thanks in advance,

Nicolas

P.S.: I have these mailing list posts bookmarked, descending date of OP:

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/ffmpeg/chromium-dev/o0qIVudjoSI/FIC2bg1IPZsJ (about using system mesa, thought I'd flag it just in case)

https://groups.google.com/a/chromium.org/forum/?fromgroups#!searchin/chromium-dev/ffmpeg/chromium-dev/VL2Hb5W581E/n4q_VDG4i7sJ (about ffvp8 being used for decodes)

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/omx/chromium-dev/W1Z-sAGQin4/m3kqG5TrdV0J

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/omx/chromium-dev/lz0WdNpDxcE/X7gOSL23hWkJ

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/ffmpeg/chromium-dev/Kf7LEQEPu-E/_OJAU6buqwgJ

https://groups.google.com/a/chromium.org/forum/?fromgroups=#!searchin/chromium-dev/ffmpeg/chromium-dev/C4ZyD8MSE20/LeMn93zrhhcJ

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

[nicolash...@gmail.com]

My 2 cents...

Thanks for chiming in Frank!

 

On Tue, Feb 19, 2013 at 7:02 AM, <nicolash...@gmail.com> wrote:

I've been looking for a few days on how to get hardware accelerated video to display in Chrome on linux.

Preferred approach is OpenMac for linux.

So even for non-chromeos cases? That's nice. I saw some OMX IL (bellagio) packages in the debian wheezy repo. I wonder if I need to do something special to "connect the two"

 

 

I'm developing an application which will run inside the browser and sometimes needs to display heavy 1080p videos (I can choose the encoder used). The devices are usually intel Atoms so they are too low-powered for smooth playback through software.

After enquiring on the IRC channel, looking at the bug tracker and searching for terms such as "ffmpeg" and "omx" I'm still not sure what would be the most fruitful thing for me to do at the moment.

Hardware platform: Intel HD 3000 and 4000 chips

Atom has GMA, not Intel HD.

You're better off with external nVidia Ion for Atom. 

You're completely right of course. We've been testing with about 3 different platforms:

1) Intel HD 3000 + celeron (CPU decoding stutters from time to time)

2) Intel HD 4000 + i3         (works best but will be too expensive for the project)

3) Nvidia ION + Atom        (did not work at all because of slowness until yesterday with mpayer-gecko)

Currently we have mostly platform (1) so I assume that for cost' sake I'll be needing to run it at least on that one

 

Some ideas that I've had:

1) Extend chromium source myself => I know C++ but have no experience with neither Chromium nor ffmpeg nor OMX nor VA-API so I assume that's a very steep learning curve and I don't think I have sufficient time for that allotted to this project

None of these work within the sandbox.

Shoot.

 

 

2) Use a bit of the work that has been done for ChromeOS HW-accelerated support. Would it reasonably work if I could get OpenMax working on debian 7?

Yes

That's encouraging. I suppose that's this OVDA thing I've been reading about, no? I wonder if both Intel and NVIDIA cards can be driven by OpenMax. I got the idea from raspberry pi.

 

3) Compile chromium with use_system_ffmpeg. I'm not sure if this will help as the posts I've read seem to hint that whatever ffmpeg (system or internal) is used, it is sandboxed and cannot reach the GPU. This will sound stupid but is there a way I can disable the sandbox? Probably not but I'm running on a closed platform so it doesn't need to be secure in that regard.

Sandbox will prevent this.   Same as (1). Historically we tested outside the sandbox, but you couldnt ship code like this. 

If it would just take disabling the sandbox, that would be fine for me. Anyway, I tried in mplayer to get ffmpeg to do hardware decode (ffmpeg) and I wasn't very succesful so that's that. I needed to resort to the VDPAU backend.

 

4) Try to procure plugins such as VLC or MPlayer-gecko. Are they allowed to decode on the hardware? Are plugins sandboxed in the same way? Can that be disabled? Is this a good idea?

Short term will work with NPAPI or ActiveX, but not Pepper.

 

I actually tried this yesterday with mplayer-gecko. I chose mplayer because it's the only thing that seems to be able to talk to VDPAU properly. VLC has some issues (probably because it goes through VA-API even for NVIDIA).

 

5) Compile own webkit and try to shoehorn playing via system ffmpeg decoded frames in.

Safari/Quicktime and Netflix work this way.

I keep wishing I won't have to do this though. Maybe I could base work of of luakit or surf or things like that if it comes to that. From their bug-trackers they seem to be not so stable for anything but rote browsing.

 

Some are of course far more far flung than others (especially the own webkit browser), but at this point I'm willing to consider everything. What do you guys think would be the most fruitful way to spend my time trying to get this to work?

OpenMax or CPU.

Since CPU is not that much of an option seeing the budget and hi-def constraints. It'll have to be OpenMax or something else, but at any rate HW-accelerated. 

 

I know the chromium devs are busy with making the web a better place for all and you probably won't be able to answer every question in my mail, but I would appreciate even a "it's not going to happen, look for something else".

Chrome has the internals to do HW decode.

Its enabled for Windows publicly.

Making it work for Linux would be the next step.

Yet windows probably uses something else than OVDA or VAVDA, no? How come this avoids the sandbox at any rate?

 

Thanks a lot for your interest!

[Ami Fischman]

I've been looking for a few days on how to get hardware accelerated video to display in Chrome on linux.

Preferred approach is OpenMac for linux.

So even for non-chromeos cases? 

Frank: why do you say that?   OVDA has seen the least amount of work of all the VDA impls in the last 6-9mos, and OMX is extremely poorly supported / unavailable on most x86 linux platforms compared with VDPAU & VAAPI (and VDPAU can be wrapped to expose a VAAPI layer by standard libs that *are* widely available).

2) Use a bit of the work that has been done for ChromeOS HW-accelerated support. Would it reasonably work if I could get OpenMax working on debian 7?

Yes

That's encouraging. I suppose that's this OVDA thing I've been reading about, no? I wonder if both Intel and NVIDIA cards can be driven by OpenMax. I got the idea from raspberry pi.

Given you're using an x86 platform, VAVDA is going to be a much shorter route to happiness than OVDA.

I know the chromium devs are busy with making the web a better place for all and you probably won't be able to answer every question in my mail, but I would appreciate even a "it's not going to happen, look for something else".

Chrome has the internals to do HW decode.

Its enabled for Windows publicly.

Making it work for Linux would be the next step.

Frank: please see the bug I linked to earlier.  There are no plans currently to "make it work for Linux".

 

Yet windows probably uses something else than OVDA or VAVDA, no? How come this avoids the sandbox at any rate?

Windows uses the same IPC mechanisms to offload decode to the GPU process.  The VDA impl in Chrome/Windows is DXVAVideoDecodeAccelerator.

-a