How to use a VPU for video decoding acceleration

[rebilla...@gmail.com]

Hi all,

I'm using chromium with QT 5.4 (qtwebengine) on an arm embedded linux device.
This is working quite fine, I can launch a browser and navigate the web.

Nevertheless, my project goal is to play video using MSE and EME APIs but the video performance is not optimal.
This is to be expected as the video decoding is done in software using FFMPEG.
So, because the chipset has a VPU, i'd like to use it in order to improve decoding performance.

I started to implement a media::VideoDecoder sub-class to replace the default one based on FFMPEG.
My problem is that I can't output in a planar YUV format which are the only ones supported (as I understood from the code).
By default the VPU output is NV12 and with help of the hardware post-processor I can output various RGB formats and other semi-planar YUV but no planar YUV.

Therefore I have two questions :
- Is it correct that only planar YUV formats are allowed for a video frame output ?
- I've seen that there is a media::VideoDecodeAccelerator class: Is it feasible to implement a subclass that could be used to decode frames and output in NV12 or RGB ?

Thank you in advance for your answers.

Aubin

[Andrew Scherkus]

Accelerated video decode is typically handled via implementing a media::VideoDecodeAccelerator that runs in the GPU process.

Depending on your architecture, you can either use the opaque video format type NATIVE_TEXTURE or use one of the existing outputs. The compositor handles the video formats inside of VideoResourceUpdater:

https://code.google.com/p/chromium/codesearch#chromium/src/cc/resources/video_resource_updater.cc&l=138

Hope that helps,

Andrew

[Aubin REBILLAT]

Hi Andrew,

Thank you for your anwer, it is indeed helpful.

I looked into the NATIVE_TEXTURE format and it is, i think, a good solution.

I don't know if you can help me, but i have some question about it :

- To create a NATIVE_TEXTURE media::VideoFrame, i have to generate a texture containing the RGBA pixels returned by the VPU and assign it to a mailbox. Is this correct ?

- From what i understand, the VideoDecoder is in a different thread than the render thread, therefore how can i access a gpu::gles2::GLES2Implementation in the videoDecoder ?

Thank you for your answer anyway.

Aubin

[Andrew Scherkus]

My memory is foggy, but yes I believe generating textures via ProvidePictureBuffers() should be sufficient as GpuVideoDecoder will take care of wrapping said textures in a NATIVE_TEXTURE.

Ideally, you shouldn't have to worry about VideoDecoder as the texture should get passed along down to the compositor via existing code.

Andrew

[Aubin REBILLAT]

Oh ok, I did not understand that i should use the NATIVE_TEXTURE format with a VideoDecodeAccelerator.

I was trying to use it inside a VideoDecoder, which was quite difficult to do.

So i implemented a VideoDecodeAccelerator and it is indeed easier.

I am now facing openGL issues, but i'm pretty sure i have some mistakes in my code.

Thank you very much for your help.

[kenz kiran]

Aubin,

I found the best way to move from FFMPEG (software video decoding) to HW Accelerated Video decoding (Software doesn't create, manipulate video frames) was using StreamTextures. Check out how Android Renders HTML Video.

Basically, if you have a GPU that supports  EGLImageExternal extension (https://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external.txt). and a VPU splitting out any YUV format, then you would get a zero-copy video rendering.

Chrome already has the necessary fragment shaders for video textures provided in NV12 format. (if I remember correctly).

If it is of any help, here is a diagram for chrome android html5 video (albeit old by an year, things could have changed).

regards,
Ravi

To unsubscribe from this group and stop receiving emails from it, send an email to graphics-dev...@chromium.org.

[Aubin REBILLAT]

Hi Kenz,

I managed to make my VideoDecodeAccelerator work correctly so for now i will not attempt another method.

Nevertheless, this is really interresting and i will take a closer look at it in the near future.

Thank you for your help.

[Andrew Scherkus]

Glad to hear you got it working!

Andrew

[abhiji...@gmail.com]

On Friday, 9 January 2015 22:09:20 UTC+5:30, Aubin REBILLAT wrote:
> Hi Kenz,
>
>
> I managed to make my VideoDecodeAccelerator work correctly so for now i will not attempt another method.
> Nevertheless, this is really interresting and i will take a closer look at it in the near future.
>
>
> Thank you for your help.
>
>
> On Thu, Jan 8, 2015 at 7:33 PM, kenz kiran <kira...@gmail.com> wrote:
>
>
>
> Aubin,
>
> I found the best way to move from FFMPEG (software video decoding) to HW Accelerated Video decoding (Software doesn't create, manipulate video frames) was using StreamTextures. Check out how Android Renders HTML Video.
>
>
> Basically, if you have a GPU that supports  EGLImageExternal extension (https://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external.txt). and a VPU splitting out any YUV format, then you would get a zero-copy video rendering.
>
>
> Chrome already has the necessary fragment shaders for video textures provided in NV12 format. (if I remember correctly).
>
>
> If it is of any help, here is a diagram for chrome android html5 video (albeit old by an year, things could have changed).
>
>
>
>
>

Hi Aubin,

I am trying to hardware accelerate video decoding for chromium browser. As far I know there are three components in Chromium for Video rendering,
1. Pipeline
2. ffmpeg
3. Webkit

For hardware acceleration, I am contemplating to change ffmpeg to use VPU. Can you please let me know what are steps that you carried out for hardware acceleration? Does ffmpeg expects standard APIs from underlying decoder?

[Dale Curtis]

You won't be able to use ffmpeg for hardware acceleration unless you're running without a sandbox (not recommended). Instead you'll want to look at implementing a MojoVideoDecoder; you can see an example of this being done on Android with MediaCodecVideoDecoder:

https://cs.chromium.org/search/?q=mediacodecvideodecoder+file:media/&sq=package:chromium&type=cs

- dale

[abhiji...@gmail.com]

Hi Dale,

Thank you very much for your reply.

Actually I am trying to use stand-alone chromium browser on Linux. However I checked that, there is media/gpu/v4l2 directory present under Chromium browser source.

Can I use gpu/v4l2 for hardware acceleration of Video Codecs?

[Dale Curtis]

Ah, that is not formally supported in Chrome, but some folks maintain a patch here you can check out: https://chromium-review.googlesource.com/c/chromium/src/+/532294

- dale