Using build_ffmpeg.py to build ffmpeg for non-ChromeOS ARM (Raspberry Pi)

[Gus]

Hi all - very new to the project, forgive (and correct) any errors in my thinking! 

I'm currently working on optimizing Chrome Remote Desktop for the Raspberry Pi. One of the first things I'm doing is making sure ffmpeg is compiled in such a way that it takes advantage of the Pi's hardware encoder. Through much searching, I've discovered build_ffmpeg.py. I'm trying to use it now. 

First, according to the instructions in the file itself, I run:

cros chrome-sdk --board=arm-generic

and then:

./build_ffmpeg.py linux arm-neon

however, this gives me the error:

armv7a-cros-linux-gnueabi-gcc is unable to create an executable file.

C compiler test failed.

Searching the log, I see a bunch of errors of this form:

WARNING: armv7a-cros-linux-gnueabi-pkg-config not found, library detection may fail.

mktemp -u XXXXXX

B3TJTn

check_cpp_condition stddef.h defined __ARM_ARCH_4__ || defined __TARGET_ARCH_4

check_cpp

BEGIN /tmp/ffconf.pRkB2H55.c

    1   #include <stddef.h>

    2   #if !(defined __ARM_ARCH_4__ || defined __TARGET_ARCH_4)

    3   #error "unsatisfied condition: defined __ARM_ARCH_4__ || defined __TARGET_ARCH_4"

    4   #endif

END /tmp/ffconf.pRkB2H55.c

armv7a-cros-linux-gnueabi-gcc -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -fno-split-dwarf-inlining -Wno-unknown-warning-option -Wno-inline-asm -I/usr/local/google/home/gusss/chromium/src/third_party/opus/src/include -march=armv7-a -mtune=cortex-a8 -mfloat-abi=hard -mfpu=neon -E -o /tmp/ffconf.hUzNR5M8.o /tmp/ffconf.pRkB2H55.c

armv7a-cros-linux-gnueabi-gcc.real.elf: error: unrecognized command line option '-fno-split-dwarf-inlining'

I'm honestly not sure I'm on the right track in the first place. Should I be using the "cros" command? From what I can tell, this is used for setting up a ChromeOS development environment; however, I'm not doing anything involving ChromeOS. It seems the script is assuming that anyone trying to compile for ARM must be doing so for a ChromeOS setup?

I've just recently pulled, so everything should be up-to-date.

Can anyone give me some pointers as to what might be wrong?

[Dale Curtis]

That script is specifically for configuring decoders+demuxers-only. I don't think anyone has ever tried to use it for configuring encoders. We've never even checked the hardware decoders/encoders since they can't be run from within the renderer sandbox.

I suspect a lot will not work. How come you're looking at the ffmpeg encoders instead of the media::VideoEncodeAccelerator framework?

- dale

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/bfd05631-2a2d-4bb0-a54e-cd7860552d31%40chromium.org.

[Gus]

Dale - thanks for the reply.

I wasn't looking at the media::VideoEncodeAccelerator framework because I didn't know about it - so thank you for informing me! I'll take a look at this now! Like I said, completely new to the project, so I've been correcting these gaps in my knowledge left and right. 

Thanks again for the help.

[Gus]

Dale, a follow-up after reading more into the VEA framework and re-reading your reply:

My plan was to use ffmpeg as:

1. It has built-in support for the hardware acceleration on Raspberry Pi (if it's compiled with the right flags)
2. I knew Chromium already depended on ffmpeg.

If ffmpeg already has built-in support for H264 encode on Raspberry Pi, I assumed it would be prudent to use it.  

[Dale Curtis]

That depends on which part of the remoting library you're working on. If you're aiming for something to be part of shipping Chrome, relying on ffmpeg for encoding like this is won't be allowed due to sandbox requirements.

Additionally, adding extra ffmpeg capabilities is quite painful since we'd need to maintain those build configs every time we update ffmpeg (~ every milestone). So we'd be reluctant to take those patches into Chrome's ffmpeg repo.

If you're working on the remoting client, you might be able to make this work, but I'm not very familiar with the client's security concerns and architecture used today. How does the client handle hardware encode on other platforms today?

- dale

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/126effec-5912-466b-a6cd-281b11a893b0%40chromium.org.

[Gus]

I'm working exclusively on the CRD host, which encodes frames of the desktop to send to the client. However, to my knowledge (Sergey should be able to confirm this), the host code isn't shipping in the Chrome binary. Not sure how this affects whether or not it falls under these sandbox requirements. (By the way - what do you mean by "sandbox requirements"? Something involving testing?)

As for hardware encoding on other platforms - Sergey should also be able to answer this. I don't know of us targeting something as specific as the RPi's Broadcom chip, but perhaps we handle hardware encode on GPU or something similar. 

[Dale Curtis]

On Mon, Jun 12, 2017 at 12:25 PM, 'Gus' via Chromium-dev <chromi...@chromium.org> wrote:

I'm working exclusively on the CRD host, which encodes frames of the desktop to send to the client. However, to my knowledge (Sergey should be able to confirm this), the host code isn't shipping in the Chrome binary. Not sure how this affects whether or not it falls under these sandbox requirements. (By the way - what do you mean by "sandbox requirements"? Something involving testing?) 

I'm referring to Chrome's security model, the renderer process in particular where we run ffmpeg normally is locked down to prevent the type of access that hardware decoding might need there. Probably the CRD host has a differnent model. See:

https://chromium.googlesource.com/chromium/src/+/master/docs/design/sandbox.md

https://chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md

 

As for hardware encoding on other platforms - Sergey should also be able to answer this. I don't know of us targeting something as specific as the RPi's Broadcom chip, but perhaps we handle hardware encode on GPU or something similar. 

If RPi has a V4L2 interface you can probably reuse a lot of the VideoEncodeAccelerator architecture that we use for CrOS.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/bbe98c5d-db6f-4f45-ac3b-2639ce445a2a%40chromium.org.

[Sergey Ulanov]

On Mon, Jun 12, 2017 at 12:41 PM, Dale Curtis <dalec...@chromium.org> wrote:

On Mon, Jun 12, 2017 at 12:25 PM, 'Gus' via Chromium-dev <chromi...@chromium.org> wrote:

I'm working exclusively on the CRD host, which encodes frames of the desktop to send to the client. However, to my knowledge (Sergey should be able to confirm this), the host code isn't shipping in the Chrome binary. Not sure how this affects whether or not it falls under these sandbox requirements. (By the way - what do you mean by "sandbox requirements"? Something involving testing?) 

I'm referring to Chrome's security model, the renderer process in particular where we run ffmpeg normally is locked down to prevent the type of access that hardware decoding might need there. Probably the CRD host has a differnent model. See:

https://chromium.googlesource.com/chromium/src/+/master/docs/design/sandbox.md

https://chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md

 

CRD host ships separately from chrome. The encoder (currently it's always libvpx) is not sandboxed there, but the content being encoded is generated locally so security vulnerabilities in the encoder are not a concern. 

[Dale Curtis]

Are you encoding vp9 lossless? Otherwise, it seems like integrating media::VideoEncodeAccelerator might be a win on all platforms then?

- dale

[Gus]

(@dale, see Sergey's email for a reply to that question)

A comment about what I learned today/the perceived decision I'm going to have to make:

You (or anybody) may or may not have any comments about this, but from my research it looks like my options at this point are:

• Use an ffmpeg library compiled for RPi encoding acceleration and package it with the host. I'm not sure about the viability of this with regards to packaging the library with the host, or with regards to licensing, etc etc.
• Use OpenMAX directly (which is what ffmpeg uses in the first place). The OpenMAX libraries enabling RPi encoding acceleration come packaged with the RPi firmware.

[Gus]

An additional comment: it seems there is no support for VA-API on Raspberry Pi, at least at the moment. OpenMAX IL seems to be the API that's available and implemented. I know they're not necessarily equivalent APIs - as I understand it, the scope of OpenMAX IL is broader than that of VA-API. However, perhaps an analogous media::VideoEncodeAccelerator could/should be implemented for OpenMAX, just as there is one implemented for VA-API (as Dale pointed out). 

Of course, then we might have to transition some of the CRD code to using this VideoEncodeAccelerator framework; however, as Dale pointed out there may be benefits to this. Alternatively, I can implement an encoder using the CRD VideoEncoder class.

Does this sound logical? Can anyone chime in with any more info regarding OpenMAX/VA-API, or include anyone on the discussion who might have an opinon? 

[Dale Curtis]

-internal mailing list, don't mix and match public/private lists - tends to be confusing.

Given the licensing and long term benefits of using media::VideoEncodeAccelerator on other platforms it seems reasonable to pursue to this approach. I don't know anything about OMAX so I'm not sure how hard it would be to write a VDA on top of it, but it does seem to be the most maintainable way forward.

- dale

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e779d6a8-2487-4294-a9be-9b4bda393877%40chromium.org.

[Gus Smith]

Makes sense - my bad. 

Looking at the equivalent code for the VA-API VDA, it looks like it would not be an insignificant task, plus would require a host of other changes. However, as you say, it would help with maintainability. Probably best to also hear the opinions of the CRD team at this point.