Captured video with alpha channel

[Dima Kostenich]

Hello guys,

I have several issues to discuss here. First of all, let me specify that our current project works closely with Chromium Embedded Framework (CEF) and Intel RealSense Integrated 3D Camera. While we were trying to integrate the support of this device into Chromium we figured out that:

1) When camera device provides video in ARGB format (like Intel RealSense Camera), Chromium enumerator shows this cam (as ARGB is in the allowed list), but at the same time sink_input_pin_win doesn't want to connect as it doesn't support ARGB. I fixed this issue. Please, see the patch attached.

2) In DirectShow all RGB formats (including ARGB) are flipped. What is the reason not to flip ARGB data in video_capture_device_client.cc in VideoCaptureDeviceClient::OnIncomingCapturedData() method? Flip parameter is set to true for RGB24 and RGB32 there but not for ARGB. So, if we allow captured data in ARGB format as described in #1, we will see the result picture upside down, that's why I also set "flip = true" in "case media::PIXEL_FORMAT_ARGB" in the method mentioned above. The result picture after that looks ok but is it correct fix for all the cases?

3) As we saw in the Chromium code, all captured formats are converted to I420. Why not to convert all RGB-based formats to any single RGB format and all YUV-based formats to any single YUV format (I420, for instance)? RGB to YUV conversion is not much effective on the CPU.

4) One of our tasks with Intel RealSense Camera is to provide video with alpha-channel from camera to the web view in such a way which allows us to display video with real transparency. I managed to pass captured video in the format VideoFrame::ARGB, but all I can see now is just simple video without any transparency. Any advises on how we can do that? I saw that Chromium can display WebM video with transparency (http://simpl.info/videoalpha/). Is it possible to do something like that for the captured video?

[Tima Vaisburd]

Hi,

>3) As we saw in the Chromium code, all captured formats are converted to I420. Why not to convert all RGB-based formats

> to any single RGB format and all YUV-based formats to any single YUV format (I420, for instance)? 

As far as I can say you convert RGB to YUV because you want to _encode_ it later, i.e. convert from YUV to H264, for instance.

You loose alpha on the stage of (A)RGB -> YUV conversion, but even if you did not, I'm not aware of any compressed video format

that preserves alpha.

> just simple video without any transparency. Any advises on how we can do that? 

From http://blender.stackexchange.com/questions/2050/any-video-formats-that-render-with-alpha-channel:

"You can consider using two files - flat video and alpha mask video."

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Vladimir Hmelyoff]

Hi Tima,

Our main purpose is that we have webcam that delivers rgb frames and generates alpha channel and we need to render it in Chrome (CEF).

Regarding video files, we already using webm with alpha transparency support (there no need to have 2 video files) and this works great. So we wonder why videos can be rendered with alpha but cameras can not.

First of all, our webcam listed in Chrome but did not work, so we provided patches how to make it work. But next we see that Chromium doing conversion of any camera format to I420 always. We tried to avoid this conversion (and send ARGB as is) but can not make it to work still. Since we don't know Chromium code so well, maybe some developer have an idea what buffer we must generate for renderer so it will be rendered with alpha transparency.

As side note, if Chromium renders in RGB (we don't know that, just assume), this may be not efficient to convert camera frames to I420 always because in renderer these frames will be converted to RGB back and YUV-RGB conversions is usually not efficient and eats CPU.

Best Regards,

Vladimir Hmelyoff

[PhistucK]

On Tue, Aug 18, 2015 at 10:06 PM, 'Tima Vaisburd' via Chromium-dev <chromi...@chromium.org> wrote:

I'm not aware of any compressed video format

that preserves alpha.

What about ​WebM?​

☆PhistucK

[Tima Vaisburd]

> What about ​WebM?​

Yes, after re-reading the info that I could find I realized I have an incomplete knowledge. Please accept my apology.

It would be nice if somebody from that project could comment.

> if Chromium renders in RGB (we don't know that, just assume), this may be not efficient to convert camera frames to I420

> always because in renderer these frames will be converted to RGB back and YUV-RGB conversions is usually not efficient and eats CPU.

Just to clarify: do you want to play video on the same device that you capture it or on another device over the internet? In the second case

your conversion chain would be 

RGB -> YUV -> VP8 -----> (transmission) -----> VP8 -> YUV -> RGB

and YUV makes perfect sense.

[Vladimir Hmelyoff]

> RGB -> YUV -> VP8 -----> (transmission) -----> VP8 -> YUV -> RGB

Completely agree. Makes sense. But in any case this is not main problem, just side note.

> Just to clarify: do you want to play video on the same device that you capture it or on another device over the internet? 

On the same device. So we load some html page in browser, choose our webcam there and we need to see webcam picture in browser with transparency, no VP8 and no transmission.

If I not mistake, Dima already figured out that YV12A colorspace used for video files (or maybe only for video files with transparency). It is similar to I420 (just different U and V planes order and additional A plane). So it maybe makes sense to try it. But this will be very helpful if somebody who know Chromium project inside can guide what approach will work (and maybe what will not) and what way is best to go. Maybe we trying to solve something that currently not possible at all and it's better to find some other solutions instead.

[Tima Vaisburd]

Vladimir,

I can't help you here, sorry. Maybe someone else with a better knowledge.

It seems you need a camera preview in the web page. A quick search revealed the following links, maybe they are

relevant to your project:

http://www.smartlab.at/articles/html5-camera-and-android/

http://stackoverflow.com/questions/5055962/how-to-create-an-h264-video-with-an-alpha-channel-for-use-with-html5-canvas

[Frank]

Hi Vladimir,

As you know we added support for displaying VP8 webm files with Alpha to Chromium. Here are a couple of the CLs for that work:

https://codereview.chromium.org/12157002/

https://codereview.chromium.org/16580006/

Unfortunately I don't know enough about the video capture Chrome code to be of help. You are probably just going to need to start with the capture code and add support for RGBA and then keep adding more code until it finally renders. Hopefully some of the rendering CLs we added for VP8 can help you.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Dima Kostenich]

Thanks for all the replies!

I've made an experiment today. I converted captured video to YV12A format in VideoCaptureDeviceClient. I mean I changed the order of U and V planes, allocated space for A-plane and filled it with necessary data in VideoFrame. I also marked this VideoFrame as YV12A format and fixed all the intermediate checks to allow this type of video as well as ARGB and I420. What I can see is the fact that my frame comes to VideoCaptureImpl (i.e. to the render process) from browser process with the parameters I set (with A-plane and YV12A format type). Unfortunately, I can't check the rest of rendering pipeline because breakpoints at these levels don't want to work because of yet unknown reason. What I see on my screen is the captured video with wrong colours and without alpha channel. So, I guess, renderer still tries to render captured video as I420 video despite the fact that I set format to YV12A. Can anybody please explain me the rest of the pipeline of the captured video frames? I figured out that they will come to MediaStream module. But what happens to them next?

[Yuri Wiitala]

Sorry I just noticed this thread.  I can help w/ info.  I own tab capture, which uses the video capture pipeline (browser-->renderer).  Though, I've got to run and I'm out tomorrow.  I'll follow-up on this thread Monday.

--