Intent to ship: Media Capture from <audio> and <video>

[Niklas Enbom]

Contact emails

mca...@chromium.org, nik...@chromium.org

Spec

https://www.w3.org/TR/mediacapture-fromelement/

Summary

This API allows WebRTC streaming of prerecorded content. Chrome 52 shipped captureStream support for the canvas element. We're now ready to ship the same functionality for <audio> and <video>. Capture will not work for protected content.

Interoperability and Compatibility Risk

As far as we know this is the first implementation. Firefox has support for canvas.captureStream

Requesting approval to ship?

Yes, targeting M54.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Tracking bugs:

https://bugs.chromium.org/p/chromium/issues/detail?id=569976

https://bugs.chromium.org/p/chromium/issues/detail?id=575492

Demo page:

https://rawgit.com/Miguelao/demos/master/videoelementcapture.html

[Chris Harrelson]

LGTM1, with questions below to confirm.

I think the spec link is a bit off. The spec is section 3 in the doc, not section 4 right? I assume there is public support from other vendors? Has this gone through a TAG review?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Niklas Enbom]

Correct, the link should be to section 3 and not 4. There doesn't seem to be a TAG review done, I've asked the editor and chairs about this, will get back on this thread.

[Philip Jägenstedt]

captureStreamUntilEnded() is commented out in HTMLMediaElementCapture.idl so I guess this intent is only for captureStream()?

The Intent to Implement for this was here:

https://groups.google.com/a/chromium.org/d/msg/blink-dev/ZuseXCSZY2U/p1gLNcCMAgAJ

At the time I filed four spec issues, but all of them remain open without feedback from anyone working on the spec. Has someone done triage on the open spec issues to see which ones might result in spec changes and if the Chromium implementation could provide answers?

I'm a bit concerned about how vague the spec is on many details. For example: "A captured MediaStreamTrack ends when playback ends (and the ended event fires) or when the track that it captures is no longer selected or enabled for playback." This doesn't say exactly when the readyState change happens and when the ended event is fired on the MediaStreamTrack compared to changes to the HTMLMediaElement, but that's something I believe ought to be tested and well defined. This is just an example, there are other timing issues as well.

I think a good way to work through these issues would be to write very pedantic test cases, and in doing so decide what behavior actually makes sense. Then change the spec and implementation to match. If I enumerate all of the bits that I think are not clear in the spec, would you be willing to turn that into tests and spec/implementation changes?

Philip

[Rick Byers]

Also please create a chromestatus.com entry so that our docs and blog posts reflect the status of this feature etc.

[Niklas Enbom]

On Wed, Jun 22, 2016 at 6:06 PM, Rick Byers <rby...@chromium.org> wrote:

Also please create a chromestatus.com entry so that our docs and blog posts reflect the status of this feature etc.

That already exist thanks to mcasas@: https://www.chromestatus.com/feature/5522768674160640

[Philip Jägenstedt]

Are the additions in https://codereview.chromium.org/2110573002/ intended to be covered by this intent? In that review I have some doubts that captureStreamUntilEnded() is actually a useful addition to the web platform. It seems to me that the benefit of an API like this could be in guaranteeing that not a single frame or audio sample after some condition appears in the MediaStream, which cannot be done by reacting to a event, but that would also make the implementation and tests more complicated.

[Miguel Casas]

On 29 June 2016 at 04:46, Philip Jägenstedt <foo...@chromium.org> wrote:

Are the additions in https://codereview.chromium.org/2110573002/ intended to be covered by this intent?

Probably not, since I too see little value in baking into Blink something that can be done with a (trivial?) polyfill.

Spec bug?

 

In that review I have some doubts that captureStreamUntilEnded() is actually a useful addition to the web platform. It seems to me that the benefit of an API like this could be in guaranteeing that not a single frame or audio sample after some condition appears in the MediaStream, which cannot be done by reacting to a event,

What do you mean? I see this as: listen for onended from the <video>/<audio> and finish the MediaStream when and if.

--

Miguel Casas-Sanchez |

Gatopardo del Software |

ydog / mca...@google.com |

+1 (650) 603 1380

[Philip Jägenstedt]

On Wed, Jun 29, 2016 at 6:31 PM Miguel Casas <mca...@google.com> wrote:

On 29 June 2016 at 04:46, Philip Jägenstedt <foo...@chromium.org> wrote:

Are the additions in https://codereview.chromium.org/2110573002/ intended to be covered by this intent?

Probably not, since I too see little value in baking into Blink something that can be done with a (trivial?) polyfill.

Spec bug?

I guess, but was anyone working on Blink involved in having it added to the spec, that we should talk to about it first?

 

In that review I have some doubts that captureStreamUntilEnded() is actually a useful addition to the web platform. It seems to me that the benefit of an API like this could be in guaranteeing that not a single frame or audio sample after some condition appears in the MediaStream, which cannot be done by reacting to a event,

What do you mean? I see this as: listen for onended from the <video>/<audio> and finish the MediaStream when and if.

I mean that it's possible for scripts to do things after mediaElement.ended becomes true but before the ended event has fired. For example if one seeks it's not obvious if this might result in some video or audio frames from the new position passing through the MediaStream before it's closed. Maybe not, but that kind of guarantee is the main reason I could see for an API of this sort.

[cgfa...@gmail.com]

Hi all,

I would like to see this API come out from behind experimental web features. Support for this feature allows AV content to be easily overdubbed when played back via the existing media DOM elements. 

Here are a couple scenarios where I see shipping this being useful.

1. Audio Interviews or podcasts where you wanted to overdub with excerpts from other sources. Specifically with a live mic feed for commenting or announcing similar to the way esports are currently broadcast.

2. Informational graphic layouts can be combined with existing video to make the content more compelling and these elements can be connected to HTTP, WebSockets, etc.

3. Graphical layouts can be added to video that provide journalistic integrity and accreditation. 

Using this technology I was able to record an interview in WebM and then overlay videos with Canvas or WebGL graphics. This opens up a whole new type of software that can be built on the web that was previously only accessible to software with access to core APIs.

Broadcast technology is expensive and cumbersome. This API dramatically lowers the barrier to entry for developers looking to create better experiences for journalists, video editors, audio editors and enterprise users that wish to use WebRTC as their communication platform. These are all positives for the open web.

Thanks for your consideration,

Chase