Intent to implement: Media tracks

[ser...@chromium.org]

Contact emails

ser...@chromium.org

Spec

https://www.w3.org/TR/html5/embedded-content-0.html#audiotracklist-and-videotracklist-objects

https://www.w3.org/TR/media-source/#audio-track-extensions

https://www.w3.org/TR/media-source/#video-track-extensions

https://www.w3.org/TR/media-source/#sourcebuffer-init-segment-received

Summary

Implement audio/video track functionality for HTMLMediaElement and MediaSource.

Motivation

This is useful functionality requested by app developers and is already implemented by other browsers (IE, Safari, Firefox).

Previous work:

https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/Jgg71B43w3Y/VDVaJltaIbkJ

The outcome of that was that basic AudioTrack, VideoTrack, AudioTrackList and VideoTrackList objects were added in blink. But little was done in Chromium media stack to make this work, so audio and video tracks that we have in blink today are actually placeholder created by HTMLMediaElement::createPlaceholderTracksIfNecessary. Those placeholder tracks don't reflect actual media track properties (such as bytestream id, language, label) and changing AudioTrack.enabled / VideoTrack.selected doesn't do anything from user point of view.

I'm continuing where Brendan left off and planning to:

1. Make the necessary changes in Chromium media pipeline to report media track metadata from demuxers to blink level. I have already partially done this, see

https://codereview.chromium.org/1716503002/
https://codereview.chromium.org/1727243002/
https://codereview.chromium.org/1735003004/

+ a few more CLs in flight

https://codereview.chromium.org/1735763002/

https://codereview.chromium.org/1659653002/

2. Implement MSE extended interfaces for audio/video tracks (so that tracks created by MediaSource/SourceBuffer have .sourceBuffer property).

https://codereview.chromium.org/1658033002/

3. Implement the necessary logic to apply AudioTrack.enabled / VideoTrack.selected changes.

Here is a work-in-progress CL that makes it possible to mute/unmute audio by setting the AudioTrack.enabled property:

https://codereview.chromium.org/1812543003/ (it depends on all of the CLs listed above).

4. Implement more compliant 'initialization segment received' algorithm from MSE spec (https://www.w3.org/TR/media-source/#sourcebuffer-init-segment-received).

Interoperability and Compatibility Risk

I think this risk should be minimal since this is implementation of relatively mature standards already implemented by other browsers.

Ongoing technical constraints

One constraint that we currently have is that Chromium media pipeline at the moment supports only one audio track and one video track at a time. I'm planning to look into fixing that as well.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

OWP launch tracking bug

http://crbug.com/249427, http://crbug.com/249428

Link to entry on the feature dashboard

https://www.chromestatus.com/features/5748496434987008

Requesting approval to ship?

I'm planning to keep this behind the existing feature flag for now (--enable-blink-features=AudioVideoTracks) and will request approval to ship once this feature stabilizes sufficiently.

[jonob...@gmail.com]

This is awesome!

[Matt Wolenetz]

Thanks for picking up this work, Sergey. You can add me as contact to this intent as well, if you like.

On Mar 24, 2016 9:16 PM, <jonob...@gmail.com> wrote:

This is awesome!

[Rick Byers]

Sounds cool.

In your implementation work, please make sure to refer to the living standards instead of the W3C "TR" snapshots, i.e.:

https://html.spec.whatwg.org/multipage/embedded-content.html#audiotracklist-and-videotracklist-objects

https://w3c.github.io/media-source/

In general in blink we ignore the "TR" snapshots as they are sometimes wildly different from what browsers are actually shipping today.  I updated the spec link in your chromestatus entry (but come time to ship you'll probably want to also enumerate there exactly which APIs are affected).

Thanks,

  Rick

[dulo...@gmail.com]

Thank you! I've waited for this for a long time.

[Philip Jägenstedt]

Another thing to keep in mind is that it's entirely possible that the specs simply fail to describe all aspects that will be observable about this feature. In particular, it seems like the order in which tracks are added to mediaElement.audioTracks and sourceBuffer.audioTracks isn't defined anywhere, nor the timing in between those events. It would be great if you can file spec issues for any small or large problem of this sort you stumble upon.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Sergey Volk]

Yeah, good point about using living standards, I've updated the URLs in blink source code after Philip's comments.

Philip, regarding the ordering of events, the order in which media tracks must be added is actually specified in the MSE spec. Take a look at https://w3c.github.io/media-source/#sourcebuffer-init-segment-received, according to steps 5.2.8.8-5.2.8.11 the media track must be added to SourceBuffer track list first, then an 'addtrack' event should be posted for SB track list, then the track must be added to HTMLMediaElement  track list and then another event must be posted for HTMLMediaElement track list. So at least for now, there's no need to file a spec issue.

I think the only underspecified thing that we've encountered so far is sourcing the value of language tag for mp4 containers, I'm planning to follow up on this once I finish implementing more basic media track functionality.

--
You received this message because you are subscribed to a topic in the Google Groups "blink-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/blink-dev/7ZWqizEIK1E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to blink-dev+...@chromium.org.

[fer...@gmail.com]

Thanks for implementing this. Looking forward to using it in my application.

[Philip Jägenstedt]

Ah thanks, I hadn't read closely enough. I guess my only issue the risk that MSE and the steps that fire addtrack events in HTML go slightly out of sync. Just like the implementation, perhaps a helper would be in order. Anyway, that has no bearing on your implementation work :)

[grant.gl...@gmail.com]

I guess this is still in development? Kicks accessibility right in the face with Chrome not supporting <audiotracks> and having sketchy (at best) <track> support.

[Sergey Volk]

Yes, still in development, and I have a couple of pending CLs that will make it possible to disable and later re-enable audio/video decoding and rendering via media track JS properties:

https://codereview.chromium.org/1922333002

https://codereview.chromium.org/1935873002

But even after these CLs land, there's still going to be a limit of max 1 enabled audio/video track at a time (fixing that limitation would require further major work on Chrome media renderer).

[grant.gl...@gmail.com]

Almost one year on and I still don't see this being implemented in Chromium. Still, at least Firefox haven't released this API into the wild yet either, so kudos for that.

I'll set a reminder on my smart watch to come back in another year for this - the irony of it all is that the idea of a smartwatch had not even mustered in the minds of technology giants when the API was drafted, yet Apple have somehow created a smartwatch, multiple times and have still implemented the API into Safari! 

\sigh

[PhistucK]

Semantics - this is actually implemented (you can enable it using about:flags, crbug.com reports for bugs you find are welcome), but not shipped as enabled by default yet.

You probably meant that this has still not shipped, which is true, for which there is another intent thread.

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.