Intent to Implement: Audio and video tracks

[Brendan Long]

Contact emails

b.l...@cablelabs.com / se...@brendanlong.com

Spec

http://www.w3.org/TR/html5/embedded-content-0.html#audiotracklist-and-videotracklist-objects

Summary

This would add AudioTrack, VideoTrack, AudioTrackList and VideoTrackList objects, which have similar attributes to TextTrack and TextTrackList (id, label, language, kind). Using AudioTrack.enabled and VideoTrack.selected, you can choose the audio or video track to play in a multi-track file.

Motivation

This is the only way to choose between multiple audio or video tracks in a single file. This would be useful if a video has multiple available languages, a commentary track, or alternate video angles.

Compatibility Risk

This is already implemented in WebKit (Apple WebKit, plus WebKitGTK and EFLWebKit):

https://bugs.webkit.org/show_bug.cgi?id=113965
https://bugs.webkit.org/show_bug.cgi?id=117039
https://bugs.webkit.org/show_bug.cgi?id=122043

Firefox seems interested but hasn't started on it:

https://bugzilla.mozilla.org/show_bug.cgi?id=744896

Apparently Internet Explorer 10+ supports this:
http://msdn.microsoft.com/en-us/library/ie/hh772592%28v=vs.85%29.aspx

Ongoing technical constraints

I don't think so? This will obviously only work on platforms where we have a media player..

Will this feature be supported on all five Blink platforms (Windows, Mac, Linux, Chrome OS and Android)?

I'm only planning to make this work for FFMPEG + MediaSource. I think Android uses a different media player, so there may be additional work involved there.

OWP launch tracking bug?

None.

Unless this counts: http://code.google.com/p/chromium/issues/detail?id=249427

Link to entry on the feature dashboard

I don't think I have permissions to create one of these.

Requesting approval to ship?

No. Should I add this to --enable-inband-text-tracks or add another --enable-audio-and-video-tracks?

[Max Heinritz]

You should have access with se...@brendanlong.com now.  Click "Log in" at the bottom to edit.  Please try adding an entry and let me know you have any trouble.

For future reference, there's a form for requesting access linked to from the footer. 

Requesting approval to ship?

No. Should I add this to --enable-inband-text-tracks or add another --enable-audio-and-video-tracks?

Would this be appropriate behind the --enable-experimental-web-platform-features flag?

[Aaron Colwell]

+1. I support this work and am happy to do the reviews. I think this can live under the  --enable-experimental-web-platform-features flag. I am also happy to help with the Android work.

Aaron

On Fri, Nov 22, 2013 at 2:19 PM, Brendan Long <se...@brendanlong.com> wrote:

[Eric Seidel]

If your'e not requesting approval to ship then you don't need any
LGTMs, but your feature will have to be behind a runtime flag (so it
does not affect the shipping stable build). When you're ready to turn
your flag on, then you'll need an Intent To Ship, unless the feature
qualifies under the "trivial" clause:
http://www.chromium.org/blink#new-features

[Brendan Long]

On 11/22/2013 04:28 PM, Max Heinritz wrote:

Link to entry on the feature dashboard

I don't think I have permissions to create one of these.

You should have access with se...@brendanlong.com now.  Click "Log in" at the bottom to edit.  Please try adding an entry and let me know you have any trouble.

Feature dashboard link:

http://www.chromestatus.com/features/5748496434987008

[PhistucK]

Thank you very much!

Why only under FFMPEG + MediaSource?

Does this mean loading MP4 or WebM files with multiple audio would not work?

Or are you not referring to the combination of FFMPEG and MediaSource, but to FFMPEG and that you will also include support for MediaSource?

☆PhistucK

[Brendan Long]

On 11/23/2013 02:52 AM, PhistucK wrote:

Why only under FFMPEG + MediaSource?

Mainly because of time constraints. If getting this working in the Android media player is easy, I might do that to, but I can't promise anything.

Does this mean loading MP4 or WebM files with multiple audio would not work?

This change should make those files work.

Or are you not referring to the combination of FFMPEG and MediaSource, but to FFMPEG and that you will also include support for MediaSource?

I mean both MediaSource and files on platforms with FFMPEG. I think Android is the only platform that doesn't use FFMPEG (from my quick look through the code).

[Ami Fischman]

On Fri, Nov 22, 2013 at 2:19 PM, Brendan Long <se...@brendanlong.com> wrote:

I'm only planning to make this work for FFMPEG + MediaSource. I think Android uses a different media player, so there may be additional work involved there.

Brendan: to be clear, you're talking about Chromium's use of FFMPEG's demuxer, not its decoder bits, right?

If so then (non-blink-gatekeeper) LGTM.

(If not, then your phrasing also excludes platforms that use HW acceleration for decode, where ffmpeg's decoder is not used.  But since the feature in question is muxing-centric I'm guessing that's not the case)

PhistucK: chromium's media pipeline (used everywhere but Android) has two demuxer implementations: one based on ffmpeg and one for MediaSource.  It sounds like Brendan is preparing to update both of them, which is great.  Aaron offered to help with Android so everything should be covered.

Cheers,

-a

[Brendan Long]

On 11/23/2013 01:33 PM, Ami Fischman wrote:

On Fri, Nov 22, 2013 at 2:19 PM, Brendan Long <se...@brendanlong.com> wrote:

I'm only planning to make this work for FFMPEG + MediaSource. I think Android uses a different media player, so there may be additional work involved there.

Brendan: to be clear, you're talking about Chromium's use of FFMPEG's demuxer, not its decoder bits, right?

Yes, just the demuxer.