[blink-dev] Intent to ship: Media tracks

[Sergey Volk]

Contact emails

ser...@chromium.org

wole...@chromium.org

Specs

https://www.w3.org/TR/html5/embedded-content-0.html#audiotracklist-and-videotracklist-objects

https://www.w3.org/TR/media-source/#audio-track-extensions

https://www.w3.org/TR/media-source/#video-track-extensions

https://www.w3.org/TR/media-source/#sourcebuffer-init-segment-received

Previous 'Intent to Implement: Media tracks':

https://groups.google.com/a/chromium.org/forum/#!searchin/blink-dev/media$20tracks/blink-dev/7ZWqizEIK1E/D3rDr_HEAwAJ

Summary:

Basic media tracks functionality is ready, and even though it's limited to up to 1 audio and up to 1 video track for now, the ability to enable/disable (select/deselect) media tracks can be already very useful. For example Mounir (mlamouri@) has been recently experimenting with deselecting/disabling video tracks in the background tabs and got noticeable improvement in CPU and battery usage.

Currently media tracks are hidden behind the --enable-experimental-web-platform-features flag, but we would like to unmark it as experimental and make it available by default.

Another benefit of this change would be that it would enable also the new implementation of MSE 'init segment received' algorithm by default.

Interoperability and Compatibility Risk

None

Ongoing technical constraints

Chromium media pipeline is currently limited to only 1 audio and 1 video track max, but I'm looking into what it would take to support multiple tracks and so far it looks like it would not require any further changes on blink level.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? Yes or no.

Yes (on Android this is going to work only with unified media pipeline).

OWP launch tracking bug

http://crbug.com/249427, http://crbug.com/249428

Link to entry on the feature dashboard

https://www.chromestatus.com/features/5748496434987008

Requesting approval to ship?

Yes

[ser...@google.com]

Here is also a CL that I've prepared to enable media tracks by default:

https://codereview.chromium.org/2249763003/

[PhistucK]

What is the plan regarding support for multiple tracks?

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Harald Alvestrand]

And does this interact correctly (as per spec) with MediaStreams created by (for instance) getUserMedia? These can have multiple tracks, and the implementation currently does something sensible with them; if the Media element supports tracks, we have to make sure the interactions are correct.

Do we have tests that cover the MediaStream tracks' interaction with Media element tracks?

[Harald Alvestrand]

Interaction spec:

https://w3c.github.io/mediacapture-main/getusermedia.html#mediastreams-in-media-elements (MediaStream to media element)

https://w3c.github.io/mediacapture-fromelement/ (capture a MediaStream from a media element)

[Sergey Volk]

I've mentioned that in the ongoing technical constraints section: multiple media tracks (more than 1 audio and/or video tracks) are currently not supported by Chromium media pipeline. But as far as I can tell everything is ready for multi-track support on blink level. I'm planning to look into possibility of supporting multiple media tracks, but I'm not yet sure if it's going to be ready for M54. But I believe that CPU/battery usage gains that could be obtained by disabling the video track in background tabs already make even the existing functionality limited to 1 audio and 1 video track fairly valuable.

[PhistucK]

Thank you for elaborating somewhat!

☆PhistucK

[Sergey Volk]

Thanks for pointing this out, Harald. I wasn't aware of this functionality.

https://w3c.github.io/mediacapture-fromelement/ section 3.1 has this paragraph:

The captured MediaStream comprises of MediaStreamTracks that render the content from the set ofselected (for VideoTracks, or other exclusively selected track types) or enabled (for AudioTracks, or other track types that support multiple selections) tracks from the media element. If the media element does not have a selected or enabled tracks of a given type, then no MediaStreamTrack of that type is present in the captured stream.

And yet from a quick glance at the implementation in HTMLMediaElementCapture::captureStream it looks like media track status is not taken into account properly. +mcasas@ who implemented this functionality.

Miguel, should we update HTMLMediaElementCapture::captureStream to capture only selected/enabled media tracks? Do we have layout tests for the media capture functionality?

[Miguel Casas]

On 17 August 2016 at 10:18, Sergey Volk <ser...@chromium.org> wrote:

Thanks for pointing this out, Harald. I wasn't aware of this functionality.

https://w3c.github.io/mediacapture-fromelement/ section 3.1 has this paragraph:

The captured MediaStream comprises of MediaStreamTracks that render the content from the set ofselected (for VideoTracks, or other exclusively selected track types) or enabled (for AudioTracks, or other track types that support multiple selections) tracks from the media element. If the media element does not have a selected or enabled tracks of a given type, then no MediaStreamTrack of that type is present in the captured stream.

And yet from a quick glance at the implementation in HTMLMediaElementCapture::captureStream it looks like media track status is not taken into account properly. +mcasas@ who implemented this functionality.

Miguel, should we update HTMLMediaElementCapture::captureStream to capture only selected/enabled media tracks? Do we have layout tests for the media capture functionality?

Yeah, I think that part was not contemplated in the implementation so far, but with the Spec having still some unclear parts, I decided to wait and see. If we update capturestream(), creating LayoutTest should be straightforward.

--

Miguel Casas-Sanchez |

Gatopardo del Software |

ydog / mca...@google.com |

+1 (650) 603 1380

[Sergey Volk]

Ok, thanks for clarifying this, Miguel.

Does anybody have any other concerns?

If not, I'd like to flip the switch and mark the media tracks feature as stable in M54:

https://codereview.chromium.org/2249763003/

[Anton Vayvod]

Thread bump. It needs some love from the API owners.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[TAMURA, Kent]

It says Edge shipped this feature.  Is the Edge implementation compatible with ours?  Does Edge support multiple tracks?

 

Requesting approval to ship?

Yes

--

TAMURA Kent
Software Engineer, Google

[Philip Jägenstedt]

I just did some quick testing of Edge 38.14393.0.0 and Safari 10.0 (12602.1.38.2) and can confirm that they support the audioTracks and videoTracks APIs. I tested with a random file of mine, and found some inconsistencies. All expose one audio track and one video track:

Chrome:

audioTracks[0]: {

 "id": "2",

 "kind": "main",

 "label": "SoundHandler",

 "language": "und",

 "enabled": true,

 "sourceBuffer": null

}

videoTracks[0]: {

 "id": "1",

 "kind": "main",

 "label": "VideoHandler",

 "language": "und",

 "selected": true,

 "sourceBuffer": null

}

Edge:

audioTracks[0]: {

 "enabled": true,

 "id": "1",

 "kind": "",

 "label": "",

 "language": "",

 "sourceBuffer": null

}

videoTracks[0]: {

 "id": "0",

 "kind": "",

 "label": "",

 "language": "",

 "selected": true,

 "sourceBuffer": null

}

Safari:

audioTracks[0]: {

 "id": "2",

 "kind": "main",

 "label": "",

 "language": "",

 "enabled": true,

 "sourceBuffer": null

}

videoTracks[0]: {

 "id": "1",

 "kind": "main",

 "label": "",

 "language": "",

 "selected": true,

 "sourceBuffer": null

}

(I used function p(o){var x={};for(var k in o){x[k]=o[k]};return JSON.stringify(x,null,' ')} in the console.)

Safari looks the most reasonable to me. Some basic shared tests in web-platform-tests should go a long way in improving this situation. It would also be very nice to see some interop tests for the basic timing of when the track list objects change and when events are fired. And test using files with actual metadata on the tracks as well as multiple audio/video tracks.

When it comes to multiple audio tracks, can you investigate whether Edge and Safari supports that?

[Sergey Volk]

Hi Philip,

I have just recently added a new layout test that verifies 'change' events being fired on both HTMLMediaElement and SourceBuffer track lists whenever selected/enabled tracks change (see https://codereview.chromium.org/2263823002/).

And we already have tests that verify actual track metadata (see https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/http/tests/media/media-source/mediasource-avtracks.html?rcl=0&l=10), although I agree we could improve our test coverage. But we do source media track attributes as described in https://dev.w3.org/html5/html-sourcing-inband-tracks/.

I'm actually curious why you think Safari looked the most reasonable in your tests, since I've just checked your file with ffprobe and it looks like our implementation is the most compliant (even though I agree that having tracks labels like SoundHandler/VideoHandler is a little weird, but that's according to spec! Those are the values that were read from the handler_name in your .mp4 file):

servolk@servolk:~$ ffprobe https://archive.org/download/edisons-forsta-dagar/MVI_4802.mp4

ffprobe version N-81387-ge2a39b1 Copyright (c) 2007-2016 the FFmpeg developers

...

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'https://archive.org/download/edisons-forsta-dagar/MVI_4802.mp4':

  Metadata:

    major_brand     : isom

    minor_version   : 512

    compatible_brands: isomiso2avc1mp41

    title           : Edisons första dagar - http://archive.org/details/edisons-forsta-dagar

    encoder         : Lavf54.3.100

    comment         : license:http://creativecommons.org/licenses/by/2.5/se/

  Duration: 00:00:20.46, start: 0.000000, bitrate: 827 kb/s

    Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 634x360, 700 kb/s, 25 fps, 25 tbr, 25 tbn, 50 tbc (default)

    Metadata:

      handler_name    : VideoHandler

    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 121 kb/s (default)

    Metadata:

      handler_name    : SoundHandler

[Philip Jägenstedt]

Since Edge and Safari have already shipped this, it could be the cast that they agree on some behavior that makes sense but is contrary to the spec, in which case we might want to change the spec spec. But let's look at the specifics.

For language, can you determine if it's simply not implemented in Edge+Safari, or if they perhaps map "und" to the empty string explicitly? If they do support it in some capacity, do they always return valid BCP 47 strings, i.e. do they turn "eng" into "en" and so on? https://tools.ietf.org/html/bcp47#section-2.1 asks for the "shortest ISO 639 code". It could be that this actually doesn't make sense to implement, in which case HTML should refer to something other than BCP 47.

For label, do Edge+Safari ever return a non-empty string, and if so where does it seem to come from?

Given differences in the underlying media frameworks I'd be surprised if we can get perfect interop here in the short term, but it would be good to know what the differences are and how we hope it'll play out, and file bugs to help us get there.

[Sergey Volk]

I've generated a couple of multi-track files using the following commands:

    ffmpeg -f lavfi -i color=c=red:size=320x240 -t 10 -c:v libvpx red.webm

    ffmpeg -f lavfi -i color=c=green:size=320x240 -t 10 -c:v libvpx green.webm

    ffmpeg -f lavfi -i "sine=frequency=300:sample_rate=48000" -t 10 -c:v libvpx a300hz.webm

    ffmpeg -f lavfi -i "sine=frequency=500:sample_rate=48000" -t 10 -c:v libvpx a500hz.webm

// The only difference between the following two is output format - webm vs mp4

    ffmpeg -i green.webm -i a300hz.webm -i red.webm -i a500hz.webm -map 0 -map 1 -map 2 -map 3 -metadata:s:a:0 title="audio_track0" -metadata:s:a:0 language="eng" -metadata:s:a:1 title="audio_track1" -metadata:s:a:1 language="en-US" -metadata:s:v:0 title="video_track0" -metadata:s:v:1 title="audio_track1"  multitrack.webm

    ffmpeg -i green.webm -i a300hz.webm -i red.webm -i a500hz.webm -map 0 -map 1 -map 2 -map 3 -metadata:s:a:0 title="audio_track0" -metadata:s:a:0 language="eng" -metadata:s:a:1 title="audio_track1" -metadata:s:a:1 language="en-US" -metadata:s:v:0 title="video_track0" -metadata:s:v:1 title="audio_track1"  multitrack.mp4

And uploaded them here:

http://storage.googleapis.com/servolk/multitrack.webm

http://storage.googleapis.com/servolk/multitrack.mp4

Then did some testing. Neither Safari nor Edge played the webm file URL (both downloaded it instead of playing).

Safari 9.1.2 picked up multiple tracks in multitrack.mp4, selected and enabled all tracks (including both video tracks - must be a bug!) by default. It showed the language as "eng" on audio track 0, but as an empty string on the secondary audio track. I have been able to disable/enable audio tracks through JS console, but had to deselect video track 0 manually before I could see the secondary video track (red). Looks like selecting a video track is broken in Safari.

Edge 14.14393 on Win10 VM also did pick up multiple tracks in multitrack.mp4, although it has a different set of quirks from Safari: it enabled the first audio track, but selected the second (last?) video track by default. Switching between tracks worked. The language on both audio tracks showed up as "en".

Neither Safari nor Edge showed any non-empty track labels.

So in short, my conclusion is: even though browsers do claim media track support, the interop story is not that great at the moment and there are bugs across the board, especially for multi-track scenarios.

[Philip Jägenstedt]

hanks Sergey, great to have another file to test with. Here's the output for multitrack.mp4 in the same form as my previous post:

Chrome:

audioTrack[0]: {

 "id": "2",

 "kind": "main",

 "label": "SoundHandler",

 "language": "eng",

 "enabled": true,

 "sourceBuffer": null

}

videoTrack[0]: {

 "id": "1",

 "kind": "main",

 "label": "VideoHandler",

 "language": "und",

 "selected": true,

 "sourceBuffer": null

}

Edge:

audioTrack[0]: {

 "enabled": true,

 "id": "2",

 "kind": "",

 "label": "",

 "language": "en",

 "sourceBuffer": null

}

audioTrack[1]: {

"enabled": false,

"id": "3",

"kind": "",

"label": "",

"language": "en",

"sourceBuffer": null

}

videoTrack[0]: {

"id": "0",

"kind": "",

"label": "",

"language": "",

"selected": false,

"sourceBuffer": null

}

videoTrack[1]: {

 "id": "1",

 "kind": "",

 "label": "",

 "language": "",

 "selected": true,

 "sourceBuffer": null

}

Safari:

audioTrack[0]: {

 "id": "2",

 "kind": "main",

 "label": "",

 "language": "en",

 "enabled": true,

 "sourceBuffer": null

}

videoTrack[0]: {

 "id": "3",

 "kind": "main",

 "label": "",

 "language": "",

 "selected": true,

 "sourceBuffer": null

}

For completeness, also Chrome with multitrack.webm:

audioTrack[0]: {

 "id": "2",

 "kind": "main",

 "label": "audio_track0",

 "language": "eng",

 "enabled": true,

 "sourceBuffer": null

}

videoTrack[0]: {

 "id": "1",

 "kind": "main",

 "label": "video_track0",

 "language": "",

 "selected": true,

 "sourceBuffer": null

}

Edge and Safari already agree on language "en" and the empty string where Chrome has "eng" and "und". Maybe Safari recently changed the handling of language?

That "SoundHandler" shows up for this file as well doesn't mean that it's necessarily wrong, but it would be worth looking into the specs to see if that's really the most reasonable field.

When testing this I noticed something else, namely that the track metadata is available for cross-origin media resources. That seems pretty bad, at least something to think hard about. In-band text tracks don't get exposed at all, but it looks like the spec doesn't say anything for audio and video tracks. Filed https://github.com/whatwg/html/issues/1735

It looks to me like a discussion needs to happen with the other vendors to come to an agreement on points of divergence, which is really all of kind, label and language. If aligning on how id is generated isn't too much of a hassle, that'd be nice too. Is this an effort you'd be interested in driving, Sergey?

[Sergey Volk]

Hi Philip,

To be honest I think that those small inconsistencies in track metadata between different browsers is a relatively minor issue, it should not have much effect on apps or general track functionality. I think it's more important to make Chrome support multiple tracks, since that would make the media tracks functionality much more useful. So atm I'm planning to prioritize this over metadata improvements.

I also don't really understand why exposing track metadata for cross-origin is big deal. If some media _data_ stream is available cross-origin, then IMO it's fine to expose the track metadata (label, language, etc) as well - the metadata is nowhere near as valuable as the data streams themselves, but let's continue that discussion in the bug that you filed.

[Anne van Kesteren]

On Thu, Sep 1, 2016 at 8:07 PM, Sergey Volk <ser...@chromium.org> wrote:
> I also don't really understand why exposing track metadata for cross-origin
> is big deal. If some media _data_ stream is available cross-origin, then IMO
> it's fine to expose the track metadata (label, language, etc) as well - the
> metadata is nowhere near as valuable as the data streams themselves, but
> let's continue that discussion in the bug that you filed.

The data stream is not exposed cross-origin. (That the user can play
the media does not mean it's exposed to the origin.) The only thing
new thing that leaks with media as I understand it is duration.


--
https://annevankesteren.nl/

[Philip Jägenstedt]

Inconsistencies, minor or not, have the potential to waste time for web developers when they develop on one browser and then find it doesn't work in another. Especially in the case of language, it's easy to imagine a langMap={"en":"English","":"Unknown language"} that'll need tweaking. The web is full of pointless differences like this, but it doesn't have to be.

When it comes to the label, you may well be correct, I just want this to be double-checked with other vendors since it looks like the labels will be useless for the most part. Just to get some variety, I also tested with an Apple trailer, here's what Chrome says:

audioTrack[0]: {

 "id": "2",

 "kind": "main",

 "label": "Apple Sound Media Handler",

 "language": "eng",

 "enabled": true,

 "sourceBuffer": null

}

videoTrack[0]: {

 "id": "1",

 "kind": "main",

 "label": "Apple Video Media Handler",

 "language": "eng",

 "selected": true,

 "sourceBuffer": null

}

A different label, but if all media frameworks just stuff a fixed string in it's not going to be of much use. It looks like FFmpeg can end up with another string, but I didn't follow it to the source. At least it seems like the title="audio_track0" you used on the command line wasn't it. Just by looking at multitrack.mp4 in a hex editor, it look like "audio_track0" ended up in the name field of a user data box ('udta'). So maybe https://dev.w3.org/html5/html-sourcing-inband-tracks/ should use that instead. Needs investigation and discussion.

Not everything has to be perfectly aligned to ship, but we should understand all of the differences, make sure we have agreement on what the right thing is, and file bugs on the other vendors.

I'd like to hear from the other API owners if they agree, or if I'm asking too much.

[Philip Jägenstedt]

I messed up the formatting of the middle. Here it is again:

When it comes to the label, you may well be correct, I just want this to be double-checked with other vendors since it looks like the labels will be useless for the most part. Just to get some variety, I also tested with an Apple trailer, here's what Chrome says:

audioTrack[0]: {
"id": "2",
"kind": "main",
"label": "Apple Sound Media Handler",
"language": "eng",
"enabled": true,
"sourceBuffer": null
}
videoTrack[0]: {
"id": "1",
"kind": "main",
"label": "Apple Video Media Handler",
"language": "eng",
"selected": true,
"sourceBuffer": null
}

[Rick Byers]

Sorry for the delay on this intent - it's a complex issue with non-obvious trade-offs.  I _think_ I've done enough research now to be able to have an informed opinion.

I agree with Philip that I'd like to see some more effort put into checking interoperability and at least filing bugs to track known differences.  It's true these are small, and each one in isolation is unlikely to be a major problem for anyone.  But, as we survey developers about what sucks about the web it's becoming increasingly clear that the totality of all these little small differences are really adding up and making web developers lives quite painful.  We're actively pushing to increase our discipline so that developers are less likely to stumble across little differences in behavior between browsers.

So for me it's not the specific issue of the language field (although it does seem there is either a spec or implementation bug in that case), but the larger risk that lots of such issues are going un-noticed and being found only when an API owner does a quick comparison of behavior between browsers.  Generally we expect the feature teams to do that and be able to roughly characterize the outstanding differences as part of the ship process, pointing to implementation and spec bugs that track the known problems (it's not scalable to expect API owners to uncover interop issues for all shipping features).

So here's what I'd suggest to ratchet up the interop discipline a bit here without imposing an unreasonable burden on shipping each new feature.  Sergey, can you spent a little time (maybe a day or two) playing with the behavior of these (and optionally the surrounding related) APIs in all major browsers (manually or even ideally via automated W3C tests) on various content.  Then file bugs (spec issues, bugs on other browsers, and/or chromium bugs as appropriate) to track differences you find (presumably there are more than just what Philip has uncovered).  Then reply to this thread with the list of issues and your judgment on their combined impact on the potential for developer pain.  If there are some chromium issues that can be fixed now to improve consistency between browsers, that's great.  But if not, that's OK too - as long as we have confidence that we're aware of most of the differences and are tracking them for discussion, future work and for developers to find and comment on when the hit them in the field.

Rick