Intent to Ship: Support codec and container switching with MSE using SourceBuffer.changeType()

[Matthew Wolenetz]

Contact emails

wole...@chromium.org

Explainer

Explainer

Spec

MSE codec-switching incubation editor's draft, containing changeType() specification and related algorithm updates

To identify the portion of the MSE API spec that is changed for this feature, see the links contained within https://github.com/wicg/media-source/blob/codec-switching/README.md:

• HTML Diff versus REC MSE spec

• HTML Diff versus upstream W3C Editor's Draft

Tag Review: Not requested, feature is incremental to the existing MSE REC API.

Summary

This feature adds support for improved cross-codec or cross-bytestream transitions in Chrome HTML5 Media Source Extensions playback using a new changeType method on SourceBuffer that allows the type (bytestream and codec(s)) of media bytes subsequently appended to the SourceBuffer to be changed. We are incubating this idea via the WICG, with goal of eventually working with WebPlatformWG to get the result of WICG incubation as part of the next version of the Media Source Extensions API (MSE).

Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/d/topic/blink-dev/atNyZDs-WXY/discussion

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Demo link

Enable chrome://flags#enable-experimental-web-platform-features on >= M69, then:

https://rawgit.com/wolenetz/shaka-player/mse-codec-switching-demo-using-shaka-player/demo/#asset=https://storage.googleapis.com/shaka-demo-assets/angel-one/dash.mpd;lang=en-US;logtoscreen;debug;build=uncompiled

Risks

Interoperability and Compatibility

Low. Firefox has implementing it as part of incubation, and in their 63 are shipping it on-by-default. Applications can detect presence of this API (SourceBuffer.prototype inspection for changeType function) and fall-back to using existing complex, high-latency element or track switching logic if this API is absent (or choose to constrain their media type choices to those which do not require this API, if this API is absent).

Edge: No signals

Firefox: Shipping on-by-default in version 63 (currently their nightly build)

Safari: No signals

Web developers: Positive. YouTube is interested in this API, has integrated and tested this API using chrome://flags, and is ready to use it when available. Feature demo is built using slightly modified shaka-player.

Ergonomics

User-visible performance effects of decoding and rendering of changeType()-caused codec transitions should be improved (done via pipeline asynchronously) versus existing workarounds for lack of this API (done synchronously by web app explicitly switching media elements or tracks).

Activation

The custom demo for this feature is built using a slightly modified shaka-player, which exists partially as an reference player for MSE.

Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

Yes, this feature’s tests are part of the larger media-source wpt suite. Specific coverage for this feature is:

• https://wpt.fyi/results/media-source/mediasource-changetype.html

• https://wpt.fyi/results/media-source/mediasource-changetype-play.html

Entry on the feature dashboard

https://www.chromestatus.com/features/5719220952236032

[Philip Jägenstedt]

The explainer for this is a great read, thank you! There are some bits in "Open Questions and Thoughts" that would have to have been answered while implementing this, would you mind updating to reflect what the spec+implementation now does? (In particular: "Should the proposed changeType method implicitly perform the reset parser state algorithm")

While this is an incremental change to MSE, it's one that I could easily imagine being quite hard to implement in some media frameworks. Would you mind filing a TAG review request (not blocking shipping) as a last opportunity for such issues to be surfaced? I see "No signals" for Edge/Safari, but take it they are at least aware of the change? If not, can you email them?

I'm glad to report that because this is behind a flag, we can see the results on wpt.fyi with ?label=experimental:

https://wpt.fyi/results/media-source/mediasource-changetype.html?label=experimental

https://wpt.fyi/results/media-source/mediasource-changetype-play.html?label=experimental

The first is entirely passing (yay) but the second is failing because calling video.play() requires a user gesture. This isn't a problem in content_shell. It would be possible to update the tests to use test_driver.click() to fix this problem in Chrome as well, but I'll leave that up to you.

With all of that, LGTM1 :)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAADho6NU04CSNDheZK_CJ5wwTekiKFFjJaMPQEyS6qP6d1jzHQ%40mail.gmail.com.

[Matthew Wolenetz]

Thanks for the excellent feedback, Philip!

I've emailed Safari and Edge folks today; note also, the MSE spec issue for this feature has been continually updated, giving transparency to other implementors and API users.

I'm working through filing the TAG review and updating the explainer.

At a little lower priority, I'll investigate using test_driver.click() into this (and other pre-existing) MSE wpt tests.

Matt

[Matthew Wolenetz]

Inline

On Tue, Aug 14, 2018 at 3:33 PM, Matthew Wolenetz <wole...@chromium.org> wrote:

Thanks for the excellent feedback, Philip!

I've emailed Safari and Edge folks today; note also, the MSE spec issue for this feature has been continually updated, giving transparency to other implementors and API users.

Edge folks responded to my email and are looking into the feature.

Safari folks responded positively to my email and are looking into the feature, too.

 

I'm working through filing the TAG review and updating the explainer.

I've filed a TAG review:  https://github.com/w3ctag/design-reviews/issues/298

At a little lower priority, I'll investigate using test_driver.click() into this (and other pre-existing) MSE wpt tests.

I'll do this and the explainer updates this week. I don't think either block further reviews or shipping this feature.

[Philip Jägenstedt]

Thanks Matt for pushing on all of those fronts.

For the test_driver.click() (or bless()) bits, if you find that it complicates many tests, consider instead filing an issue about always bypassing the user gesture requirements when running wpt. That's what content_shell does, many tests assume it, so it's possible we should just accept that reality. (Downside: testing something a bit less similar to what web developers really deal with.)

[Matthew Wolenetz]

Safari (Mac) support for changeType was committed today (https://bugs.webkit.org/show_bug.cgi?id=188626), IIUC behind a runtime flag. They commented that the implementation was relatively straightforward.

[Philip Jägenstedt]

Sweet, that's 3/4 then. API owners, if you missed it, my LGTM1 was buried at the bottom of a longer reply.

[Yoav Weiss]

LGTM2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYcuP1TdXyt2apf9CCF1-vMA%2BHmFqFC_1pOM1g3-ON1%2BJw%40mail.gmail.com.

[Chris Harrelson]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEiksC_M9ds5nqysW2HZ83OBqnd6jPUjdfFGwq4fxDFktQ%40mail.gmail.com.

[Matthew Wolenetz]

On Tue, Aug 14, 2018 at 1:39 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

The explainer for this is a great read, thank you! There are some bits in "Open Questions and Thoughts" that would have to have been answered while implementing this, would you mind updating to reflect what the spec+implementation now does? (In particular: "Should the proposed changeType method implicitly perform the reset parser state algorithm")

I've updated the explainer today. It reflects the shipped API now, and also includes some links to implementation status and wpt tests.

LGTM3

LGTM2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYcuP1TdXyt2apf9CCF1-vMA%2BHmFqFC_1pOM1g3-ON1%2BJw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.