Intent to Implement: Support codec and container switching with MSE using SourceBuffer.changeType()

[Matthew Wolenetz]

Contact emails

wole...@chromium.org

Explainer

https://github.com/wicg/media-source/blob/codec-switching/codec-switching-explainer.md

Design doc/Spec

• MSE codec-switching incubation editor's draft, containing changeType() specification and related algorithm updates
  

Further helpful links are contained within https://github.com/wicg/media-source/blob/codec-switching/README.md:

• HTML Diff versus REC MSE spec
• HTML Diff versus upstream W3C Editor's Draft

Tag Review: Not requested, feature is incremental to the existing MSE REC API.

Summary

This feature adds support for improved cross-codec or cross-bytestream transitions in Chrome HTML5 Media Source Extensions playback using a new changeType method on SourceBuffer that allows the type (bytestream and codec(s)) of media bytes subsequently appended to the SourceBuffer to be changed. We are incubating this idea via the WICG, with goal of eventually working with WebPlatformWG to get the result of WICG incubation as part of the next version of the Media Source Extensions API (MSE).

Motivation

MSE already supports same-codec and same-bytestream ad-insertion, stream encoding adaptation (like bitrate and resolution) and media splicing. However, for cross-codec or cross-bytestream media playbacks, MSE REC spec requires the web app to take explicit steps to accomplish the transition (switching among multiple HTMLMediaElements or SourceBuffer tracks), increasing application complexity and user-visible latency (such transitions require the web app to take synchronous action on the renderer main thread). This transition latency impairs the smoothness of media playback across transitions.

See also further motivation in the explainer, linked above.

Risks

Interoperability and Compatibility

Firefox is already implementing it as part of incubation. Applications can detect presence of this API and use existing complex, high-latency element or track switching logic if this API is absent.

Edge: No signals

Firefox: In development

Safari: No signals

Web developers: Positive

Ergonomics

User-visible performance effects of decoding and rendering of changeType()-caused codec transitions should be improved (done via pipeline asynchronously) versus existing workarounds for lack of this API (done synchronously by web app explicitly switching media elements or tracks).

Activation

A demo application would be useful. As with existing MSE REC buffering logic, implementations vary in their tolerance for how to determine buffered range gaps resulting from media appends by applications. Successful adoption of usage of this API may benefit from later incubations being considered, such as a "play through unbuffered regions" MSE API.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Implementation tracking bug

http://crbug.com/605134

Link to entry on the feature dashboard

https://www.chromestatus.com/features/5719220952236032

Requesting approval to ship?

No