Intent to Implement: Fix Chrome MSE PTS/DTS Compliance

[Matthew Wolenetz]

Contact email

wole...@chromium.org

Spec

https://www.w3.org/TR/media-source/

(No TAG review deemed necessary since this spec is already a W3C REC)

 

Summary

This change is intended to fix Chrome Media Source Extensions (MSE) implementation's management and reporting of buffered media timeline ranges to be done compliantly by presentation timestamp (PTS) ranges, rather than by decode timestamp (DTS) ranges.  Longstanding assumptions of equivalence of PTS and DTS throughout the Chrome MSE implementation dating back to initial work based on formats like WebM, where there was no such distinction between PTS and DTS, must be fixed to achieve compliance.

Motivation

Though many MSE media streams do not differentiate DTS and PTS (WebM and MP3 in particular), ISO-BMFF (MP4) streams frequently contain out-of-order presentation versus decode time streams (eg, for codecs like h.264) causing DTS and PTS to differ. MP4 MSE streams can also contain a rudimentary ELST (edit list box) allowing further shifting of PTS from DTS.

The result is that Chrome MSE reports (usually just slightly) different buffered ranges and duration values than expected of a compliant implementation for these streams, surprising the API’s users (and failing relevant portions of the W3C media-source web-platform-tests).

Interoperability and Compatibility Risk

Firefox: Shipped (Comment on related MSE spec issue)

Edge: Public support (Multiple comments and author of related MSE spec clarification by MSFT co-editor on related issue)

Safari: Public support (Comment on related MSE spec issue)

Web developers: Positive (Shaka Player tracking of PTS/DTS issues, Multiple stars on related crbugs)

Ongoing technical constraints

None other than maintaining both old and new buffered range management algorithms while experimenting and before fully shipping.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

[Philip Jägenstedt]

No LGTMs necessary for an Intent to Implement, but I take it you intend to just try shipping this with no flag? That makes sense, and I'd consider it a bug fix more than anything else.

Are these format-specific facets of MSE tested by web-platform-tests? Some of this is undoubtedly best tested using unit tests, but for the web-facing APIs (buffered/duration) will it be possible to write some shared tests?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAADho6M8TGJB%3DoG%2BABMfdNTynUdo2-PqCJ2m8XvZrh6wwrvP3g%40mail.gmail.com.

[Matthew Wolenetz]

No LGTMs necessary for an Intent to Implement, but I take it you intend to just try shipping this with no flag? That makes sense, and I'd consider it a bug fix more than anything else.

Before fully launching, since this API compliance change can impact web app behavior (and since the implementation will require significant changes), I plan to experiment and such experiments will likely be overrideable by a flag -- probably MediaSourceExperimental. Googlers can refer to the internal launch bug: https://crbug.com/760264.  If that is *not* the recommended way to proceed, please let me know ASAP. Thanks!

Are these format-specific facets of MSE tested by web-platform-tests? Some of this is undoubtedly best tested using unit tests, but for the web-facing APIs (buffered/duration) will it be possible to write some shared tests?

 

This format-specific piece is covered by web-platform-tests, but those tests will need some additional work to run all supported media types (or at least swap MP4 with WebM in preference order of which format to test if both are supported, since PTS==DTS always in WebM). This is why Chrome passed the tests (WebM-based expectations match) but wouldn't pass MP4 versions of some of those same tests until this ships.

[Philip Jägenstedt]

On Wed, Aug 30, 2017 at 7:55 PM Matthew Wolenetz <wole...@chromium.org> wrote:

No LGTMs necessary for an Intent to Implement, but I take it you intend to just try shipping this with no flag? That makes sense, and I'd consider it a bug fix more than anything else.

Before fully launching, since this API compliance change can impact web app behavior (and since the implementation will require significant changes), I plan to experiment and such experiments will likely be overrideable by a flag -- probably MediaSourceExperimental. Googlers can refer to the internal launch bug: https://crbug.com/760264.  If that is *not* the recommended way to proceed, please let me know ASAP. Thanks!

Sure, if there will be a flag and a later Intent to Ship, then all is well.

 

Are these format-specific facets of MSE tested by web-platform-tests? Some of this is undoubtedly best tested using unit tests, but for the web-facing APIs (buffered/duration) will it be possible to write some shared tests?

 

This format-specific piece is covered by web-platform-tests, but those tests will need some additional work to run all supported media types (or at least swap MP4 with WebM in preference order of which format to test if both are supported, since PTS==DTS always in WebM). This is why Chrome passed the tests (WebM-based expectations match) but wouldn't pass MP4 versions of some of those same tests until this ships.

A bit off topic for this intent, but do you have a link to those tests? Sounds like it could be a legitimate case for https://github.com/GoogleChrome/wptdashboard/issues/98

[Matthew Wolenetz]

Experimentation is proceeding since late M67, into M69 dev/canary currently. I'll soon expand the experiment into M69 Beta with goal of eventually M69 Stable 100%.

Looking back, I realize there was just the intent to implement sent so far. Is it too late to express the possibility via an intent-to-ship that this feature may go 100% eventually in M69 Stable+Beta+Dev/Canary?

Matt

[Matthew Wolenetz]

Considering this is mostly an internal buffering bug being fixed for compliance with spec, not a new web feature implementation, I would prefer to not even need to continue the blink intent process (see also Philip's initial response). To be able to ship on-by-default, I'll have to fix the related layout tests' buffered range expectations (this is known work, and in scope of this bug-fix).

Unless I hear otherwise, I'll proceed on plan to increase experimentation in Chrome M69 with goal of eventually shipping with the compliant fix on-by-default in M69.

Matt

[fillmor...@gmail.com]

Hi Matt, I don't see this issue on ChromeStatus for Chrome 69 anymore:

https://www.chromestatus.com/features/schedule

Has it been delayed to a later version? I ask because this will impact DirecTV NOW users (see https://crbug.com/844799). Our timeline for delivering a fix is tight.

Thanks,

Chris

[Matthew Wolenetz]

Hi Chris,

Please see Dan's reply (https://bugs.chromium.org/p/chromium/issues/detail?id=844799#c49).

tl;dr: MseBufferByPts is in 50/50 experiments in M69 Beta, M70 Dev/Canary currently, and if data continues to show no significant regression, the goal is to increase the experimentation to 1% Stable M69, and eventually 100% Stable M69. If there are regressions, the fixes for them might require the feature's 100% Stable milestone to slip from M69.

Matt