Intent to Ship: HTMLVideoElement.getVideoPlaybackQuality()

225 views
Skip to first unread message

Chris Cunningham

unread,
Nov 22, 2019, 7:43:53 AM11/22/19
to blink-dev, Mounir Lamouri

Contact emails

chcunn...@chromium.org, mlam...@chromium.org 


Explainer

Way back in 2016, this API was extracted from the Media Source Extensions (MSE) specification (discussion).


Spec

https://w3c.github.io/media-playback-quality/


A tag review was not requested (given the age of the spec, and existing implementations).


Summary

This API tells sites how a video playback is performing. When performance is poor, sites may use the information to reduce the video bitrate / framerate / resolution for a better user experience. 


Edge, Firefox, and Safari shipped the API quite some time (years) ago. Chrome implemented behind a flag (MediaSourceExperimental) and shipped parts of the API as webkit prefixed attributes: webkitDecodedFrameCount and webkitDroppedFrameCount.


Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.


Debuggability

The API is a simple getter for a bag of performance metrics. Debugging is generally not expected. 


Poor video performance may sometimes be the fault of the site. Sites should use existing page profiling tools to ensure optimal performance. Poor performance may also be due to the device's limited capabilities (CPU, GPU, memory, load). In both cases, sites should monitor the playback quality stats and dynamically select a lower complexity of video if performance is poor.



Risks

Interoperability and Compatibility

No interop risk. 


Edge: Shipped

Firefox: Shipped

Safari: Shipped

Web / Framework developers: Supportive (used across numerous video sites) 


Ergonomics

The API will frequently be used with MSE (which facilitates dynamic changes to video resolution / framerate / bitrate). 


The API is functional and widely used, but it's not easy. As is, sites must poll the API at an interval of their choosing to learn when new frames are dropped. They may externally track the ratio of dropped/decoded video frames in a windowed average to evaluate performance over time.


But the above is also true of our existing prefixed attributes. The pain of prefixing is well known. Additionally, the new API is preferred because it returns related statistics in an atomic snapshot.


Once shipped, we will explore new events / additions to improve on the ergonomic pitfalls above. 


Activation

Shipping this API will allow developers to delete polyphils that bridged the gap between Chrome and other UAs.


Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.


Interface test

https://github.com/web-platform-tests/wpt/blob/master/media-playback-quality/idlharness.window.js

https://wpt.fyi/results/media-playback-quality/idlharness.window.html?label=master&label=experimental&aligned


Behavior test (dated assumptions - I will update this prior to shipping)

https://github.com/web-platform-tests/wpt/blob/master/media-source/mediasource-getvideoplaybackquality.html

https://wpt.fyi/results/media-source/mediasource-getvideoplaybackquality.html?label=experimental&label=master&aligned


Entry on the feature dashboard

https://www.chromestatus.com/feature/5687791428042752


Rick Byers

unread,
Nov 26, 2019, 4:05:46 PM11/26/19
to Chris Cunningham, blink-dev, Mounir Lamouri
Matching what the other engines have already shipped seems like it should be uncontroversial to me (despite the lack of TAG review etc.). Awhile back I found this was the ONE API that all the other engines had but chromium did not (though confluence doesn't think Safari has it now for some reason, and I'm getting some other results since last I checked. Thanks for pushing on it!

But is the dictionary really the same shape across all these engines? Eg. I see Blink's IDL file has a 'corruptedVideoFrames' member not present in the spec/tests.

Note that MDN (and so Caniuse) appear to be incorrect in saying Chrome has long had support for this API. I filed the feedback form with a link to the chromestatus entry to report the error.

Rick

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALG6eSrD1TqCv72QDNR5SDYaq5GfX2kgUSLdkrp08%2BVkb_oYZg%40mail.gmail.com.

Mounir Lamouri

unread,
Nov 26, 2019, 5:10:20 PM11/26/19
to Rick Byers, Chris Cunningham, blink-dev
Replying for Chris as he isn't around this week. The spec dropped "corruptedFrames" indeed following a discussion at TPAC. The property was never fully implemented. Chrome had it behind a flag returning 0 so did other browsers (some tried to report something IIRC). Dropping it from the spec was tracked in https://github.com/w3c/media-playback-quality/issues/5 and the PR was approved by Mozilla and Apple.

-- Mounir

Rick Byers

unread,
Nov 26, 2019, 5:37:59 PM11/26/19
to Mounir Lamouri, Chris Cunningham, blink-dev
Great, so presumably it'll be dropped from our implementation (or put behind a flag) before shipping too, right?

What does the dictionary currently returned by Safari and Firefox look like?

Boris Zbarsky

unread,
Nov 26, 2019, 11:53:47 PM11/26/19
to blink-dev
On 11/26/19 5:37 PM, Rick Byers wrote:
> What does the dictionary currently returned by Safari and Firefox look like?

For Firefox as of today [1] it looks like this:

interface VideoPlaybackQuality {
readonly attribute DOMHighResTimeStamp creationTime;
readonly attribute unsigned long totalVideoFrames;
readonly attribute unsigned long droppedVideoFrames;
readonly attribute unsigned long corruptedVideoFrames;
};

So `corruptedVideoFrames` is still there. I don't obviously see a bug
filed to remove it...

-Boris

[1]
https://searchfox.org/mozilla-central/rev/cce8b90aece0f42e5025e45282de16066eeaa662/dom/webidl/VideoPlaybackQuality.webidl#13-21

Chris Cunningham

unread,
Nov 27, 2019, 4:43:50 PM11/27/19
to Boris Zbarsky, blink-dev
:P I'm only mostly OOO

> Great, so presumably it'll be dropped from our implementation (or put behind a flag) before shipping too, right?

We might consider a slower deprecation. While the attribute has no clear implementation, that doesn't mean sites out there aren't calling it (ex: if only to log it). We never shipped, so we don't know what its usage looks like atm. 

> What does the dictionary currently returned by Safari and Firefox look like?

Like Firefox, Safari also includes it for the time being. 

> I don't obviously see a bug filed to remove it...

I'm can file one if you like? I plan to add a wpt test to fail if its detected - not sure if that automatically triggers a bug report. 

Chris


--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3fcff193-87af-df0a-f38f-d61029254b86%40mit.edu.

Boris Zbarsky

unread,
Nov 27, 2019, 4:56:09 PM11/27/19
to Chris Cunningham, blink-dev
On 11/27/19 4:43 PM, Chris Cunningham wrote:
> I'm can file one if you like?

Please! Ideally, this would be part of the PR process in general, as it
is for the HTML and DOM specs.

> I plan to add a wpt test to fail if its
> detected - not sure if that automatically triggers a bug report.

Unfortunately, it does not.

-Boris

Yoav Weiss

unread,
Nov 28, 2019, 4:17:21 PM11/28/19
to Boris Zbarsky, Chris Cunningham, blink-dev
LGTM1 for the slow roll out plan with bugs filed on Firefox and Safari.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Daniel Bratell

unread,
Nov 28, 2019, 4:26:19 PM11/28/19
to Yoav Weiss, Boris Zbarsky, Chris Cunningham, blink-dev

LGTM2

/Daniel

Chris Harrelson

unread,
Nov 28, 2019, 6:16:52 PM11/28/19
to Daniel Bratell, Yoav Weiss, Boris Zbarsky, Chris Cunningham, blink-dev
LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3a2a1e3e-51ab-38d7-d120-bb6da579cba2%40gmail.com.

Rick Byers

unread,
Nov 29, 2019, 1:33:14 PM11/29/19
to Chris Cunningham, Boris Zbarsky, blink-dev
On Wed, Nov 27, 2019 at 4:43 PM Chris Cunningham <chcunn...@chromium.org> wrote:
:P I'm only mostly OOO

> Great, so presumably it'll be dropped from our implementation (or put behind a flag) before shipping too, right?

We might consider a slower deprecation. While the attribute has no clear implementation, that doesn't mean sites out there aren't calling it (ex: if only to log it). We never shipped, so we don't know what its usage looks like atm. 

I see, you're worried about usage behind a feature-detect for the getVideoPlaybackQuality which could break if this member isn't present?

The launch process requires that we only ship new APIs (which includes dictionary members) which are defined in some current spec somewhere. So either we should add this (broadly supported) API back into the draft spec (even if you want to mark it as 'deprecated for legacy purposes only' - eg. precedent), or we should attempt shipping without it and see if we get any reports of breakage. Either way is fine with me (LGTM4 - not that you need it).

Chrome Cunningham

unread,
Dec 6, 2019, 8:41:20 PM12/6/19
to blink-dev, chcunn...@chromium.org, bzba...@mit.edu
>> I'm can file one if you like?

> Please!  Ideally, this would be part of the PR process in general, as it
> is for the HTML and DOM specs.

Bug filed.

I'll do this for my PRs going forward. 

> I see, you're worried about usage behind a feature-detect for the getVideoPlaybackQuality which could break if this member isn't present?

Yes. 

> The launch process requires that we only ship new APIs (which includes dictionary members) which are defined in some current spec somewhere. So either we should add this (broadly supported) API back into the draft spec (even if you want to mark it as 'deprecated for legacy purposes only' - eg. precedent), or we should attempt shipping without it and see if we get any reports of breakage.

Understood. What if I leave it in while it rolls out to beta and then remove it if use counters show its safe? Alternatively, happy to add the deprecated note to the spec.

Yoav Weiss

unread,
Dec 9, 2019, 5:14:13 AM12/9/19
to Chrome Cunningham, blink-dev, Boris Zbarsky
The main part that concerns me about that is that typically we don't consider beta numbers to be representative, so even if usage in beta is low, it could bump up once we reach stable.

IMO the safest route would be: send a PR to the spec (with the caveat that this could be a very temporary spec change), land the attribute with use counters, wait till it spent a few weeks in stable and then revert the PR if possible.

Does that make sense?
 
Alternatively, happy to add the deprecated note to the spec.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Chrome Cunningham

unread,
Dec 9, 2019, 4:53:52 PM12/9/19
to blink-dev, chcunn...@chromium.org, bzba...@mit.edu
> Does that make sense?

Sounds good. Spec PR to mark as deprecated here https://github.com/w3c/media-playback-quality/pull/21
Reply all
Reply to author
Forward
0 new messages