Intent to Ship: Media Capabilities: decoding

[Husain Bengali]

Contact emails

chcunn...@chromium.org, mlam...@chromium.org, hben...@chromium.org

Explainer

Link to explainer (and readme)

Spec

https://wicg.github.io/media-capabilities/

This intent to ship is only for the decoding capabilities pieces of the API.

Link to tag review.

Summary

The API allows websites to get more information about the decoding abilities of the client. This enables more informed selection of media streams for the user, avoiding scenarios where the client is unable to smoothly and power efficiently decode a resolution that might have been picked based only on available bandwidth and screen size, for example.

Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/d/msg/blink-dev/EAqWCAdw1lI/zk3MvsBbCQAJ

Origin Trial feedback summary

YouTube, the only origin trial user that provided feedback, shared that they observed a non-trivial improvement in the mean time between rebuffers (MTBR) for the experimental group that used the Media Capabilities API to inform the adaptive bitrate stream resolution cap, compared to the control group.

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Demo link

Demo page link.

Risks

Interoperability and Compatibility

No compatibility risk. The API does not have a direct impact on other aspects of video playback. It is purely about helping web authors choose which videos to play.

Interoperability risk is moderate: Chrome is the first to implement but we have wide interest and support from other browser vendors.

Edge: No public signals

Firefox: Discussed at FOMS conference

Safari: In development (https://bugs.webkit.org/show_bug.cgi?id=181064)

Web developers: Strong public support (eg. Netflix contributes directly to the specification and YouTube ran an Origin Trial).

Ergonomics

Video streaming sites will use the Media Capabilities decodingInfo() API to decide which format and resolution (e.g. VP9 at 1080p) to play.

Sites may use the CSSOM View Screen interface in tandem with MediaCapabilities to decide what resolution of video is appropriate.

Sites may use the the powerEfficient value from a resolved decodingInfo() promise in coordination with the Battery Status API to decide when to prioritize power efficient formats and resolutions.

Once a format is selected, sites will often use Media Source Extensions to playback the stream.

The default usage of the API does not impact Chrome performance. The API is asynchronous and responding to the API queries is not resource intensive. There are no inherent process/thread requirements.

Activation

The inputs to the API are fairly low level descriptions of video formats. The target audience (streaming sites with number of formats available for any one video) is savvy about these details and can begin using the API immediately as-is.

Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

The web platform tests are here:

https://github.com/w3c/web-platform-tests/blob/master/media-capabilities/decodingInfo.html

The WPT cases cover validation of the inputs and resolution of the returned promise (either successfully or with appropriate error state).

In tests where the promise successfully resolves to produce a MediaCapabilitiesInfo object, the contents of that object (e.g. boolean powerEfficient) are not inspected because the values inside the MediaCapabilitiesInfo object are subject to change depending on the UA and the device running the tests.

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/5869632707624960

[Philip Jägenstedt]

 LGTM1 to ship navigator.mediaCapabilities.decodingInfo

I can recommend a thorough read of the explainer, it goes into great detail about some decisions, e.g., search for "compatibility" to see why the results are boolean. Lots of thought has also gone into how this API could be extended.

In the explainer, the semicolon in strings like "video/webm codec=vp9.0" is consistently omitted. I've sent a PR, but just want to confirm that the parsing of the MIME types will use the same code path as canPlayType? (Also filed an issue to use the new MIME type parser.)

Testing that, I also noticed that video.canPlayType("video/webm; codecs=vp9.0") is "probably", and yet navigator.mediaCapabilities.decodingInfo with the same string or variations thereof return false, type 'file' and 'media-source'. That seems like a bug, is it a known issue?

As expected, https://wpt.fyi/media-capabilities/decodingInfo.html is failing everywhere since the API is new, but I've confirmed it passing on Chrome Canary. Not Safari Technology Preview, though, yet. Hopefully the tests will help avoid accidental differences in the parsing of the framerate string and the like. Thanks for flying with web-platform-tests :)

Finally, I pointed here from https://github.com/w3ctag/design-reviews/issues/218 and don't think shipping should block awaiting further feedback.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMN6X7GG-Fw4-_%2BJYLHM9rPHbVCNLCPeNuCCwRaxKu-qzi99g%40mail.gmail.com.

[Chrome Cunningham]

Thanks Philip!

Testing that, I also noticed that video.canPlayType("video/webm; codecs=vp9.0") is "probably", and yet navigator.mediaCapabilities.decodingInfo with the same string or variations thereof return false, type 'file' and 'media-source'. That seems like a bug, is it a known issue?

I've replied on your PR, but copying here for completeness. The "vp9.0" format is an older style that is not allowed by the new API. The old string was too ambiguous about aspects of the codec (profile, level, bit depth, etc). The new format (see "Codecs Parameter String" here: https://www.webmproject.org/vp9/mp4/) requires these things to be explicitly specified (or chooses defaults where possible), and that allows us to give a confident supported = "true" or "false" rather than the "maybe" / "probably" style of canPlayType().

I'd be happy to update the spec with reference to the new MIME parsing. I'll spend some time looking over that and will look for a way to highlight that ambiguous mimes (which may be allowed by other APIs for historical reasons) are not allowed in MediaCapabilities. Suggestions welcome!

Chris

[Jochen Eisinger]

will this API be available to insecure contexts?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/9a657faf-87cd-4456-97c1-a5a92c82acfc%40chromium.org.

[Mounir Lamouri]

Yes. Media playback is allowed on insecure contexts and similar but less powerful APIs are too. Is there something in particular that would make this API not appropriate for insecure contexts?

-- Mounir

On Wed, 28 Feb 2018, at 10:14, Jochen Eisinger wrote:
> will this API be available to insecure contexts?
>
> On Wed, Feb 28, 2018 at 3:43 AM Chrome Cunningham <chcunn...@chromium.org>
> wrote:
>
> > Thanks Philip!
> >
> > Testing that, I also noticed that video.canPlayType("video/webm;
> >> codecs=vp9.0") is "probably", and yet
> >> navigator.mediaCapabilities.decodingInfo with the same string or variations

> >> thereof return false, type 'file' and 'media-source'. That *seems* like

> >> a bug, is it a known issue?
> >
> >
> > I've replied on your PR, but copying here for completeness. The "vp9.0"
> > format is an older style that is not allowed by the new API. The old string
> > was too ambiguous about aspects of the codec (profile, level, bit depth,
> > etc). The new format (see "Codecs Parameter String" here:
> > https://www.webmproject.org/vp9/mp4/) requires these things to be
> > explicitly specified (or chooses defaults where possible), and that allows
> > us to give a confident supported = "true" or "false" rather than the
> > "maybe" / "probably" style of canPlayType().
> >
> > I'd be happy to update the spec with reference to the new MIME parsing.
> > I'll spend some time looking over that and will look for a way to highlight

> > that *ambiguous* mimes (which may be allowed by other APIs for historical

> > reasons) are not allowed in MediaCapabilities. Suggestions welcome!
> >
> > Chris
> >
> > On Tuesday, February 27, 2018 at 2:01:04 AM UTC-8, Philip Jägenstedt wrote:
> >>
> >> LGTM1 to ship navigator.mediaCapabilities.decodingInfo
> >>
> >> I can recommend a thorough read of the explainer, it goes into great
> >> detail about some decisions, e.g., search for "compatibility" to see why
> >> the results are boolean. Lots of thought has also gone into how this API
> >> could be extended.
> >>
> >> In the explainer, the semicolon in strings like "video/webm codec=vp9.0"
> >> is consistently omitted. I've sent a PR

> >> <https://github.com/WICG/media-capabilities/pull/70>, but just want to

> >> confirm that the parsing of the MIME types will use the same code path as
> >> canPlayType? (Also filed an issue to use the new MIME type parser

> >> <https://github.com/WICG/media-capabilities/issues/69>.)

> >>
> >> Testing that, I also noticed that video.canPlayType("video/webm;
> >> codecs=vp9.0") is "probably", and yet
> >> navigator.mediaCapabilities.decodingInfo with the same string or variations

> >> thereof return false, type 'file' and 'media-source'. That *seems* like

> >> a bug, is it a known issue?
> >>
> >> As expected, https://wpt.fyi/media-capabilities/decodingInfo.html is
> >> failing everywhere since the API is new, but I've confirmed it passing on
> >> Chrome Canary. Not Safari Technology Preview, though, yet. Hopefully the
> >> tests will help avoid accidental differences in the parsing of the
> >> framerate string and the like. Thanks for flying with web-platform-tests :)
> >>
> >> Finally, I pointed here from
> >> https://github.com/w3ctag/design-reviews/issues/218 and don't think
> >> shipping should block awaiting further feedback.
> >>
> >> On Mon, Feb 26, 2018 at 11:09 PM Husain Bengali <hben...@chromium.org>
> >> wrote:
> >>
> >>> Contact emails
> >>>
> >>> chcunn...@chromium.org, mlam...@chromium.org, hben...@chromium.org
> >>>
> >>> Explainer
> >>>
> >>> Link to explainer

> >>> <https://github.com/WICG/media-capabilities/blob/master/explainer.md>
> >>> (and readme
> >>> <https://github.com/WICG/media-capabilities/blob/master/README.md>)

> >>>
> >>> Spec
> >>>
> >>> https://wicg.github.io/media-capabilities/
> >>>
> >>> This intent to ship is only for the decoding capabilities pieces of the
> >>> API.
> >>>

> >>> Link to tag review <https://github.com/w3ctag/design-reviews/issues/218>
> >>> .

> >>>
> >>> Summary
> >>>
> >>> The API allows websites to get more information about the decoding
> >>> abilities of the client. This enables more informed selection of media
> >>> streams for the user, avoiding scenarios where the client is unable to
> >>> smoothly and power efficiently decode a resolution that might have been
> >>> picked based only on available bandwidth and screen size, for example.
> >>>
> >>> Link to “Intent to Implement” blink-dev discussion
> >>>
> >>>
> >>> https://groups.google.com/a/chromium.org/d/msg/blink-dev/EAqWCAdw1lI/zk3MvsBbCQAJ
> >>>
> >>> Origin Trial feedback summary
> >>>
> >>> YouTube, the only origin trial user that provided feedback, shared that
> >>> they observed a non-trivial improvement in the mean time between rebuffers
> >>> (MTBR) for the experimental group that used the Media Capabilities API to
> >>> inform the adaptive bitrate stream resolution cap, compared to the control
> >>> group.
> >>>
> >>> Is this feature supported on all six Blink platforms (Windows, Mac,
> >>> Linux, Chrome OS, Android, and Android WebView)?
> >>>
> >>> Yes.
> >>>
> >>> Demo link
> >>>
> >>> Demo page link

> >>> <https://googlechrome.github.io/samples/media-capabilities/decoding-info.html>
> >>> .

> >>>
> >>> Risks
> >>>
> >>> Interoperability and Compatibility
> >>>
> >>> No compatibility risk. The API does not have a direct impact on other
> >>> aspects of video playback. It is purely about helping web authors choose
> >>> which videos to play.
> >>>
> >>> Interoperability risk is moderate: Chrome is the first to implement but
> >>> we have wide interest and support from other browser vendors.
> >>>
> >>> Edge: No public signals
> >>>
> >>> Firefox: Discussed at FOMS conference
> >>>
> >>> Safari: In development (https://bugs.webkit.org/show_bug.cgi?id=181064)
> >>>
> >>> Web developers: Strong public support (eg. Netflix contributes directly
> >>> to the specification and YouTube ran an Origin Trial).
> >>>
> >>> Ergonomics
> >>>
> >>> Video streaming sites will use the Media Capabilities decodingInfo() API
> >>> to decide which format and resolution (e.g. VP9 at 1080p) to play.
> >>>
> >>> Sites may use the CSSOM View Screen

> >>> <https://www.w3.org/TR/cssom-view-1/#screen> interface in tandem with

> >>> MediaCapabilities to decide what resolution of video is appropriate.
> >>>
> >>> Sites may use the the powerEfficient

> >>> <https://wicg.github.io/media-capabilities/#dom-mediacapabilitiesinfo-powerefficient>

> >>> value from a resolved decodingInfo() promise in coordination with the Battery

> >>> Status API <https://www.w3.org/TR/battery-status/> to decide when to

> >>> prioritize power efficient formats and resolutions.
> >>>
> >>> Once a format is selected, sites will often use Media Source Extensions

> >>> <https://www.w3.org/TR/media-source/> to playback the stream.

> >>>
> >>> The default usage of the API does not impact Chrome performance. The API
> >>> is asynchronous and responding to the API queries is not resource
> >>> intensive. There are no inherent process/thread requirements.
> >>>
> >>> Activation
> >>>
> >>> The inputs to the API are fairly low level descriptions of video
> >>> formats. The target audience (streaming sites with number of formats
> >>> available for any one video) is savvy about these details and can begin
> >>> using the API immediately as-is.
> >>>
> >>>
> >>> Is this feature fully tested by web-platform-tests

> >>> <https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md>?

> >>> Link to test suite results from wpt.fyi.
> >>>
> >>> The web platform tests are here:
> >>>
> >>>
> >>> https://github.com/w3c/web-platform-tests/blob/master/media-capabilities/decodingInfo.html
> >>>
> >>> The WPT cases cover validation of the inputs and resolution of the
> >>> returned promise (either successfully or with appropriate error state).
> >>>
> >>> In tests where the promise successfully resolves to produce a
> >>> MediaCapabilitiesInfo

> >>> <https://wicg.github.io/media-capabilities/#media-capabilities-info>

> >>> object, the contents of that object (e.g. boolean powerEfficient) are not
> >>> inspected because the values inside the MediaCapabilitiesInfo object are
> >>> subject to change depending on the UA and the device running the tests.
> >>>
> >>> Link to entry on the feature dashboard <https://www.chromestatus.com/>
> >>>
> >>> https://www.chromestatus.com/feature/5869632707624960
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "blink-dev" group.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMN6X7GG-Fw4-_%2BJYLHM9rPHbVCNLCPeNuCCwRaxKu-qzi99g%40mail.gmail.com

> >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMN6X7GG-Fw4-_%2BJYLHM9rPHbVCNLCPeNuCCwRaxKu-qzi99g%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>> .

> >>>
> >> --
> > You received this message because you are subscribed to the Google Groups
> > "blink-dev" group.
> > To view this discussion on the web visit

> > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/9a657faf-87cd-4456-97c1-a5a92c82acfc%40chromium.org
> > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/9a657faf-87cd-4456-97c1-a5a92c82acfc%40chromium.org?utm_medium=email&utm_source=footer>
> > .

> >
>
> --
> You received this message because you are subscribed to the Google
> Groups "blink-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send
> an email to blink-dev+...@chromium.org.

> To view this discussion on the web visit

> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALjhuicsJ5%2Bre8ZsbiSbvuRP1R6nsQao85ictyNJ%2BF%2Bs2nCo_w%40mail.gmail.com.

[Jochen Eisinger]

as the explainer points out, this API increases the fingerprint of the user, so I wonder whether we need to take steps to prevent this information from being easily accessible to anybody observing the network traffic.

[Chrome Cunningham]

Someone passively sniffing unsecured network traffic would not be able to query the API for fingerprinting purposes. At best (worst?) they may be able to see that a particular stream type (e.g. vp9 at 1080p) is downloaded, but requested streams are often a function of network bandwidth and real-time media performance (not covered by MediaCapabilities) such that it would be very difficult to assume anything about the returns of the MediaCapabilities API. 

To successfully fingerprint, you'd really need to run lots of MediaCapabilities queries with the hope of finding the specific limits of a given machine a across a wide enough range of codecs/profiles to uniquely identify it. Machines will generally fall into big buckets of common capabilities (i.e. every googler workstation can smoothly decode the vp9 at 1080p example above), so this isn't trivial. 

> an email to blink-dev+unsubscribe@chromium.org.

[Jochen Eisinger]

fair enough

lgtm3

> an email to blink-dev+...@chromium.org.

> To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALjhuicsJ5%2Bre8ZsbiSbvuRP1R6nsQao85ictyNJ%2BF%2Bs2nCo_w%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/407b1da5-92e9-4f5c-b0a1-0e8b57813c96%40chromium.org.

[Yoav Weiss]

LGTM from me as well

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALjhuif8wP%2BuETokfJwRZKqsaScuxhkecmkr%2B7nUGvHM2AJcsg%40mail.gmail.com.