Intent to Implement: HTMLMediaElement latencyHint
Chris Cunningham
Contact emails
Explainer
https://github.com/WICG/media-latency-hint/blob/master/explainer.md
Design doc/Spec
Not yet written. Will update the thread when available.
Inclined to skip TAG review - the API adds a single attribute to HTMLMediaElement. But happy to pursue it folks would like.
Summary/Motivation
The latencyHint attribute gives web authors influence over how much decoded output must be buffered for a user agent to begin playback or resume following a seek.
Today the buffer thresholds are a UA-specific implementation detail. For example, Chromium defaults to require 200 milliseconds of decoded audio and 3 decoded video frames. In the absence of any site-provided hint, UAs will try to balance smoothness and latency, but generally favor smoothness as non-real-time video use-cases (movies, tv, vlogging, ...) are the most popular types of content.
Most sites continue to be well served by the default values, but we've heard from several sites/companies (Twitch, Sony, Stadia) that they need to pick lower values to unlock more interactive use cases. We would also like to experiment with sites like YouTube who might use the attribute to set values that are higher than the default to improve playback smoothness.
Risks
Interoperability and Compatibility
Sony, Twitch, Edge and others have expressed support for this proposal in the discourse discussion here:
Note: the proposal went through several API shapes. The current shape is proposed after extensive some discussion in that thread.
Edge: Public support
Firefox: Public support for concept. Not opinionated about API details.
Safari:
Supported a higher level API with content hint categories including low-latency and other things, but they're not currently able to implement a latency hint irrespective of API shape given limited control over the underlying AVFoundation media pipeline. The contentHint proposal was discussed and resolved to be abandoned at TPAC.
Web / Framework developers: Supportive (see discourse).
Ergonomics
Are there any other platform APIs this feature will frequently be used in tandem with?
It is likely to be used with MediaSourceExtensions, but this is not a requirement (discussed at TPAC). MSE is expected to have separate features added for latency sensitive apps, but those will focus on media demuxing (MSE is a demuxing abstraction, whereas this latencyHit is about rendering/decoding configuration).
Could the default usage of this API make it hard for Chrome to maintain good performance (i.e. synchronous return, must run on a certain thread, guaranteed return timing)?
No. The default behavior for this API will be that Chrome behaves as it does today.
Activation
Will it be challenging for developers to take advantage of this feature immediately, as-is?
No
Would this feature benefit from having polyfills, significant documentation and outreach, and/or libraries built on top of it to make it easier to use?
Documentation/outreach always welcome. Libraries/polyfills not necessary.
Debuggability
The API is a set-and-forget hint. The implementer will apply the hint in a best effort fashion. There will be some reasonable limits to what UAs can achieve based on platform and memory limitations. Chrome will log debugging information about the hint to chrome://media-internals.
Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
Yes.
Link to entry on the feature dashboard
https://www.chromestatus.com/feature/5160704082444288
Requesting approval to ship?
No