Intent to Implement and Ship MediaStreamTrack.applyConstraints()
Guido Urdaneta
Contact emails
Explainer
The following document serves a good explainer for this API (and the whole MediaStreamTrack Constraints API): https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints
Spec
https://w3c.github.io/mediacapture-main/
TAG review: https://github.com/w3ctag/design-reviews/issues/57
Summary
applyConstraints() allows applying new constraints to an existing MediaStreamTrack. This gives applications the ability to, for example, change the resolution or frame rate of a track obtained with getUserMedia().
Motivation
Spec compliance and interoperability with other engines.
Risks
Interoperability and Compatibility
This will improve interoperability with Firefox, Edge and Safari, which already ship functional versions of these APIs, so the interoperability risk is low. The main risk comes from the fact that the constraints API provides plenty of implementation freedom for user agents. For example, if an applyConstraints() call requests a maximum height of 1080, the user agent is allowed to configure the track with any resolution whose height does not exceed 1080. Different implementations can produce very different results for the same input without violating the spec and this might surprise users.
The compatibility risk is low too, but not completely zero. The reason is that this API has already shipped in Blink for the Image Capture API using a non-spec compliant implementation of the AP. As shipped for Image Capture, the constraints processing algorithm is not spec compliant and successful applyConstraints calls are resolved with a value consisting of the applied constraints instead of with no value, which is what the spec mandates. The constraints-processing algorithm implemented for Image Capture constraints is quite different from the specification.
If used with non-Image Capture constraints, the current implementation of applyConstraints is basically a noop and, depending on the constraints string passed, applyConstraints either resolves or rejects with a DOMException with name "NotSupportedError".
Our plan is to first ship a version of applyConstraints() such that:
It will reject with OverconstrainedError if Image Capture and Media Capture properties are specified at the same time. This is not the most desirable behavior, but it is at least spec compliant.
If only Image Capture properties are specified, it will use the existing non-spec compliant behavior.
If only Media Capture properties are specified, it will use the new spec-compliant implementation.
If no properties are specified, both algorithms will run and applyConstraints() will always resolve, as the spec mandates.
This initial version should have very low compatibility risk since the only significant behavior change will be that applyConstraints will work with media-capture constraints instead of rejecting with a DOMException.
A follow-up version will integrate Image Capture constraints into the spec-compliant algorithm so that the Image Capture constraints implementation becomes spec compliant as well. This might cause compatibility problems for applications that rely on the existing nonstandard implementation.
Also, we plan to first ship full support for video tracks (preferably in M63) and very limited, but technically spec compliant, support for audio tracks. For audio tracks, the first version will reject with OverconstrainedError any input that cannot be satisfied with the current track settings. applyConstraints() will succeed with other changes, but will not change the settings of the track. While not very useful, this first version will be spec compliant. A follow-up version will add better support for audio tracks.
Note also that this intent covers applyConstraints() for MediaStreamTracks produced by getUserMedia() device capture. Other types of track will be covered by separate intents. The implementation for those will not change track settings and will resolve or reject with an OverconstrainedError object according to the input, as per the spec.
Edge: Shipped
Firefox: Shipped
Safari: Shipped
Web developers: Positive
Ergonomics
Are there any other platform APIs this feature will frequently be used in tandem with?
applyConstraints will be used frequently together with getUserMedia().
Could the default usage of this API make it hard for Chrome to maintain good performance (i.e. synchronous return, must run on a certain thread, guaranteed return timing)?
No.
Activation
Will it be challenging for developers to take advantage of this feature immediately, as-is?
No.
Would this feature benefit from having polyfills, significant documentation and outreach, and/or libraries built on top of it to make it easier to use?
Since this API is already supported by many browsers and is from a relatively mature spec, there is plenty of documentation on how to use this API.
Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
Yes.
Is this feature fully tested by web-platform-tests?
Testing this API in web platform tests presents challenges related to the fact that behavior depends on the media-capture hardware present in the system.
The current WPT suite has very limited support for applyConstraints. However, we plan to provide additional tests that mimic existing tests that pass constraints to getUserMedia(). The new tests would invoke getUserMedia without constraints, and would invoke applyConstraints with the same arguments as the existing getUserMedia tests.
Also, since most of the initial implementation will be in content, browser tests will also be provided.
Link to entry on the feature dashboard
https://www.chromestatus.com/feature/5131064242601984
Requesting approval to ship?
Yes.
Philip Jägenstedt
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/637b228c-de2d-4241-8d3b-5bc8641aa039%40chromium.org.
Rick Byers
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdb4bXRaeZ0nuYQEGM7A8E92DdTZaZOxg6SBV54XwhaYA%40mail.gmail.com.
Dimitri Glazkov
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY9eMTWW68yYyukynLTmBbUFrJbeL5DUpAAj494jE61O7Q%40mail.gmail.com.