Intent to Implement and Ship: Support for nonstandard constraints with standard syntax.

[Guido Urdaneta]

Contact emails

gui...@chromium.org

Spec

https://w3c.github.io/mediacapture-main/

Tag review:https://github.com/w3ctag/spec-reviews/issues/57

Summary

Chromium currently supports two different syntaxes for specifying video and audio constraints for the getUserMedia function. The old syntax has support for one set of constraints, including several nonstandard ones, and the new syntax has support for a different set of constraints, all of which are standardized. The new syntax gives developers more flexibility to control how their application will work, but the old syntax provides access to many more features. The intersection between the two sets is not empty, but neither set is a superset of the other.

This intent covers allowing the new syntax to support all the constraints supported by the old syntax.

Motivation

There are several features in Chromium that are available through nonstandard constraints (often prefixed by "goog" or "chrome"). By forcing developers to use the old syntax to access these, we are depriving them of the benefits of using the new syntax. To make matters worse, neither set of constraints is a superset of the other. For example, it is currently impossible for a developer to use the facingMode constraint to select front/back camera, and at the same time apply noise reduction to the resulting video track by using the googNoiseReduction constraint.

Ideally, we would like all constraints to be standardized and to support only the standard syntax, but the reality is that many applications take advantage of the nonstandard constraints and it is probably better to let those applications use the new syntax, so that we can start the process of deprecating the old syntax.

Note that this intent largely amounts to adding the following fields to the MediaTrackConstraintSet and MediaTrackSupportedConstraints dictionaries:

Audio and video

• ConstraintDOMString chromeMediaSource
  
  

Video only

• ConstraintLong googPowerLineFrequency

• ConstraintBool googNoiseReduction
  
  

Audio only:

• ConstraintBool chromeRenderToAssociatedSink

• ConstraintBool googAutoGainControl

• ConstraintBool googExperimentalAutoGainControl

• ConstraintBool googNoiseSuppression

• ConstraintBool googExperimentalNoiseSuppression

• ConstraintBool googHighpassFilter

• ConstraintBool googTypingNoiseDetection

• ConstraintBool googBeamforming

• ConstraintBool googArrayGeometry

• ConstraintBool googAudioMirroring

Note: in MediaTrackSupportedConstraints the fields are of type bool.

Our plan is to first add support for video constraints, for which spec-compliant algorithm support exists; and add support for audio constraints when the corresponding spec-compliant algorithm is supported.

Interoperability and Compatibility Risk

This intent does not improve or worsen interoperability with other browsers. Applications that use nonstandard constraints can continue to do so with the old syntax. Migrating those uses to the new syntax may actually improve interoperability since all other major engines that support getUserMedia are committed to supporting the standard syntax, while not all of them (e.g., Edge) support the old syntax.

If an application tries to use an unsupported constraint (standard or not) on any browser with the new syntax, the unsupported constraint would just be ignored.

A polyfill like adapter.js may be used to map nonstandard constraints across browsers. Note that Firefox supports nonstandard prefixed constraints such as mozNoiseSupression and mozAutoGainControl.

In terms of compatibility with existing applications, this change does not introduce any additional risk since such applications will have the same existing support regardless of whether this feature ships or not.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

OWP launch tracking bug

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/5838698172121088

Requesting approval to ship?

Yes.

[Philip Jägenstedt]

This is tricky. The current situation sounds pretty bad, and we need some way to make progress towards having only a single syntax for constraints and then an interoperable set of constraint names.

In the long term, failure to migrate everything to the new constraints syntax should mean that more people are left using the old syntax, which then continues to be an interoperability problem and possibly a compat problem for other engines. Avoiding that outcome is worth taking some (smaller) interop risks, I think.

To explore another option, what would it take for us to ship all of these constraints with the new syntax, but without any prefixes? In other words, what would it take to standardize them? At least some of them sound very generic and sensible, including the googNoiseReduction that you mention.

Also, do we have use counters or other metrics for how often each of these is used? If so, then it could be that standardizing just one or two of them would solve most of the problem, and buy us some more time to standardize the rest, or remove them if they actually don't make sense.

WDYT?

[PhistucK]

Microsoft mentions in its WebRTC blog post that they are looking at adding the screen capture functionality next.

Since the current situation is a prefixed (and protected) implementation by each of the vendors (mozMediaSource, chromeMediaSource), this is probably a good opportunity to try and standardize and move the specification (and the implementation) forward and move away from the prefixed (and protected and whitelist based) world, at least in the screen sharing area.

Regarding the other googFoo constraints, like Philip mentions, they must be measured. I would be really surprised if they were so widely used (after all, they are only supported in Chrome and perhaps Opera. Firefox has only two nonstandard constraints, more or less). Also, I think each one of those constraints should be measured separately. It is not enough to measure whether the old syntax is used, or whether any nonstandard constraint in the old syntax is used.

Also, since Google Hangouts probably employs most of them, it would be useful to ignore its use somehow (though I realize this is currently not a working practice). Remember that Hangouts has a Native Client part as well anyway, so it might be able to just employ it for things that are no longer supported in Chrome.

I really think they should not be rolled into the new constraints. And if you want to discourage the old syntax, just add deprecation warnings for googFoos (specifically) in the old syntax and just remove them. If the experience is worsen, those are probably WebRTC bugs that Chrome should fix anyway.

adapter.js should accommodate the remaining compatibility issues (chromeMediaSource will be renamed to mediaSource or whatever the specification ends up deciding).

But to get the ball rolling, use counters are really needed here (sorry, but I am kind of surprised they are not already measured, at least publicly in chromestatus.com. Such an intent would usually be backed by usage).

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[jbru...@mozilla.com]

+1 on standardizing. I'd like to apologize for mozAutoGainControl and mozNoiseSuppression. I've opened w3c/mediacapture-main#444 to try to standardize autoGainControl and noiseSuppression instead, reasoning they have two working implementations effectively. This would let us remove the moz ones, which I'd like to do.

[Guido Urdaneta]

Currently there are no counters, but we can add them.

Shipping without prefixes with the new syntax would technically be no problem since no one has ever used the prefixed forms with the new syntax, so nothing would break. I don't know how long it would take to have them in the standard, though.

[Philip Jägenstedt]

The counters will be great, although it'll be a long while before we have results so some tentative would be good. To get some idea, here are hit counts from httparchive:har.2017_03_15_chrome_requests_bodies:

• chromeMediaSource: 68
  
• googPowerLineFrequency: 0
  
• googNoiseReduction: 1
  
• chromeRenderToAssociatedSink: 3
  
• googAutoGainControl: 56
  
• googExperimentalAutoGainControl: 0
  
• googNoiseSuppression: 57
  
• googExperimentalNoiseSuppression: 0
  
• googHighpassFilter: 55
  
• googTypingNoiseDetection: 23
  
• googBeamforming: 0
  
• googArrayGeometry: 0
  
• googAudioMirroring: 22
  

These are encouraging numbers across the board, suggesting that usage doesn't go far into the long tail, and isn't in some widely used WebRTC library. If use counters show high usage for any of these, it's probably because of usage on something like hangouts.com.

Prioritizing standardization roughly in order of above usage probably makes sense. Jan-Ivar, of the above things with hits >10, are there any that you don't know what they mean, or already know would be hard to support in Gecko?

[PhistucK]

Note that chromeMediaSource depends on a Chrome extension, so there is at least a way to contact the authors directly. So the relatively high usage should not be so scary.

☆PhistucK

[PhistucK]

Checking GitHub instead of HTTPArchive (which only crawls home pages, right?), googAutoGainControl has many matches, however, most of them set it to false (which is the default anyway, right?).

I have not researched others.

☆PhistucK

[Philip Jägenstedt]

Right, httparchive crawls the Alex top 1 million, but only has ~500k sites in the results. I don't know if it's just the top 500k, a random subset, or something else. +Patrick Meenan might know.

I usually don't bother checking GitHub, because there are usually so many instances of the same code from forks, and AFAIK no way to download all the hits in order to categorize them. And plenty of stuff is abandoned and probably not used in the wild.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Patrick Meenan]

It's the top 500k from Alexa but as noted, just the landing pages. 

[Rick Byers]

If either Edge or Firefox is interested in implementing any of these for compatibility, then perhaps we should just add them to the compat spec?

I also like the plan to avoid shipping the ones that really aren't necessary based on usage, and otherwise ship them with unprefixed names (ideally based on getting them into the WebRTC spec, but the compat spec or any other document that precisely describes their definition is good enough as a stop gap IMHO - Chrome already ships them, better to just be clear about it).

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Simon Pieters]

On Thu, 18 May 2017 23:46:49 +0200, Rick Byers <rby...@chromium.org> wrote:

> If either Edge or Firefox is interested in implementing any of these for
> compatibility, then perhaps we should just add them to the compat spec

> <https://compat.spec.whatwg.org/>?

I think it's generally better to specify things in the main spec for a
feature, even if the thing has a vendor prefix or is something that
shouldn't be used. Delta specs are sub-optimal for the goal of reaching
interop because it is easy to miss that the delta spec exists when
implementing a feature or when the main spec is extended, refactored, and
so on.

Put another way, the following situations are ordered from best to worst:

1. The standard for a feature documents everything necessary for web
compat, and no ugly warts are necessary for web compat.
2. The standard for a feature documents everything necessary for web
compat, warts and all.
3. The standard for a feature documents non-ugly things necessary for web
compat, and a separate document specifies the warts necessary for web
compat.
4. The standard for a feature documents non-ugly things necessary for web
compat, and the warts necessary for web compat are not specified anywhere.

I don't know what the web compat situation is like here, but in general I
wouldn't want to jump straight to (3) without first evaluating if (1) is
possible and, if not, trying to do (2).

> I also like the plan to avoid shipping the ones that really aren't
> necessary based on usage, and otherwise ship them with unprefixed names
> (ideally based on getting them into the WebRTC spec, but the compat spec
> or
> any other document that precisely describes their definition is good
> enough
> as a stop gap IMHO - Chrome already ships them, better to just be clear
> about it).

I guess we're in agreement about the ideal... :-)

--
Simon Pieters
Opera Software

[jbru...@mozilla.com]

On Friday, May 19, 2017 at 5:24:03 AM UTC-4, Simon Pieters (zcorpan) wrote:On Thu, 18 May 2017 23:46:49 +0200, Rick Byers <rby...@chromium.org> wrote:

> If either Edge or Firefox is interested in implementing any of these for
> compatibility, then perhaps we should just add them to the compat spec
> <https://compat.spec.whatwg.org/>?

I think there's some confusion here. Chrome only supports goog constraints in its non-spec constraints syntax ({mandatory{}, optional:[{}]}). Neither Edge nor Firefox use that syntax. I wouldn't expect a reversal on that (it would be over constraints more important than these).

This issue is about introducing several goog constraints anew:

• in navigator.mediaDevices.getSupportedConstraints()
• in track.getSettings(),
• in track.getCapabilities(),
• in track.getConstraints(),
• in track.applyConstraints(), and
• accepting them in getUserMedia's new constraints syntax ({foo: {min, max, exact, ideal}}).
  

There is no web compat reason for doing so. We agreed earlier in this thread that the path to interop instead is to standardize the most popular ones ASAP.

I had on the May 2nd WebRTC interim agenda to standardize autoGainControl and NoiseSuppression in mediacapture spec itself, but the WG ran out of time to discuss it. Hopefully they'll revisit in June instead.

Failing that, or maybe regardless of that, maybe it's time to open a delta spec. Sure, main specs beat delta-specs, but delta-specs beat no spec.

[jbru...@mozilla.com]

Correction: autoGainControl and noiseSuppression are now in the mediacapture-main spec (github tip) - Two down. 🎉

[Philip Jägenstedt]

Did you mean a spec to document goog-prefixed constraints specifically, or something broader?

If we had bits of WebRTC APIs that 2+ vendors want to keep indefinitely but is rejected by the WebRTC WG, then I think adding to the compat spec or starting a new WebRTC compat spec would be a good idea, but I don't think we're quite there yet.

With autoGainControl and noiseSuppression now standardized, next my upthread list would be googHighpassFilter. Does Firefox have anything like that?

[jbru...@mozilla.com]

I meant if mediacapture-main at some point decides to stop accepting new proposals; a place to house further attempts at standardization.

Again, I don't see this qualifying as web compat, unless we mean keeping the old mandatory/optional syntax around? (I hope not)

> next my upthread list would be googHighpassFilter. Does Firefox have anything like that?

No we don't have a constraint like that. Is the intent to be able to disable a high-pass filter feature of a hardware mic? Beyond that, I don't see much use for this since a highpass filter is trivial to do in Web Audio.

FWIW I actually like Chrome's existing approach of inferring intent from the other constraints, like echoCancellation, because people who want raw audio will curse every time we add a new constraint otherwise.

[Philip Jägenstedt]

On Mon, May 22, 2017 at 9:01 PM <jbru...@mozilla.com> wrote:

I meant if mediacapture-main at some point decides to stop accepting new proposals; a place to house further attempts at standardization.

Again, I don't see this qualifying as web compat, unless we mean keeping the old mandatory/optional syntax around? (I hope not)

It's not a compat concern in the sense that more or less existing sites will work depending on this, but our choices here do affect the prospects of eventually removing the old constraints API surface. Basically, the more things that can only be done using the old API, the more reason people have to use it, and the harder it'll be to remove.

However, adding all the constraints without the prefixes would solve the problem as well, so I think we should standardize everything that does something useful and we don't want to remove.

 

> next my upthread list would be googHighpassFilter. Does Firefox have anything like that?

No we don't have a constraint like that. Is the intent to be able to disable a high-pass filter feature of a hardware mic? Beyond that, I don't see much use for this since a highpass filter is trivial to do in Web Audio.

FWIW I actually like Chrome's existing approach of inferring intent from the other constraints, like echoCancellation, because people who want raw audio will curse every time we add a new constraint otherwise.

Maybe this behavior should be spelled out in the spec, then? Guido?

[jbru...@mozilla.com]

Do we need to? Constraints don't have defaults, because they're externalized requirements of a requesting app. The spec gives UAs final say on what device settings to use at any time:

"Without provided track constraints, implementations are free to select a source's settings from the full ranges of its supported capabilities. Implementations may also adjust source settings at any time within the bounds imposed by all applied constraints."

Basically, if you care about a setting, constrain it. This grants UAs the necessary freedom to arbitrate simultaneous access to what may be singular hardware device settings.

.: Jan-Ivar :.

[Philip Jägenstedt]

Don't web developers still have to understand and adapt to the fact that one constraint renders lots of others no-ops? At the very least, if it doesn't work the same in all implementations, then it'll be confusing.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/326e5e1a-fd0e-4e7b-98b2-606993c84e44%40chromium.org.

[Guido Urdaneta]

By design, constraints are not intended to work the same in all implementations. Yes, it is confusing and it makes testing across browsers difficult.

The thing is, if a constraint is not specified or has a value which is a range, the browser is allowed to use any setting within the range and is allowed to change the setting as long as it stays within the range. In these cases, different browsers implement different policies which lead to different behavior, which can lead to confusion.

With regards to the echoCancellation constraint in chrome, it does not turn the other constraints into no ops.

With the new syntax, it is currently the only constraint that affects Chrome's audio-processing module, so when it is set to false, it disables all audio processing.

With the old syntax (and hopfully the new one as well in the future) there are many other (goog-prefixed) constraints that affect the APM. In this case, when echoCancellation is explicitly set to false, it changes the default value for the other audio-processing properties to OFF. However, if any of the goog constraints is specified, its value is used to set the corresponding property regardless of the echoCancellation constraint. This behavior is according to spec and if it didn't do that it would also be according to spec, hence the possible confusion.

[jbru...@mozilla.com]

To clarify, the spec does lay down some rules on constraints. E.g. `exact` constraints must be met or throw OverconstrainedError, vs. `ideal` constraints where the browser "SHOULD use the one with the smallest fitness distance", where fitness distance is a well-defined algorithm.

A browser choosing raw audio in response to {echoCancellation: false} violates neither the MUST nor the SHOULD.
A browser choosing raw audio in response to {echoCancellation: false, autoGainControl: true } would violate the SHOULD.
A browser choosing raw audio in response to {echoCancellation: false, autoGainControl: {exact: true}} would violate the MUST, unless it threw.

The biggest loophole, frankly, is the reduction algorithm is applied to a pool of all possible available settings, but each browser is free to define this pool, and include invented virtual settings that aren't native to the device.