Intent to Ship: MediaStreamTrack Constraints API (part 1)

[Harald Alvestrand]

Contact emails

h...@google.com

Spec

http://w3c.github.io/mediacapture-main/getusermedia.html#media-track-constraints and surrounding materials

The TAG has reviewed this specification as part of the WebRTC effort reviews.

Summary

This I2S covers the basic constraints interface: The basic/advanced structure with max/min/exact, plus

the getSupportedConstraints and track.getConstraints functions.

Three functions/capabilities are left for later and not covered by this I2S:

- applyConstraints

- ideal (which includes "naked values") (blocked on crbug.com/524424)

- getSettings

Intent to implement: https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/s-NMzjSfVQs

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Demo link

None.

Debuggability

Not believed relevant.

Interoperability and Compatibility Risk

Somewhat low.

The omission of "ideal" will cause us to ignore attempts to specify ideal constraints at the API. It will not cause calls to fail that would have succeeded, or vice versa, but may cause the settings chosen to be surprising.

Firefox and Edge implement constraints according to spec.

OWP launch tracking bug

https://crbug.com/543997

Entry on the feature dashboard

https://www.chromestatus.com/feature/5807737287999488

[PhistucK]

On Wed, Apr 20, 2016 at 5:06 PM, 'Harald Alvestrand' via blink-dev <blin...@chromium.org> wrote:

It will not cause calls to fail that would have succeeded, or vice versa, but may cause the settings chosen to be surprising.

Can you elaborate?​

☆PhistucK

[Harald Alvestrand]

There is a separate experimental flag that controls whether Blink will honor setting of the "ideal" values. (Naked values aren't supported yet.)

So the ideal value will simply disappear (according to the rules for dictionaries), and the Chrome implementation will never see it.

So, on a system that prefers 1024x768, but where 640x480 is also available, if you call

getUserMedia({video: { width: { ideal: 640 }}})

you will get exactly the same result as for

getUserMedia({video: { width: { }}})

which means that you get the resolution the platform chooses for you (1024x768)

while if you call

getUserMedia{video: { width: { exact: 640 }})

you will get 640x480.

It is a result that is permitted by the spec, but may be surprising to the user.

[PhistucK]

Thank you for the clarification.

Regarding naked value (I suppose this is getUserMedia({video: true}), right?), the old implementation did support this, so is Blink dropping support for naked values?

☆PhistucK

[Harald Alvestrand]

No, naked value is getUserMedia{{video: { height: 480 }})

getUserMedia({video: true}) will continue to be supported, just like getUserMedia({video: { mandatory: { maxHeight: 480 }}}) (old-style).

I'm trying to make sure we don't break existing code.

[PhistucK]

Do you happen to know how prevalent naked values are in the wild?

☆PhistucK

[Harald Alvestrand]

No app that runs on chrome uses them, since they were not passed before. Some apps that run on both Firefox and chrome and do not use adapter.js might use them. No idea how many these are.

[Philip Jägenstedt]

Is the scope of this precisely the bits guarded by [RuntimeEnabled=MediaConstraints] in IDL files?

In MediaTrackConstraintSet.idl there's a comment about the types with reference to https://crbug.com/524424 that looks a bit troubling. If I'm reading it right, only the range forms are supported, so getUserMedia({video: { width: { exact: 640 }}}) would work, while getUserMedia({video: { width: 640 }}) would throw an exception because 640 isn't a dictionary. If correct, that seems like a potentially serious interop problem, can you confirm?

(As an aside, it's interesting that getSupportedConstraints() is basically a workaround for the fact that it's tricky to feature detect dictionary members, especially in a case like this where one would have to make an async request.)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Harald Alvestrand]

Yes, the scope is what's guarded by [RuntimeEnabled=MediaConstraints] (unless I made a mistake).

Yes, crbug 524424 troubles me too, and haraken has promised to try to address it this quarter, but it's apparently not a trivial change to the WebIDL compiler. It's part of the reason why I installed a special flag for [ideal] (the rest of the reason is that I haven't implemented the "ideal" algorithm yet).

Yes, the result of getUserMedia({video: {width: 400}}) will be TypeError (just tried it). I think this is slightly better than silently ignoring the attempt to set the constraint when this form is not supported.

BTW - getSuppotedConstraints() is synchronous. In fact, it's the simplest function I've ever implemented in Blink - it's all defined in the IDL file except an empty C++ function.

Harald

[Rick Byers]

On Tue, Apr 26, 2016 at 5:04 PM, 'Harald Alvestrand' via blink-dev <blin...@chromium.org> wrote:

Yes, the scope is what's guarded by [RuntimeEnabled=MediaConstraints] (unless I made a mistake).

Yes, crbug 524424 troubles me too, and haraken has promised to try to address it this quarter, but it's apparently not a trivial change to the WebIDL compiler. It's part of the reason why I installed a special flag for [ideal] (the rest of the reason is that I haven't implemented the "ideal" algorithm yet).

Yes, the result of getUserMedia({video: {width: 400}}) will be TypeError (just tried it). I think this is slightly better than silently ignoring the attempt to set the constraint when this form is not supported.

BTW - getSuppotedConstraints() is synchronous. In fact, it's the simplest function I've ever implemented in Blink - it's all defined in the IDL file except an empty C++ function.

Note that this issue tracks adding a generic mechanism to WebIDL for this: https://github.com/heycam/webidl/issues/107

[Harald Alvestrand]

Note that the missing feature in crbug 524424 wouldn't be helped by feature detction in dictionaries - it's about this construct:

dictionary foo {

   (long OR bar) foomember

}

dictionary bar {

  ...

}

that is - a dictionary as member of an union inside another dictionary.

[Philip Jägenstedt]

bashi@, do you have some idea about the difficulty of fixing issue 524424, at least enough to support the IDL in https://w3c.github.io/mediacapture-main/getusermedia.html#media-track-constraints?

[Kenichi Ishibashi]

https://codereview.chromium.org/953123003/ was an attempt to fix it, but Jens pointed out that the solution would bring build flakiness. One workaround for build dependency would be explicitly putting all file names for union types in .gn/.gypi but I'm not sure we want to put such burden on developers (they will need to add LongOrBar.{cpp,h} when they use (long or bar) in their IDL files).

[Harald Alvestrand]

What's the bug filed for GN on this issue (side effect .h files)?

It seems like a generic issue with files generated and depended on as a side effect - my memory says "protobufs" and "BUILD rules with side effects", but I'm not sure what the status is there at the moment.

[Philip Jägenstedt]

If the build dependency workaround works and the resulting build would behave as required by WebIDL, would that be an acceptable temporary wart in order to not block shipping this?

On Thu, Apr 28, 2016 at 2:42 AM, Kenichi Ishibashi <ba...@chromium.org> wrote:

[Harald Alvestrand]

So it's been 2 weeks, and no LGTMs gathered. What conclusions can be drawn from this?

In particular, what's the most important thing to fix before I come back with another I2S?

And what does @bashi think about what we should do next on his year-old patch?

[Rick Byers]

Sorry for the delay on this.

In general we'd really rather not ship an API where certain argument forms are not interoperable with what other browsers have already shipped, it just makes it too easy to easy for developers to be burned by an API appearing to be supported but then failing only in Chrome in a particular use case.  haraken@ can someone from the bindings team work with Harald to come up with some short-term plan you're OK with for shipping an API that matches the other browsers here?

On a related note (since this interop issue was kind of subtle and easy to miss), what's the state of the interop tests here?  I see there are a few (pretty simple) constraint-related tests here, but it looks like we're skipping the entire directory in blink.  When I run them manually (in prefixes mode, and with experimental web platform features enabled) I see they still largely fail.  Is there an effort to get these tests passing on blink one way or the other?

Other than those, shipping this looks good to me.

[Kentaro Hara]

I'll chat with bashi@ about this.

(Sorry about the delay -- Tokyo is on national holidays until May 8.)

--

Kentaro Hara, Tokyo, Japan

[Dimitri Glazkov]

Any updates here?

:DG<

[Harald Alvestrand]

If I read the responses correctly, I won't get LGTM to ship without support for at least the full syntax for Ideal.

So waiting for bashi's updates.

[Kentaro Hara]

Sorry about the delayed response...

bashi@ is now working on it, and nested IDL unions & dictionaries are going to be supported by this CL and a couple of follow-ups.

[Kentaro Hara]

And a tracking bug is here.

[Kentaro Hara]

I think the nested IDL unions & dictionaries are now supported by r393130.

[Harald Alvestrand]

Almost got the IDL-facing code done before I saw that the CL was rolled back.

Seems straightforward, except for one case where I'm chasing a bug.

[Harald Alvestrand]

Resurrecting the Intent to Ship thread:

We now have working parsing of naked constraint values, so that we don't get TypeError when people use this form.

(We ignore the values in some cases, but that's actually legal according to spec, and we can fix quite a few cases easily.)

We also have getConstraints (retrieving the current constraints values) working, and the getSettings call exists (but getting the right values is work in progress - this may warrant its own flag).

I'd like to get intent-to-ship approval, so that we can aim for flipping the MediaConstraints flag in M53.

[PhistucK]

I thought I saw that something landed regarding ideal values - are they included in this intent?

☆PhistucK

[Harald Alvestrand]

Yes, shipping the code landed in https://codereview.chromium.org/1977513004/ is included.

[Rick Byers]

On Tue, May 31, 2016 at 1:31 AM, Harald Alvestrand <h...@google.com> wrote:

Resurrecting the Intent to Ship thread:

We now have working parsing of naked constraint values, so that we don't get TypeError when people use this form.

(We ignore the values in some cases, but that's actually legal according to spec, and we can fix quite a few cases easily.)

We also have getConstraints (retrieving the current constraints values) working, and the getSettings call exists (but getting the right values is work in progress - this may warrant its own flag).

So what can you say overall about interoperability here with other browsers now?  I.e. how do we evaluate the extent to which code which uses media track constraints will work the same on multiple browsers (which pass feature detection - eg. for the 'applyConstraints' API).

Again I'd love to hear that there are some interop tests (eg. in web-platform-test) and how our results compare to other browsers, but that's still far from being the norm in web standards development so not (yet) a requirement (or even an expectation).  

Ultimately this is a judgement call, and only people experts in the real-world usage of these APIs (i.e. yourself) will have a good sense for the grey area between "not at all interoperable" and "perfectly interoperable".  So if you say that the vast majority of typical use cases will now work consistently between Edge, Chrome and Firefox (and nobody has arguments to the contrary) then that's enough for me and I'm happy :-)

[Harald Alvestrand]

The current situation is that Chrome is dominant in the marketplace for WebRTC browsers. So people implement our old non-standard constraint syntax, and as an afterthought they add conditional support for Firefox' standards compliant syntax.

The graph for the name/value form (what we currently have) is here:

https://www.chromestatus.com/metrics/feature/timeline/popularity/1150

And the standards-compliant one is here:

https://www.chromestatus.com/metrics/feature/timeline/popularity/1152

(obviously flat, since it hasn't been released yet)

This is a situation we want to get away from, and the first step is obviously to give the users the option to use the standard.

There is a W3C test suite that exercises constraints to some degree under "mediacapture-streams"; we currently pass 9 and fail 6 - one of these failures concerns constraints; I'll take a closer look at that.

My "feel" is that we'll learn of breakages when people try to convert to the common syntax. I think we'll be interoperable "apart from bugs".

[Rick Byers]

Thank you for the details!  LGTM1 to ship.

On Wed, Jun 1, 2016 at 7:24 AM, Harald Alvestrand <h...@google.com> wrote:

The current situation is that Chrome is dominant in the marketplace for WebRTC browsers. So people implement our old non-standard constraint syntax, and as an afterthought they add conditional support for Firefox' standards compliant syntax.

The graph for the name/value form (what we currently have) is here:

https://www.chromestatus.com/metrics/feature/timeline/popularity/1150

And the standards-compliant one is here:

https://www.chromestatus.com/metrics/feature/timeline/popularity/1152

(obviously flat, since it hasn't been released yet)

This is a situation we want to get away from, and the first step is obviously to give the users the option to use the standard.

There is a W3C test suite that exercises constraints to some degree under "mediacapture-streams"; we currently pass 9 and fail 6 - one of these failures concerns constraints; I'll take a closer look at that.

Excellent!  Yes please investigate the failure and (assuming Firefox passes that test) please circle back here if you're unable to fix our behavior to match Firefox prior to shipping.  Have you imported the web platform tests in question into our tree so that we'll get bot failures if we regress them (and be able to lock in the wins as we start passing more)?

My "feel" is that we'll learn of breakages when people try to convert to the common syntax. I think we'll be interoperable "apart from bugs".

Yes, that's certainly always the case.  But I think it's also our responsibility to invest up-front to try to convince ourselves that the likely common scenarios work the same across browsers (especially when some other engine has already shipped behavior according to some spec ahead of us).  All too often I've seen our bugs cause developer pain, often developers don't bother filing bugs against us and (even if they did file a bug) change their code to expect our bugs, then other vendors end up have to copy our bugs to be web compatible.  In general this is a big problem for the long-term health of the web platform.

[Harald Alvestrand]

Thanks Rick!

Other API owners: More LGTMs or comments?

[Chris Harrelson]

LGTM2

[Jochen Eisinger]

lgtm3

[Harald Alvestrand]

Thanks all!