Intent to Implement and Ship: ImageCapture support for Pan/Tilt

577 views
Skip to first unread message

rijubrat...@intel.com

unread,
Mar 21, 2019, 12:58:12 PM3/21/19
to blink-dev
rijubrat...@intel.com, mcas...@chromium.org, guidou@chromium.org This feature gives developers the ability to control the Pan and Tilt motion of cameras using getUserMedia. Specification: https://github.com/w3c/mediacapture-image/ https://github.com/w3c/mediacapture-image/pull/182 ImageCapture API is already TAG reviewed and shipping. This feature is a small addition to the property set for capabilities, constraints and settings. This feature allows developers to get the range for Pan and Tilt in the camera and as well as set those values using media constraints in getUserMedia. https://w3c.github.io/mediacapture-image/#dom-mediatracksupportedconstraints-pan https://w3c.github.io/mediacapture-image/#dom-mediatrackcapabilities-tilt Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. Chrome has been using a private extension(webcam_private.idl) to satisfy this use case. Implementing this feature will allow developers to use this feature in a standards compliant manner using media constraints in getUserMedia.
No other browser implements this feature yet, so there should be no interoperability concerns. No applications are using this feature, so there are no compatibility risks. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals N/A None required.
No Linux, Chrome OS, Windows have platform support. Android does not have platform support.

Daniel Bratell

unread,
Mar 21, 2019, 1:07:18 PM3/21/19
to blink-dev, rijubrat...@intel.com
LGTM1

/Daniel

On Thu, 21 Mar 2019 17:58:12 +0100, <rijubrat...@intel.com> wrote:

rijubrat...@intel.com, mca...@chromium.org, gui...@chromium.org This feature gives developers the ability to control the Pan and Tilt motion of cameras using getUserMedia. Specification: https://github.com/w3c/mediacapture-image/ https://github.com/w3c/mediacapture-image/pull/182 ImageCapture API is already TAG reviewed and shipping. This feature is a small addition to the property set for capabilities, constraints and settings. This feature allows developers to get the range for Pan and Tilt in the camera and as well as set those values using media constraints in getUserMedia. https://w3c.github.io/mediacapture-image/#dom-mediatracksupportedconstraints-pan https://w3c.github.io/mediacapture-image/#dom-mediatrackcapabilities-tilt Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. Chrome has been using a private extension(webcam_private.idl) to satisfy this use case. Implementing this feature will allow developers to use this feature in a standards compliant manner using media constraints in getUserMedia.
No other browser implements this feature yet, so there should be no interoperability concerns. No applications are using this feature, so there are no compatibility risks. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals N/A None required.
No Linux, Chrome OS, Windows have platform support. Android does not have platform support.
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0237a9b8-937e-4443-a71b-8e1f7d79967f%40chromium.org.



--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */

Yoav Weiss

unread,
Mar 21, 2019, 1:13:47 PM3/21/19
to Bhaumik, Rijubrata, blink-dev


On Thu, Mar 21, 2019 at 5:58 PM <rijubrat...@intel.com> wrote:
rijubrat...@intel.com, mca...@chromium.org, gui...@chromium.org This feature gives developers the ability to control the Pan and Tilt motion of cameras using getUserMedia. Specification: https://github.com/w3c/mediacapture-image/ https://github.com/w3c/mediacapture-image/pull/182 ImageCapture API is already TAG reviewed and shipping. This feature is a small addition to the property set for capabilities, constraints and settings. This feature allows developers to get the range for Pan and Tilt in the camera and as well as set those values using media constraints in getUserMedia. https://w3c.github.io/mediacapture-image/#dom-mediatracksupportedconstraints-pan https://w3c.github.io/mediacapture-image/#dom-mediatrackcapabilities-tilt Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. Chrome has been using a private extension(webcam_private.idl) to satisfy this use case. Implementing this feature will allow developers to use this feature in a standards compliant manner using media constraints in getUserMedia.
No other browser implements this feature yet, so there should be no interoperability concerns.

Interoperability is about the long term prospects of the web platform converging on supporting the feature, and the ways developers can cope with the lack of universal support in the meantime. The fact that no browser supports this does not mean there are no interoperability concerns.

What's the feature detection story for this? I'm guessing developers can use the MediaTrackSupportedConstraints dictionary to know if the UA supports the feature or not. Is that correct?

No applications are using this feature, so there are no compatibility risks. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals

Have you reached out to other vendors?
 
N/A None required.
No Linux, Chrome OS, Windows have platform support. Android does not have platform support.

Can you expand on why?
 

rijubrat...@intel.com

unread,
Mar 21, 2019, 3:22:06 PM3/21/19
to blink-dev, rijubrat...@intel.com, rei...@chromium.org


On Thursday, March 21, 2019 at 7:13:47 PM UTC+2, Yoav Weiss wrote:


On Thu, Mar 21, 2019 at 5:58 PM <rijubrat...@intel.com> wrote:
rijubrat...@intel.com, mca...@chromium.org, gui...@chromium.org This feature gives developers the ability to control the Pan and Tilt motion of cameras using getUserMedia. Specification: https://github.com/w3c/mediacapture-image/ https://github.com/w3c/mediacapture-image/pull/182 ImageCapture API is already TAG reviewed and shipping. This feature is a small addition to the property set for capabilities, constraints and settings. This feature allows developers to get the range for Pan and Tilt in the camera and as well as set those values using media constraints in getUserMedia. https://w3c.github.io/mediacapture-image/#dom-mediatracksupportedconstraints-pan https://w3c.github.io/mediacapture-image/#dom-mediatrackcapabilities-tilt Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. Chrome has been using a private extension(webcam_private.idl) to satisfy this use case. Implementing this feature will allow developers to use this feature in a standards compliant manner using media constraints in getUserMedia.
No other browser implements this feature yet, so there should be no interoperability concerns.

Interoperability is about the long term prospects of the web platform converging on supporting the feature, and the ways developers can cope with the lack of universal support in the meantime. The fact that no browser supports this does not mean there are no interoperability concerns.

What's the feature detection story for this? I'm guessing developers can use the MediaTrackSupportedConstraints dictionary to know if the UA supports the feature or not. Is that correct?
Apologies, yes you are right, MediaTrackSupportedConstraints tells us that browser understands the request. "Correct values" like Infinity will denote if device supports it. 

No applications are using this feature, so there are no compatibility risks. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals

Have you reached out to other vendors?
No. This request came Google (crbug.com/934063),  Miguel and +Reilly can fill us more on this.
Apparently video-conferencing  apps were using non-standard webcam_private.idl private extension for Pan/Tilt feature on CrOS.
Please check : crbug.com/944445
 
N/A None required.
No Linux, Chrome OS, Windows have platform support. Android does not have platform support.

Can you expand on why?
Pan/Tilt feature is usually attained using mechanical motor which are not possible for phones or tablets (packaging issue) where Android is popular.
Although electronic Pan/Tilt is possible in some CMOS chips, range of pan and tilt is much smaller, for example, 10 degrees in any direction.
Cameras using chips with electronic pan and tilt simply have a lens over the CMOS chip.
Most importantly phones and tablets (Android) are used by hands, which people can move in any direction.
So manufactures do not feel a compelling need to put much higher priced cameras and thereby no platform support.
Linux, Chrome OS, Windows run on variety of other systems where more expensive external Webcams can be attached. 

Alex Russell

unread,
Mar 21, 2019, 3:32:04 PM3/21/19
to blink-dev, rijubrat...@intel.com, rei...@chromium.org
Hey all,

Per the mention of TAG review, we frequently request that new features which are additions to existing features seek broad feedback. It's often the case that small additions can have big consequences or create interop and consistency hazards.

The API OWNERS are more likely to look to the TAG for signals about these sorts of concerns when Chromium is the first to implement and ship a feature. Suggest you file for a review for this feature.

Regards

Philipp Hancke

unread,
Mar 23, 2019, 2:36:20 AM3/23/19
to rijubrat...@intel.com, blink-dev
With my web developer hat on: yes please!
I've been looking into this ever since reading Scott Hanselmans
back in 2012.I expected having to use something like webusb but MediaStreamTrack.applyConstraints is *much* more elegant.

Am Do., 21. März 2019 um 17:58 Uhr schrieb <rijubrat...@intel.com>:
rijubrat...@intel.com, mca...@chromium.org, gui...@chromium.org This feature gives developers the ability to control the Pan and Tilt motion of cameras using getUserMedia. Specification: https://github.com/w3c/mediacapture-image/ https://github.com/w3c/mediacapture-image/pull/182 ImageCapture API is already TAG reviewed and shipping. This feature is a small addition to the property set for capabilities, constraints and settings. This feature allows developers to get the range for Pan and Tilt in the camera and as well as set those values using media constraints in getUserMedia. https://w3c.github.io/mediacapture-image/#dom-mediatracksupportedconstraints-pan https://w3c.github.io/mediacapture-image/#dom-mediatrackcapabilities-tilt Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. Chrome has been using a private extension(webcam_private.idl) to satisfy this use case. Implementing this feature will allow developers to use this feature in a standards compliant manner using media constraints in getUserMedia.
No other browser implements this feature yet, so there should be no interoperability concerns. No applications are using this feature, so there are no compatibility risks. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals N/A None required.
No Linux, Chrome OS, Windows have platform support. Android does not have platform support.

Philip Jägenstedt

unread,
Mar 26, 2019, 5:47:09 AM3/26/19
to Bhaumik, Rijubrata, Jan-Ivar Bruaroey, blink-dev, Reilly Grant
On Thu, Mar 21, 2019 at 8:22 PM <rijubrat...@intel.com> wrote:


On Thursday, March 21, 2019 at 7:13:47 PM UTC+2, Yoav Weiss wrote:


On Thu, Mar 21, 2019 at 5:58 PM <rijubrat...@intel.com> wrote:
rijubrat...@intel.com, mca...@chromium.org, gui...@chromium.org This feature gives developers the ability to control the Pan and Tilt motion of cameras using getUserMedia. Specification: https://github.com/w3c/mediacapture-image/ https://github.com/w3c/mediacapture-image/pull/182 ImageCapture API is already TAG reviewed and shipping. This feature is a small addition to the property set for capabilities, constraints and settings. This feature allows developers to get the range for Pan and Tilt in the camera and as well as set those values using media constraints in getUserMedia. https://w3c.github.io/mediacapture-image/#dom-mediatracksupportedconstraints-pan https://w3c.github.io/mediacapture-image/#dom-mediatrackcapabilities-tilt Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. Chrome has been using a private extension(webcam_private.idl) to satisfy this use case. Implementing this feature will allow developers to use this feature in a standards compliant manner using media constraints in getUserMedia.
No other browser implements this feature yet, so there should be no interoperability concerns.

Interoperability is about the long term prospects of the web platform converging on supporting the feature, and the ways developers can cope with the lack of universal support in the meantime. The fact that no browser supports this does not mean there are no interoperability concerns.

What's the feature detection story for this? I'm guessing developers can use the MediaTrackSupportedConstraints dictionary to know if the UA supports the feature or not. Is that correct?
Apologies, yes you are right, MediaTrackSupportedConstraints tells us that browser understands the request. "Correct values" like Infinity will denote if device supports it.

To spell that out, it's `navigator.mediaDevices.getSupportedConstraints().pan` and `navigator.mediaDevices.getSupportedConstraints().tilt`. (This simpler than it usually is with detecting support for dictionary members, because of the getSupportedConstraints convience method.)

No applications are using this feature, so there are no compatibility risks. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals

Have you reached out to other vendors?
No. This request came Google (crbug.com/934063),  Miguel and +Reilly can fill us more on this.
Apparently video-conferencing  apps were using non-standard webcam_private.idl private extension for Pan/Tilt feature on CrOS.
Please check : crbug.com/944445

+Jan-Ivar Bruaroey (Mozilla) reviewed https://github.com/w3c/mediacapture-image/pull/182. Jan-Ivar, do you have concerns about these constraints that you didn't raise in the PR? Do you expect you'll implement this in Firefox?
 
 
N/A None required.
No Linux, Chrome OS, Windows have platform support. Android does not have platform support.

Can you expand on why?
Pan/Tilt feature is usually attained using mechanical motor which are not possible for phones or tablets (packaging issue) where Android is popular.
Although electronic Pan/Tilt is possible in some CMOS chips, range of pan and tilt is much smaller, for example, 10 degrees in any direction.
Cameras using chips with electronic pan and tilt simply have a lens over the CMOS chip.
Most importantly phones and tablets (Android) are used by hands, which people can move in any direction.
So manufactures do not feel a compelling need to put much higher priced cameras and thereby no platform support.
Linux, Chrome OS, Windows run on variety of other systems where more expensive external Webcams can be attached. 
 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0237a9b8-937e-4443-a71b-8e1f7d79967f%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

Philip Jägenstedt

unread,
Mar 26, 2019, 5:57:56 AM3/26/19
to Alex Russell, youenn fablet, blink-dev, Bhaumik, Rijubrata, Reilly Grant
Rijubrata, https://github.com/w3ctag/design-reviews/issues is the place to file TAG review requests.

However, I share your assessment that this "is a small addition to the property set for capabilities, constraints and settings" and need not go through review. More important is to gauge interest from other vendors directly. I've already poked Jan-Ivar in this thread, and for WebKit perhaps +youenn fablet is a good contact. Does any Apple hardware have cameras that can pan/tilt? In either case, do you (Youenn) have any concerns with these constraints?

FWIW, this is a case where I think the risk of not supporting the feature when others do is particularly low, because of the semantics of constraints and ignoring unknown constraints. Supporting the API when no hardware can actually be panned/tilted ought to behave exactly as not supporting it, apart from the feature detection. (Mandatory constraints would be an exception to this, but for that to work for a general purpose web app hardware supporting pan/tilt would have to be ubiquitous first, which seems unlikely to happen.)


To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c4743f25-e158-4f8c-bf35-71dfddde06f3%40chromium.org.

rijubrata bhaumik

unread,
Mar 26, 2019, 11:11:27 AM3/26/19
to blink-dev, sligh...@google.com, yfa...@apple.com, rijubrat...@intel.com, rei...@chromium.org
Thanks Alex and Philip,
I have filed a TAG review request.

Miguel Casas-Sanchez

unread,
Mar 26, 2019, 1:42:27 PM3/26/19
to rijubrata bhaumik, blink-dev, Alex Russell, yfa...@apple.com, Bhaumik, Rijubrata, Reilly Grant
On Tue, 26 Mar 2019 at 11:11, rijubrata bhaumik <riju...@gmail.com> wrote:
Thanks Alex and Philip,
I have filed a TAG review request.

On Tuesday, 26 March 2019 11:57:56 UTC+2, Philip Jägenstedt wrote:
Rijubrata, https://github.com/w3ctag/design-reviews/issues is the place to file TAG review requests.

However, I share your assessment that this "is a small addition to the property set for capabilities, constraints and settings" and need not go through review. More important is to gauge interest from other vendors directly. I've already poked Jan-Ivar in this thread, and for WebKit perhaps +youenn fablet is a good contact. Does any Apple hardware have cameras that can pan/tilt? In either case, do you (Youenn) have any concerns with these constraints?

Only thing I have to add is that this PR was suggested by an external contributor @dsanders11 (https://github.com/w3c/mediacapture-image/issues/177) for a project of theirs;  the only discussion in the PR was about using arc-seconds or other measures, that would allow or not the JS code to measure back when the target pan or tilt would be achieved; @jan-ivar was very active in the discussion.
 

FWIW, this is a case where I think the risk of not supporting the feature when others do is particularly low, because of the semantics of constraints and ignoring unknown constraints. Supporting the API when no hardware can actually be panned/tilted ought to behave exactly as not supporting it, apart from the feature detection. (Mandatory constraints would be an exception to this, but for that to work for a general purpose web app hardware supporting pan/tilt would have to be ubiquitous first, which seems unlikely to happen.)


On Thu, Mar 21, 2019 at 8:32 PM 'Alex Russell' via blink-dev <blin...@chromium.org> wrote:
Hey all,

Per the mention of TAG review, we frequently request that new features which are additions to existing features seek broad feedback. It's often the case that small additions can have big consequences or create interop and consistency hazards.

The API OWNERS are more likely to look to the TAG for signals about these sorts of concerns when Chromium is the first to implement and ship a feature. Suggest you file for a review for this feature.

Regards

On Thursday, March 21, 2019 at 12:22:06 PM UTC-7, rijubrat...@intel.com wrote:


On Thursday, March 21, 2019 at 7:13:47 PM UTC+2, Yoav Weiss wrote:


On Thu, Mar 21, 2019 at 5:58 PM <rijubrat...@intel.com> wrote:
rijubrat...@intel.com, mca...@chromium.org, gui...@chromium.org This feature gives developers the ability to control the Pan and Tilt motion of cameras using getUserMedia. Specification: https://github.com/w3c/mediacapture-image/ https://github.com/w3c/mediacapture-image/pull/182 ImageCapture API is already TAG reviewed and shipping. This feature is a small addition to the property set for capabilities, constraints and settings. This feature allows developers to get the range for Pan and Tilt in the camera and as well as set those values using media constraints in getUserMedia. https://w3c.github.io/mediacapture-image/#dom-mediatracksupportedconstraints-pan https://w3c.github.io/mediacapture-image/#dom-mediatrackcapabilities-tilt Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. Chrome has been using a private extension(webcam_private.idl) to satisfy this use case. Implementing this feature will allow developers to use this feature in a standards compliant manner using media constraints in getUserMedia.
No other browser implements this feature yet, so there should be no interoperability concerns.

Interoperability is about the long term prospects of the web platform converging on supporting the feature, and the ways developers can cope with the lack of universal support in the meantime. The fact that no browser supports this does not mean there are no interoperability concerns.

What's the feature detection story for this? I'm guessing developers can use the MediaTrackSupportedConstraints dictionary to know if the UA supports the feature or not. Is that correct?
Apologies, yes you are right, MediaTrackSupportedConstraints tells us that browser understands the request. "Correct values" like Infinity will denote if device supports it. 

No applications are using this feature, so there are no compatibility risks. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals

Have you reached out to other vendors?
No. This request came Google (crbug.com/934063),  Miguel and +Reilly can fill us more on this.
Apparently video-conferencing  apps were using non-standard webcam_private.idl private extension for Pan/Tilt feature on CrOS.
Please check : crbug.com/944445

Correct, webcam_private.idl (https://cs.chromium.org/chromium/src/extensions/common/api/webcam_private.idl?sq=package:chromium&dr&g=0) is a Chrome private AI that we would like to migrate to the open Web Standards. Image Capture already supports |focus| and |zoom|, with this ITS we'll support |pan| and |tilt|.
 
 
N/A None required.
No Linux, Chrome OS, Windows have platform support. Android does not have platform support.

Can you expand on why?
Pan/Tilt feature is usually attained using mechanical motor which are not possible for phones or tablets (packaging issue) where Android is popular.
Although electronic Pan/Tilt is possible in some CMOS chips, range of pan and tilt is much smaller, for example, 10 degrees in any direction.
Cameras using chips with electronic pan and tilt simply have a lens over the CMOS chip.
Most importantly phones and tablets (Android) are used by hands, which people can move in any direction.
So manufactures do not feel a compelling need to put much higher priced cameras and thereby no platform support.
Linux, Chrome OS, Windows run on variety of other systems where more expensive external Webcams can be attached. 
 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0237a9b8-937e-4443-a71b-8e1f7d79967f%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c4743f25-e158-4f8c-bf35-71dfddde06f3%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

jbru...@mozilla.com

unread,
Mar 27, 2019, 11:01:07 AM3/27/19
to blink-dev, rijubrat...@intel.com, j...@mozilla.com, rei...@chromium.org
On Tuesday, March 26, 2019 at 5:47:09 AM UTC-4, Philip Jägenstedt wrote:
+Jan-Ivar Bruaroey (Mozilla) reviewed https://github.com/w3c/mediacapture-image/pull/182. Jan-Ivar, do you have concerns about these constraints that you didn't raise in the PR?

I think it's great Chrome is moving to this from their proprietary solution.

I don't have many concerns with the API. This will be the first constraint dealing with a physical motor, so we've discussed some questions there, whether these are target settings (they are) vs measured settings, and whether applyConstraints would wait for the motor to complete (probably), in https://github.com/w3c/mediacapture-main/issues/470#issuecomment-310921337

As long as Chrome engineers reach out to discuss any issues they find during implementation, I don't see a problem. Happy to hear what works and what doesn't.
 
Do you expect you'll implement this in Firefox?
 
It's an interesting feature. Since we don't have a proprietary solution to build on, we would need to assess investing resources into this, before making a commitment.

.: Jan-Ivar :.

Philip Jägenstedt

unread,
Apr 11, 2019, 3:16:53 PM4/11/19
to jbru...@mozilla.com, Kenneth Christiansen, blink-dev, Bhaumik, Rijubrata, Jan-Ivar Bruaroey, Reilly Grant
+Kenneth Christiansen this intent is blocked on the TAG review, do you have an update on when that will be resolved?

It looks like the discussion is mostly around what unit is used, and the spec says clearly that it's arc seconds.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/144482f1-0f53-4348-98b9-70fb6f6c02fc%40chromium.org.

Christiansen, Kenneth R

unread,
Apr 15, 2019, 11:21:05 AM4/15/19
to Philip Jägenstedt, jbru...@mozilla.com, blink-dev, Bhaumik, Rijubrata, Jan-Ivar Bruaroey, Reilly Grant

Let me bring it up in our next sync meeting, to get the input for the others.

 

I am personally fine with this now.

 

Kenneth

Chris Harrelson

unread,
May 2, 2019, 10:43:30 AM5/2/19
to Christiansen, Kenneth R, Philip Jägenstedt, jbru...@mozilla.com, blink-dev, Bhaumik, Rijubrata, Jan-Ivar Bruaroey, Reilly Grant
Hi,

It looks like the TAG review finished, with one additional comment about permissions. Once you update the thread with a conclusion on units and the permissions, I think this is good to go.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/60F18B024F3783428EBEA0B7D7DF31CA85928D9D%40IRSMSX108.ger.corp.intel.com.

rijubrat...@intel.com

unread,
May 6, 2019, 6:09:40 AM5/6/19
to blink-dev, kenneth.r.c...@intel.com, foo...@chromium.org, jbru...@mozilla.com, rijubrat...@intel.com, j...@mozilla.com, rei...@chromium.org


So, TAG has provided comments and let me summarize that for this audience.
Units: arc-seconds are fine.

Privacy & Security:
TAG's opinion is the feature is a useful addition but needs to be gated behind a Pan/Tilt/Zoom specific "permission".

How UA manages this permission prompt in conjunction with 'getUserMedia' is something implementer can decide, like single prompt showing 2 distinct permissions,etc
user should be able to opt-out of just this (PTZ) feature, while still giving permission to video stream (getUserMedia).


On Thursday, May 2, 2019 at 5:43:30 PM UTC+3, Chris Harrelson wrote:
Hi,

It looks like the TAG review finished, with one additional comment about permissions. Once you update the thread with a conclusion on units and the permissions, I think this is good to go.

To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

Philip Jägenstedt

unread,
May 8, 2019, 9:57:32 AM5/8/19
to Bhaumik, Rijubrata, blink-dev, Kenneth Christiansen, jbru...@mozilla.com, Jan-Ivar Bruaroey, Reilly Grant
Does this imply that there should be a new entry in https://w3c.github.io/permissions/#permission-registry? What should it be called?

This would be a bit like the split between "camera" and "microphone", which are separate, but I think were originally a single permission. It's only if some implementer intends so surface separate permissions to users that the split at the permissions level will be meaningful.

Do you have the right contacts for security and UX review to figure out whether new UI in Chromium is warranted for this?

Bhaumik, Rijubrata

unread,
May 8, 2019, 10:41:45 AM5/8/19
to Philip Jägenstedt, blink-dev, Kenneth Christiansen, jbru...@mozilla.com, Jan-Ivar Bruaroey, Reilly Grant
On Wed, May 8, 2019 at 4:57 PM Philip Jägenstedt <foo...@chromium.org> wrote:
Does this imply that there should be a new entry in https://w3c.github.io/permissions/#permission-registry? What should it be called?
Looks like that. 

This would be a bit like the split between "camera" and "microphone", which are separate, but I think were originally a single permission. It's only if some implementer intends so surface separate permissions to users that the split at the permissions level will be meaningful.

Do you have the right contacts for security and UX review to figure out whether new UI in Chromium is warranted for this?
Thanks, Reilly promised to help out with this. I guess we need some re-work in terms of implementation and design with respect to permission handling. Let me formulate the next steps offline with the Privacy and UX Security folks and related stakeholders and update this audience.  

Philipp Hancke

unread,
May 8, 2019, 2:15:06 PM5/8/19
to rijubrat...@intel.com, blink-dev, kenneth.r.c...@intel.com, Philip Jägenstedt, Jan-Ivar Bruaroey, rei...@chromium.org
So how exactly is this going to work with GUM? I had to ask Jan-Ivar even...
  navigator.mediaDevices.getUserMedia({video: {ptz: 1}})
could add an optional checkbox to the prompt.getUserMedia((optionally with {ptz: {exact: 1}}) which would fail if the ptz request is denied).
What if the user initially used a camera without PTZ, said "yes" to video, then attaches a camera with PTZ capability?

From a privacy point of view the situation is also different for cameras llke the Brio4k (where the user might not be aware of the PTZ capability) which do digital zoom and pan/tilt inside the original field of view (i.e. you can never see more than without PTZ) and devices like the venerable Logitech BCC 950 which people buy because they can be moved around.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/e8a43d60-fcdb-40ce-a281-0caf82bc1a8e%40chromium.org.

jbru...@mozilla.com

unread,
May 8, 2019, 2:44:17 PM5/8/19
to blink-dev, rijubrat...@intel.com, kenneth.r.c...@intel.com, foo...@chromium.org, j...@mozilla.com, rei...@chromium.org
(To clarify since I was mentioned) I was looking into if we could infer app interest from existing constraints here, e.g.:

    getUserMedia({video: {pan: 0}});

...either in order to satisfy the request, or perhaps even from mere presence, if we worry about not touching the motor needlessly e.g.

    getUserMedia({video: {pan: {min: -180*3600, max: 180*36000}}});

This would require very little change to the API, was my thinking.

On Wednesday, May 8, 2019 at 2:15:06 PM UTC-4, Philipp Hancke wrote:
So how exactly is this going to work with GUM? I had to ask Jan-Ivar even...
  navigator.mediaDevices.getUserMedia({video: {ptz: 1}})
could add an optional checkbox to the prompt.getUserMedia((optionally with {ptz: {exact: 1}}) which would fail if the ptz request is denied).
What if the user initially used a camera without PTZ, said "yes" to video, then attaches a camera with PTZ capability?

This would depend on what the user was asked and what was granted, I assume. It's a second getUserMedia() call, so, say we key off the presence of {pan, tilt, zoom} constraints, as I suggested, these might cause the constraints algorithm to consider the new camera over the existing one—this typically takes `exact` constraints in Firefox—and if that works, it would seem necessary to prompt again for this permission elevation IMHO, if it wasn't asked initially.
 
From a privacy point of view the situation is also different for cameras llke the Brio4k (where the user might not be aware of the PTZ capability) which do digital zoom and pan/tilt inside the original field of view (i.e. you can never see more than without PTZ) and devices like the venerable Logitech BCC 950 which people buy because they can be moved around.

I agree permission seem less necessary in the digital zoom case, though I'm unsure to what extent browsers can detect this, other than keep a list of known models.

.: Jan-Ivar :.

Rijubrata Bhaumik

unread,
Sep 29, 2020, 8:35:26 AM9/29/20
to blink-dev, rijubrat...@intel.com, kenneth.r.c...@intel.com, foo...@chromium.org, j...@mozilla.com, rei...@chromium.org, fbea...@chromium.org, Hakkinen, Eero

Thank you all for the patience, now we are ready to notify that we have processed all the necessary information from TAG, Privacy team at Chrome and all other stakeholders in the WebRTC community. To recap, our model of thinking has been:

The Pan-Tilt-Zoom permission can only be requested (and only shown in the prompt) if the currently connected/selected camera hardware supports PTZ.


Camera permission without PTZ (obtained using an older version of the user agent, or with another camera) is not implicitly upgraded to Camera permission with PTZ (even if the hardware supports PTZ). The permission has to be re-requested.


In w3ctag/design-reviews#358, TAG wants to make sure that from the website’s perspective: Pan-Tilt-Zoom needs to be explicitly requested as an extension to video capture permission.


We decided to extend the getUserMedia() API to support requesting the PTZ permission in addition to the camera permission. If the selected/connected camera does not support PTZ, the UA will fall back to the camera permission.


Example code :

// User is prompted to grant both camera and PTZ access in a single call.

// If the camera does not support PTZ or user denies PTZ permission, it falls

// back to a regular camera prompt.

const videoStream = await navigator.mediaDevices.getUserMedia({

  // [NEW] Website asks to control camera PTZ as well.

  video: { pan: true, tilt: true, zoom: true },

});


A change from last time is that now zoom is gated by the PTZ permission on Desktop but left unchanged on Android, as pan/tilt APIs are not supported on Android. Another change is that the website is allowed to control camera PTZ only when the page is visible to the user.

We have documented integration with the Permissions API, Security and possible Fingerprinting issues, among other things in the PTZ explainer. Best,

Riju.


Yoav Weiss

unread,
Sep 29, 2020, 4:20:24 PM9/29/20
to Rijubrata Bhaumik, blink-dev, Kenneth Christiansen, Philip Jägenstedt, j...@mozilla.com, Reilly Grant, Francois Beaufort, Hakkinen, Eero
Thanks for doing the work of addressing the feedback. This is a capability I'm personally excited about and would love to see in the wild.

LGTM2 (assuming Daniel's LGTM from 6 months ago still stands)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3a7cbc20-613a-489d-9e38-451fc6d309d4o%40chromium.org.

Chris Harrelson

unread,
Sep 29, 2020, 5:01:10 PM9/29/20
to Yoav Weiss, Rijubrata Bhaumik, blink-dev, Kenneth Christiansen, Philip Jägenstedt, Jan-Ivar Bruaroey, Reilly Grant, Francois Beaufort, Hakkinen, Eero
LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEh1se3v52XjTKQ0%2B0QzcjfpTMrgo2%2BgQTuTrxjR0TqRyw%40mail.gmail.com.

Daniel Bratell

unread,
Sep 30, 2020, 12:02:57 PM9/30/20
to Chris Harrelson, Yoav Weiss, Rijubrata Bhaumik, blink-dev, Kenneth Christiansen, Philip Jägenstedt, Jan-Ivar Bruaroey, Reilly Grant, Francois Beaufort, Hakkinen, Eero

Yep, my LGTM1 still stands. Happy to see that you have improved it further!

/Daniel

Yoav Weiss

unread,
Sep 30, 2020, 12:14:28 PM9/30/20
to Daniel Bratell, Chris Harrelson, Rijubrata Bhaumik, blink-dev, Kenneth Christiansen, Philip Jägenstedt, Jan-Ivar Bruaroey, Reilly Grant, Francois Beaufort, Hakkinen, Eero
Trying out the implementation on Canary, a zoom level set in one session seems to stick when opening a media session in another, unrelated browser window.

In the explainer this is being considered as part of the threat model, so I assumed this is something the Chromium implementation handles. Is that not the case? Or is what I'm seeing unexpected behavior and I should report it as a bug?

François Beaufort 🇫🇷

unread,
Oct 1, 2020, 2:28:57 AM10/1/20
to Yoav Weiss, Daniel Bratell, Chris Harrelson, Rijubrata Bhaumik, blink-dev, Kenneth Christiansen, Philip Jägenstedt, Jan-Ivar Bruaroey, Reilly Grant, Francois Beaufort, Hakkinen, Eero
It is true that setting/getting the PTZ values creates a cross-origin communication channel, but it is not dissimilar to preexisting attack vectors presented by other device APIs (e.g. reading/writing a GATT characteristic on a BLE device). Similarly to those API, the mitigation here is also to require both origins to obtain a permission.

Yoav Weiss

unread,
Oct 1, 2020, 2:42:24 AM10/1/20
to François Beaufort 🇫🇷, Daniel Bratell, Chris Harrelson, Rijubrata Bhaumik, blink-dev, Kenneth Christiansen, Philip Jägenstedt, Jan-Ivar Bruaroey, Reilly Grant, Francois Beaufort, Hakkinen, Eero
OK, thanks. I think the mitigation could be further improved, but if this was considered, that doesn't seem like a blocker.
Reply all
Reply to author
Forward
0 new messages