Intent to Prototype: PointerEvent.deviceId for Mult-Pen Inking

[Sahir Vellani]

Contact emails

bema...@microsoft.com, sahir....@microsoft.com

Explainer

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

Specification

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

Summary

As devices with advanced pen input capabilities are becoming increasingly prevalent, it is important that the web platform continues to evolve to fully support these advanced features in order to unlock rich experiences for both end users and developers. One such advancement is the ability for a device's digitizer to recognize more than one pen device interacting with it simultaneously. This feature is an extension to the PointerEvent interface to include a new attribute, deviceId, that represents a session-persistent, document isolated, unique identifier that a developer can reliably use to identify individual pens interacting with the page.

Blink component

Blink>Input

Motivation

Currently, developers have no way to distinguish between two individual pens on an ink-enabled digitizer. The existing PointerEvent.id attribute is implemented in different ways and does not always persist for each ink stroke or interaction with the screen. Developers can use this change to have a secure and reliable way to identify individual pen (pointers) interacting with the screen in order to set specific colors or pen shapes for each device interacting with the digitizer.

Initial public proposal

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

TAG review

TAG review status

Pending

Risks

Fingerprinting risks, which will be mitigated by randomizing the ID each renderer session.

Interoperability and Compatibility

Gecko: Request for Position: Extending the PointerEvent with Unique DeviceId Attribute · Issue #715 · mozilla/standards-positions · GitHub

WebKit: Request for Position: Extending the PointerEvent with Unique DeviceId Attribute · Issue #102 · WebKit/standards-positions · GitHub

Web developers: No signals

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

deviceId will be available via DevTools for front-end debugging.

Is this feature fully tested by web-platform-tests?

No

Flag name

PointerEventDeviceId

Requires code in //chrome?

False

Estimated milestones

No milestones specified

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5114132234240000

This intent message was generated by Chrome Platform Status.

 

[Rick Byers]

Cool, looks like a nice little addition to me. +Mustaq Ahmed and +Robert Flack from the Chrome interactions team who are likely code reviewers. 

The only real potential debate I see is around the details of the privacy protections. If I understand correctly "per document instance" means that the IDs will even be different across iframes in the same tab, and also when a page is reloaded. Is that right? If so, I can't see how it would be an issue.

On Mon, Dec 5, 2022 at 12:49 PM 'Sahir Vellani' via blink-dev <blin...@chromium.org> wrote:

Contact emails

bema...@microsoft.com, sahir....@microsoft.com

Explainer

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

Specification

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

FWIW with my former PointerEvents spec editor hat on, I pinged this issue on the PointerEvents spec. We've had 'extensions' to the PointerEvents spec in the past, so the PEWG may be amenable to something simple and pragmatic that'll naturally make it into the next official PE spec rather than starting with WICG. But with my Blink API owner hat on, either path is fine.

We've got automation plumbing for the rest of PointerEvents I believe. Any reason not to add plumbing through WebDriver for this too and an automated WPT test? I suppose the value is pretty low, but IMHO would be nice if it's not too much more expensive than the alternative chromium-only tests. 

Flag name

PointerEventDeviceId

Requires code in //chrome?

False

Estimated milestones

No milestones specified

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5114132234240000

This intent message was generated by Chrome Platform Status.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SA0PR00MB1033E5DE0BDE42239E647E9AFB189%40SA0PR00MB1033.namprd00.prod.outlook.com.

[Sahir Vellani]

Thank you for all the feedback!

 

Rick, yes that’s correct. The ID will be refreshed on reload and iframes will have a different ID to their parent/each other. Also, we can definitely explore integration with web driver and adding a WPT test.

 

Ben, any thoughts on the PEWG path Rick mentions below?

 

Sahir

 

From: Rick Byers <rby...@chromium.org>
Sent: Tuesday, December 6, 2022 12:34 PM
To: Sahir Vellani <Sahir....@microsoft.com>; Mustaq Ahmed <mus...@chromium.org>; Robert Flack <fla...@chromium.org>
Cc: blin...@chromium.org; Ben Mathwig <Benjamin...@microsoft.com>
Subject: [EXTERNAL] Re: [blink-dev] Intent to Prototype: PointerEvent.deviceId for Mult-Pen Inking

 

You don't often get email from rby...@chromium.org. Learn why this is important

[Rick Byers]

On Thu, Dec 8, 2022 at 1:17 PM Sahir Vellani <Sahir....@microsoft.com> wrote:

Thank you for all the feedback!

 

Rick, yes that’s correct. The ID will be refreshed on reload and iframes will have a different ID to their parent/each other. Also, we can definitely explore integration with web driver and adding a WPT test.

Perfect. I'm not a privacy reviewer, but that makes a lot of sense to me. 

Ben, any thoughts on the PEWG path Rick mentions below?

Sounds like the current editor / WG chair prefers a WICG spec until L3 gets finalized anyway. But knowing it's just a process thing and can trivially be moved into the PE spec once L3 reaches REC seems good enough to me.

[Mustaq Ahmed]

Supporting consistent IDs for supported pens sounds great.

Sahir & Ben: wondering why a new deviceId field seems more logical now against the original proposal of repurposing the pointerId field?  Is it because of the compat concern you raised in w3c/pointerevents/issues/353?

[Ben Mathwig]

---

Sahir & Ben: wondering why a new deviceId field seems more logical now against the original proposal of repurposing the pointerId field?  Is it because of the compat concern you raised in w3c/pointerevents/issues/353?

---

The primary reason is the deterministic nature of deviceId compared to pointerId. The scenario would be: (a) If the device supports getting a unique hardware identifier, we randomly generate an integer and persist that integer for the document lifespan. (b) If the device does not support getting a unique hardware identifier, does pointerId fall back to its original behavior? In the spec, we decided between null, undefined, and integer as the deterministic values of deviceId.

We could use pointerId instead, but we'd have to align on the solution for (b)

[Rick Byers]

On Mon, Dec 12, 2022 at 1:07 PM 'Ben Mathwig' via blink-dev <blin...@chromium.org> wrote:

---

Sahir & Ben: wondering why a new deviceId field seems more logical now against the original proposal of repurposing the pointerId field?  Is it because of the compat concern you raised in w3c/pointerevents/issues/353?

---

The primary reason is the deterministic nature of deviceId compared to pointerId. The scenario would be: (a) If the device supports getting a unique hardware identifier, we randomly generate an integer and persist that integer for the document lifespan. (b) If the device does not support getting a unique hardware identifier, does pointerId fall back to its original behavior? In the spec, we decided between null, undefined, and integer as the deterministic values of deviceId.

We could use pointerId instead, but we'd have to align on the solution for (b)

I guess we should take this to a GitHub repo somewhere, but I can't help but interject: sounds like a pretty good use case for InputDeviceCapabilities to me (i.e. 'hasDeterministicPointerId') ☺. Perhaps along with a boolean on the event for the limitation case discussed in the explainer.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/870380fe-ab9a-4925-8db1-261efa233295n%40chromium.org.

[Sahir Vellani]

---

From: Rick Byers <rby...@chromium.org>
Sent: Wednesday, December 14, 2022 1:05 PM
To: Ben Mathwig <Benjamin...@microsoft.com>
Cc: blink-dev <blin...@chromium.org>; Mustaq Ahmed <mus...@google.com>; fla...@chromium.org <fla...@chromium.org>; Sahir Vellani <Sahir....@microsoft.com>; Ben Mathwig <Benjamin...@microsoft.com>
Subject: Re: [EXTERNAL] Re: [blink-dev] Intent to Prototype: PointerEvent.deviceId for Mult-Pen Inking

 

You don't often get email from rby...@chromium.org. Learn why this is important

On Mon, Dec 12, 2022 at 1:07 PM 'Ben Mathwig' via blink-dev <blin...@chromium.org> wrote:

---

Sahir & Ben: wondering why a new deviceId field seems more logical now against the original proposal of repurposing the pointerId field?  Is it because of the compat concern you raised in w3c/pointerevents/issues/353?

---

The primary reason is the deterministic nature of deviceId compared to pointerId. The scenario would be: (a) If the device supports getting a unique hardware identifier, we randomly generate an integer and persist that integer for the document lifespan. (b) If the device does not support getting a unique hardware identifier, does pointerId fall back to its original behavior? In the spec, we decided between null, undefined, and integer as the deterministic values of deviceId.

We could use pointerId instead, but we'd have to align on the solution for (b)

I guess we should take this to a GitHub repo somewhere, but I can't help but interject: sounds like a pretty good use case for InputDeviceCapabilities to me (i.e. 'hasDeterministicPointerId') ☺. Perhaps along with a boolean on the event for the limitation case discussed in the explainer.

TIL! Just so I understand this correctly, we would have something like:

event.sourceCapabilities.hasDeterministicPointerId as a check

to let the web developer know that the event.pointerId is a valid device ID?

Jeffrey had made a similar suggestion about mitigating the potential confusion of null/undefined by adding a separate boolean.

Open to moving this discussion to the appropriate forum.

[Rick Byers]

On Wed, Dec 14, 2022 at 4:28 PM Sahir Vellani <Sahir....@microsoft.com> wrote:

---

From: Rick Byers <rby...@chromium.org>
Sent: Wednesday, December 14, 2022 1:05 PM
To: Ben Mathwig <Benjamin...@microsoft.com>
Cc: blink-dev <blin...@chromium.org>; Mustaq Ahmed <mus...@google.com>; fla...@chromium.org <fla...@chromium.org>; Sahir Vellani <Sahir....@microsoft.com>; Ben Mathwig <Benjamin...@microsoft.com>
Subject: Re: [EXTERNAL] Re: [blink-dev] Intent to Prototype: PointerEvent.deviceId for Mult-Pen Inking

 

You don't often get email from rby...@chromium.org. Learn why this is important

On Mon, Dec 12, 2022 at 1:07 PM 'Ben Mathwig' via blink-dev <blin...@chromium.org> wrote:

---

Sahir & Ben: wondering why a new deviceId field seems more logical now against the original proposal of repurposing the pointerId field?  Is it because of the compat concern you raised in w3c/pointerevents/issues/353?

---

The primary reason is the deterministic nature of deviceId compared to pointerId. The scenario would be: (a) If the device supports getting a unique hardware identifier, we randomly generate an integer and persist that integer for the document lifespan. (b) If the device does not support getting a unique hardware identifier, does pointerId fall back to its original behavior? In the spec, we decided between null, undefined, and integer as the deterministic values of deviceId.

We could use pointerId instead, but we'd have to align on the solution for (b)

I guess we should take this to a GitHub repo somewhere, but I can't help but interject: sounds like a pretty good use case for InputDeviceCapabilities to me (i.e. 'hasDeterministicPointerId') ☺. Perhaps along with a boolean on the event for the limitation case discussed in the explainer.

TIL! Just so I understand this correctly, we would have something like:

event.sourceCapabilities.hasDeterministicPointerId as a check

to let the web developer know that the event.pointerId is a valid device ID?

Yep.

Jeffrey had made a similar suggestion about mitigating the potential confusion of null/undefined by adding a separate boolean.

Open to moving this discussion to the appropriate forum.

Filed https://github.com/WICG/input-device-capabilities/issues/43 for lack of a clearly better place. This also has the benefit of already being a WICG spec you could easily augment. But I'm biased, I'm the editor and still kinda sad we never had a reason to expand this beyond the one bool we added years ago the last time we had a pretty specific problem that I proposed a tiny extension to PointerEvents to solve. I forget who it was, but someone at the time argued for a separate extensible pattern like this rather than bolting onto another boolean onto PointerEvent.