Intent to Ship: PointerEvent.deviceId for Mult-Pen Inking

[Sahir Vellani]

Contact emails

gerc...@microsoft.com, sahir....@microsoft.com

Explainer

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

Specification

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

Summary

As devices with advanced pen input capabilities are becoming increasingly prevalent, it is important that the web platform continues to evolve to fully support these advanced features in order to unlock rich experiences for both end users and developers. One such advancement is the ability for a device's digitizer to recognize more than one pen device interacting with it simultaneously. This feature is an extension to the PointerEvent interface to include a new attribute, deviceId, that represents a session-persistent, document isolated, unique identifier that a developer can reliably use to identify individual pens interacting with the page.

Blink component

Blink>Input

TAG review

https://github.com/w3ctag/design-reviews/issues/880

TAG review status

Pending. TAG review has been pending for 2 months.

Risks

Interoperability and Compatibility

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/715)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/102)

Web developers: No signals

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

Is this feature fully tested by web-platform-tests?

No. However, there are web tests in Chromium that test PointerEventInit with this feature.

Flag name on chrome://flags

PointerEventDeviceId

Finch feature name

Non-finch justification

Edge origin trials successfully underway.

Requires code in //chrome?

False

Measurement

PointerEventDeviceId use counter implemented.

Availability expectation

Initially available on Chromium browsers on Windows.

Adoption expectation

Feature is used by specific partner(s) to provide functionality immediately upon launch.

Adoption plan

This feature has been through origin trials on Edge. It is being used by partners that provide features on devices with multi pen support.

Non-OSS dependencies

Does the feature depend on any code or APIs outside the Chromium open source repository and its open-source dependencies to function?

Operating system API's are used to obtain the device id, and are necessary for this feature to function. Please see the security questionnaire in the TAG review on security and privacy concerns related to the use of these APIs.

Estimated milestones

Shipping on desktop

120

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

WICG Proposal: https://github.com/WICG/proposals/issues/101 No web compat/interop risk.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5114132234240000

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SA0PR00MB1033E5DE0BDE42239E647E9AFB189%40SA0PR00MB1033.namprd00.prod.outlook.com

This intent message was generated by Chrome Platform Status.

[Mike Taylor]

Any Origin Trial feedback you can share?

Requires code in //chrome?

False

Measurement

PointerEventDeviceId use counter implemented.

Availability expectation

Initially available on Chromium browsers on Windows.

Out of curiosity, are there limitations on other platforms that prevent supporting this feature?

Adoption expectation

Feature is used by specific partner(s) to provide functionality immediately upon launch.

Adoption plan

This feature has been through origin trials on Edge. It is being used by partners that provide features on devices with multi pen support.

Non-OSS dependencies

Does the feature depend on any code or APIs outside the Chromium open source repository and its open-source dependencies to function?

Operating system API's are used to obtain the device id, and are necessary for this feature to function. Please see the security questionnaire in the TAG review on security and privacy concerns related to the use of these APIs.

Estimated milestones

Shipping on desktop

120

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

WICG Proposal: https://github.com/WICG/proposals/issues/101 No web compat/interop risk.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5114132234240000

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SA0PR00MB1033E5DE0BDE42239E647E9AFB189%40SA0PR00MB1033.namprd00.prod.outlook.com

This intent message was generated by Chrome Platform Status.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR00MB1349B7917876E7AC505E790AFBCBA%40PH0PR00MB1349.namprd00.prod.outlook.com.

[Sahir Vellani]

Absolutely, the feature has been working well. Our partners (Microsoft Whiteboard) have enabled the feature that is dependent on this API for their general audience! We did not receive any constructive feedback. This API is being used by them on Microsoft Surface Hub devices, which support multi-pen inking.

Requires code in //chrome?

False

Measurement

PointerEventDeviceId use counter implemented.

Availability expectation

Initially available on Chromium browsers on Windows.

Out of curiosity, are there limitations on other platforms that prevent supporting this feature?

We haven't been able to get our hands on hardware that supports other platforms in addition to multi pen inking in order to implement and appropriately test this feature. We welcome any sponsors for implementing and testing this, especially on Linux/Android.

[Daniel Bratell]

I see that various mandatory steps in chromestatus (privacy,security,enterprise,debuggability,testing) seems to be unstarted. It is possible they were made mandatory after you create the entry, but it would be good if you could get them started now at least.

Also, it's unfortunate that TAG and standards positions requests have not resulted in anything, but that is hardly your fault. There were some questions in the WebKit request. Is all that ok now?

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c8f16bc4-8d21-450b-9178-964cba818a68n%40chromium.org.

[Sahir Vellani]

Thanks for the feedback, I wasn't aware they were mandatory. The steps have been started, just awaiting feedback from Security and Privacy reviewers. I've received LGTMs for all other areas. After our response to WebKit's question, they did not have any further follow-up questions. So I'm assuming all is well.

[Mike Taylor]

LGTM1

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8077a67d-4104-48c6-9f9b-09f9e6c8b022n%40chromium.org.

[Yoav Weiss]

On Tue, Oct 17, 2023 at 12:42 AM Mike Taylor <mike...@chromium.org> wrote:

LGTM1

On 10/15/23 11:07 AM, 'Sahir Vellani' via blink-dev wrote:

Thanks for the feedback, I wasn't aware they were mandatory. The steps have been started, just awaiting feedback from Security and Privacy reviewers. I've received LGTMs for all other areas. After our response to WebKit's question, they did not have any further follow-up questions. So I'm assuming all is well.

On Wednesday, October 11, 2023 at 4:59:15 AM UTC-7 Daniel Bratell wrote:

I see that various mandatory steps in chromestatus (privacy,security,enterprise,debuggability,testing) seems to be unstarted. It is possible they were made mandatory after you create the entry, but it would be good if you could get them started now at least.

Also, it's unfortunate that TAG and standards positions requests have not resulted in anything, but that is hardly your fault. There were some questions in the WebKit request. Is all that ok now?

/Daniel

On 2023-10-06 20:10, 'Sahir Vellani' via blink-dev wrote:

On Friday, October 6, 2023 at 9:03:35 AM UTC-7 mike...@chromium.org wrote:

On 10/4/23 7:43 PM, 'Sahir Vellani' via blink-dev wrote:

Contact emails

gerc...@microsoft.com, sahir....@microsoft.com

Explainer

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

Specification

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/PointerEventDeviceId/explainer.md

Is there a more formal spec for this?

Any support outside of Microsoft that would enable y'all to move this to the WICG?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/13bad65e-6276-4567-b6e3-0961e44bc6d1%40chromium.org.

[Mike Taylor]

Apologies, I missed that there wasn't yet a spec. I'll retract my LGTM until then.

[Alex Russell]

I agree that this needs a spec PR and the explainer should at least migrate to WICG before we agree to ship. Also, can you please link to the TAG review?

Best,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8077a67d-4104-48c6-9f9b-09f9e6c8b022n%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Sahir Vellani]

Hi all, good news!

Reviving this thread because we have accomplished:

1. TAG Review Completion:  Extending the PointerEvent with Unique DeviceId Attribute · Issue #880 · w3ctag/design-reviews (github.com) Resolution: Satisfied

2. WICG Repository for standardization discussions. Link to explainer in WICG Repo:  pointer-event-extensions/pointer-event-device-id-explainer.md at main · WICG/pointer-event-extensions (github.com)

3. A PR against the PointerEvent spec with proposed changes:  Add deviceId to PointerEvent spec by sahirv · Pull Request #495 · w3c/pointerevents (github.com) We will be waiting for Spec Level 3 to come out before this can be merged; but this provides an official starting point for the standardized description of this feature. Based on the feedback received, I don't anticipate any major changes to the design.

Reviewers, can we please get another review for shipping this feature?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8077a67d-4104-48c6-9f9b-09f9e6c8b022n%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/13bad65e-6276-4567-b6e3-0961e44bc6d1%40chromium.org.

[Vladimir Levin]

On Thu, Jan 11, 2024 at 6:08 PM 'Sahir Vellani' via blink-dev <blin...@chromium.org> wrote:

Hi all, good news!

Reviving this thread because we have accomplished:

1. TAG Review Completion:  Extending the PointerEvent with Unique DeviceId Attribute · Issue #880 · w3ctag/design-reviews (github.com) Resolution: Satisfied

2. WICG Repository for standardization discussions. Link to explainer in WICG Repo:  pointer-event-extensions/pointer-event-device-id-explainer.md at main · WICG/pointer-event-extensions (github.com)

3. A PR against the PointerEvent spec with proposed changes:  Add deviceId to PointerEvent spec by sahirv · Pull Request #495 · w3c/pointerevents (github.com) We will be waiting for Spec Level 3 to come out before this can be merged; but this provides an official starting point for the standardized description of this feature. Based on the feedback received, I don't anticipate any major changes to the design.

For posterity, I was initially unsure why this wasn't an issue on the w3c/pointerevents, but it does seem like the discussion happened there and folks agreed to move this in WICG: https://github.com/w3c/pointerevents/issues/353 \o/

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/cbc6e96b-0165-4e28-8f16-786f0dea7ac8n%40chromium.org.

[Yoav Weiss (@Shopify)]

On Wednesday, January 17, 2024 at 3:51:59 PM UTC+1 vmp...@google.com wrote:

On Thu, Jan 11, 2024 at 6:08 PM 'Sahir Vellani' via blink-dev <blin...@chromium.org> wrote:

Hi all, good news!

Reviving this thread because we have accomplished:

1. TAG Review Completion:  Extending the PointerEvent with Unique DeviceId Attribute · Issue #880 · w3ctag/design-reviews (github.com) Resolution: Satisfied

2. WICG Repository for standardization discussions. Link to explainer in WICG Repo:  pointer-event-extensions/pointer-event-device-id-explainer.md at main · WICG/pointer-event-extensions (github.com)

3. A PR against the PointerEvent spec with proposed changes:  Add deviceId to PointerEvent spec by sahirv · Pull Request #495 · w3c/pointerevents (github.com) We will be waiting for Spec Level 3 to come out before this can be merged; but this provides an official starting point for the standardized description of this feature. Based on the feedback received, I don't anticipate any major changes to the design.

Thanks for the PR! Was it reviewed by other WG members?

For example, "User agents MAY reserve a generic `deviceId` value of `0` or `1` for events generated by the primary mouse device." seems risky from an interop perspective. E.g. developers may rely on some UAs doing that and fail when others don't.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8077a67d-4104-48c6-9f9b-09f9e6c8b022n%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/13bad65e-6276-4567-b6e3-0961e44bc6d1%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Sahir Vellani]

Appreciate the feedback! Yes the PR was reviewed by WG members for any major concerns; but I believe there will be more comprehensive feedback once the Level 3 spec lands. Regarding your concern, the language is based on that of PointerEvent.pointerId. The ultimate goal here is to differentiate between devices, and like pointerId, the way the ids are assigned has been left to the UA. I think web developers should be able to rely on PointerEvent.pointerType to confirm whether the pointer event comes from the primary mouse device. In Chromium, we transmit 1 for the mouse for PointerEvent.deviceId.

[Mike Taylor]

Forgive my ignorance around this API generally, but is there any reason the spec can't require a single value? If not, why not?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6afb2718-89f6-47d5-9052-d5ab2a8f849dn%40chromium.org.

[Sahir Vellani]

I can't say what the reasoning is for that behavior in PointerEvent.pointerId, as I was not involved. However, I will make a change in the spec to only use the value of 1 for primary mouse device.

There may be a scenario where PointerEvent.deviceId is unsupported by the UA (separate from an invalid id of -1); e.g, on platforms where the feature is unimplemented. In that case, developers might have a check like if(event.deviceId) {...}. If the deviceId is 0 for a valid reason, it will fail that check. I see no harm in limiting the deviceId from primary mouse to 1. It will avoid this interop issue and make the feature more friendly to web developers.

[Sahir Vellani]

Just to clarify, I've made the change in the deviceId verbiage in the spec, not pointerId :)

[Sahir Vellani]

Hi all, any more questions or concerns?

[Robert Flack]

FWIW, in the PEWG call last week there was some question of how this relates to the pen customizations proposal. I suppose the general question is whether this should be an additional part of some hardware specific device customization in pointerEvent.penCustomizationsDetails. I think there is a risk of shipping this without consensus on the general shape, as it will make it harder to change if it's decided there's a better way to expose the information without breaking existing uses. We should probably take the API that has been incubated and bring it back for discussion in the working group.

On Tue, Jan 23, 2024 at 12:15 PM 'Sahir Vellani' via blink-dev <blin...@chromium.org> wrote:

Hi all, any more questions or concerns?

On Thursday, January 18, 2024 at 8:44:58 PM UTC-8 Sahir Vellani wrote:

Just to clarify, I've made the change in the deviceId verbiage in the spec, not pointerId :)

On Thursday, January 18, 2024 at 8:40:00 PM UTC-8 Sahir Vellani wrote:

I can't say what the reasoning is for that behavior in PointerEvent.pointerId, as I was not involved. However, I will make a change in the spec to only use the value of 1 for primary mouse device.

There may be a scenario where PointerEvent.deviceId is unsupported by the UA (separate from an invalid id of -1); e.g, on platforms where the feature is unimplemented. In that case, developers might have a check like if(event.deviceId) {...}. If the deviceId is 0 for a valid reason, it will fail that check. I see no harm in limiting the deviceId from primary mouse to 1. It will avoid this interop issue and make the feature more friendly to web developers.

0 was originally reserved to mean a non-pointer device. This was discussed in https://github.com/w3c/pointerevents/issues/343 where we discovered that Firefox has, and continues to use 0 for mouse pointer events where chrome and safari use 1.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8ac964b1-99a0-4c7d-b9eb-d02b26bdb306n%40chromium.org.

[Rick Byers]

This is a pretty niche and tiny addition which matters a lot to probably a couple websites in the world, and not at all to everyone else. It's a shame that it's taken over a year of discussion on what to ship. Worst case and we get this wrong and have something better in the future, I can't imagine it would be hard to change over and deprecate this. Sahir, do you agree? If so, I don't think it's a good use of resources to spend a lot more time debating this tiny addition, the stakes are IMHO not high here.

However, I definitely think we want to give PEWG a fair chance to weigh in on the spec PR. The PR was just filed 2 weeks ago, so maybe another week delay to wait for feedback on it is reasonable? Rob, can you get this on the agenda for the next PEWG call and indicate that I think Chromium should just ship it as-is unless there's a concrete counter-proposal in the next couple weeks?

Rick

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJh39TMzzU%2B8e%3DNPTLTGGKBBRg3FCdSK%2BQnkPv87pZ5RBnuGHw%40mail.gmail.com.

[Robert Flack]

On Tue, Jan 23, 2024 at 1:00 PM Robert Flack <fla...@chromium.org> wrote:

FWIW, in the PEWG call last week there was some question of how this relates to the pen customizations proposal. I suppose the general question is whether this should be an additional part of some hardware specific device customization in pointerEvent.penCustomizationsDetails. I think there is a risk of shipping this without consensus on the general shape, as it will make it harder to change if it's decided there's a better way to expose the information without breaking existing uses. We should probably take the API that has been incubated and bring it back for discussion in the working group.

I've raised this for discussion to try to reach consensus on the general idea. I also realize my comment comes across more critical than I meant it to be. There is risk to locking in the current proposal, but I also suspect we'd be able to make a breaking change if needed, so I did not intend my comment to be blocking. Apologies!

[Robert Flack]

On Tue, Jan 23, 2024 at 2:00 PM Rick Byers <rby...@chromium.org> wrote:

This is a pretty niche and tiny addition which matters a lot to probably a couple websites in the world, and not at all to everyone else. It's a shame that it's taken over a year of discussion on what to ship. Worst case and we get this wrong and have something better in the future, I can't imagine it would be hard to change over and deprecate this. Sahir, do you agree? If so, I don't think it's a good use of resources to spend a lot more time debating this tiny addition, the stakes are IMHO not high here.

However, I definitely think we want to give PEWG a fair chance to weigh in on the spec PR. The PR was just filed 2 weeks ago, so maybe another week delay to wait for feedback on it is reasonable? Rob, can you get this on the agenda for the next PEWG call and indicate that I think Chromium should just ship it as-is unless there's a concrete counter-proposal in the next couple weeks?

Absolutely! Happy to!

[Sahir Vellani]

Thanks Rick and Robert! 

Rick, I agree that it would be relatively easy to change/deprecate deviceId. There are not too many devices at the moment that support multiple pens, and not many web apps either. I think the cost to the websites of not having Chrome support this for the foreseeable future is greater than changing where the deviceId gets read from. The Pen Customizations api looks quite cool, although deviceId is more generic.

Robert, I'm happy to wait a week or two more for the PEWG to discuss this further. Thanks for putting deviceId in the agenda, and of course we can rework this if a more appropriate alternative is proposed. :)

[Yoav Weiss (@Shopify)]

Thanks for chiming in, Robert!

Sahir - can you let us know once the PEWG has discussed this and you feel this is good to go?

[Robert Flack]

We just discussed this yesterday where the consensus was that we thought it would be pragmatic to put this new device specific identifier into a common structure on the pointer event where other device specific customizations could later be added (e.g. those from the pen customizations proposal). This avoids needing to have new properties on the pointer event for each device specific customization attribute added.

Another question that came up was how universal this customization was, i.e. whether there would be support for this on other platforms.

@Sahir Vellani would you be able to suggest what you think the best structure name / property name for pointer specific identifier properties might be?

I do agree with @Rick Byers that this is likely a niche API which we'll likely be able to change, I'm just trying to get ahead of breaking changes early.

[Sahir Vellani]

Thanks for the update Robert! That makes sense. 

What about deviceProperties? That would place the emphasis on the device generating the pointer event; with "properties" being generic enough to include different supported functions of the device. So we would have something like PointerEvent.deviceProperties.deviceId/PointerEvent.deviceProperties.preferredInkingColor etc.

Would you be able to clarify the next steps once the structure name has been agreed upon? I believe we'll need to update the explainer, spec PR and change the chromium implementation. Anything else?

[Robert Flack]

I think deviceProperties is reasonable. The other question was when deviceId would be populated? E.g. should touches on the same touchscreen device have the same deviceId or should we only have a deviceId when we can uniquely identify the particular pointer? I suspect maybe the latter since otherwise you couldn't infer uniqueness from its existence.

Could you comment on the pointerevents issue so that the context is there for the next meeting? I'll request that it be discussed again.

On Thu, Feb 1, 2024 at 4:31 PM 'Sahir Vellani' via blink-dev <blin...@chromium.org> wrote:

Thanks for the update Robert! That makes sense. 

What about deviceProperties? That would place the emphasis on the device generating the pointer event; with "properties" being generic enough to include different supported functions of the device. So we would have something like PointerEvent.deviceProperties.deviceId/PointerEvent.deviceProperties.preferredInkingColor etc.

Would you be able to clarify the next steps once the structure name has been agreed upon? I believe we'll need to update the explainer, spec PR and change the chromium implementation. Anything else? 

This sounds good.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/42fff33f-62b2-4179-82aa-177fc758f52bn%40chromium.org.