Intent to Ship: Idle Detection

[Reilly Grant]

Contact emails

ay...@chromium.org, rei...@chromium.org

Explainer

https://github.com/WICG/idle-detection/blob/main/README.md

Specification

https://wicg.github.io/idle-detection

API spec

Yes

Design docs

https://web.dev/idle-detection/

https://github.com/WICG/idle-detection/blob/main/mdn-drafts/QUICK-REFERENCE.md

Summary

The Idle Detection API notifies developers when a user is idle, indicating such things as lack of interaction with the keyboard, mouse, screen, activation of a screensaver, locking of the screen, or moving to a different screen. A developer-defined threshold triggers the notification.

To prevent a regression in experience for users who have already granted a site permission to access this information we are requesting a seamless transition from Origin Trial to Stable.

Blink component

Blink>Input

TAG review

https://github.com/w3ctag/design-reviews/issues/336

TAG review status

Issues addressed

Origin Trial feedback

Duplicate mobile notifications are a top user complaint for our partner's messaging application. After deploying this Origin Trial their metrics showed that users who granted their site permission to know when they were active on their desktop device received far fewer mobile notifications. Metrics tracking whether users responded to messages however remained steady, showing that this improvement in user experience had no adverse effect on message delivery. In other words, it worked precisely as intended.

Risks

Interoperability and Compatibility

The IdleDetector interface complements existing indicators of user activity such as mouse and keyboard events, and the page visibility state. Sites using an IdleDetector should function correctly in browsers which do not implement it (as they do today) but with all the known problems documented in the motivations for introducing this API.

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/453)

WebKit: Negative (https://lists.webkit.org/pipermail/webkit-dev/2020-October/031565.html)

Web developers: Positive (https://discourse.wicg.io/t/idle-detection-api/2959)

Ergonomics

This API uses asynchronous events and Promise-returning functions to allow Chrome to maintain good performance and future-proof the design.

Activation
https://github.com/dropbox/idle.ts provides a polyfill for a version of this API which only uses input events that are available to the page. The benefit of this API comes from the (limited) information it provides about user input events that are not available to the page. It is possible that this polyfill could be updated to use this API to provide a more complete picture of user presence and activity.

Developer outreach has already begun, including a post about the feature on web.dev.

Security

The security and privacy risks are discussed in the explainer and draft specification.

Debuggability

An option to override the Idle Detection state for testing purposes is available through Chrome DevTools in the Sensors panel.

Is this feature fully tested by web-platform-tests?

Yes

Flag name

IdleDetection

Requires code in //chrome?

True, the IdleDetectionPermissionContext depends on VisibilityTimerTabHelper to implement incognito detection mitigations.

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=878979

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1129665

Sample links

https://web.dev/idle-detection/

https://github.com/WICG/idle-detection/blob/master/HOWTO.md

https://reillyeon.github.io/scraps/idle.html

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/4590256452009984

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msg/blink-dev/OuwzBmH02M4/5ChXdXZQBwAJ

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/9OwINXHzbUE/m/LOcbtEjAAAAJ

This intent message was generated by Chrome Platform Status and hand-crafted with love.

[Thomas Steiner]

https://twitter.com/ChromiumDev/status/1262330843961458688 hat 99 ReTweets and 317 Likes.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEmk%3DMZHEYVTMX3HYb9SsVwSLq%2BjDGGG6CuRyw7C0Gr3nsxd-A%40mail.gmail.com.

--

Thomas Steiner, PhD—Developer Advocate (https://blog.tomayac.com, https://twitter.com/tomayac)

Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.23 (GNU/Linux)

iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom.hTtPs://xKcd.cOm/1181/
-----END PGP SIGNATURE-----

[fri...@jeka.info]

Thomas Steiner schrieb am Dienstag, 27. Juli 2021 um 11:11:29 UTC+2:

On Mon, Jul 26, 2021 at 10:51 PM Reilly Grant <rei...@chromium.org> wrote:

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/453)

WebKit: Negative (https://lists.webkit.org/pipermail/webkit-dev/2020-October/031565.html)

Mozilla's position gets even more negative now:

https://github.com/mozilla/standards-positions/issues/453#issuecomment-888225596

jj

[Chris Harrelson]

Hi Reilly,

Sorry it's taken so long for an API owner to chime in.

Could you reply with more details about the analysis of and mitigations taken to address the kinds of risks raised in the Webkit and Mozilla feedback?

Thanks,

Chris

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/68e36d42-9a51-45a6-92cf-0b4b543092ben%40chromium.org.

[Reilly Grant]

On Mon, Aug 16, 2021 at 1:34 PM Chris Harrelson <chri...@chromium.org> wrote:

Hi Reilly,

Sorry it's taken so long for an API owner to chime in.

Could you reply with more details about the analysis of and mitigations taken to address the kinds of risks raised in the Webkit and Mozilla feedback?

The conversation with Webkit was unfortunately unproductive as they disengaged with the thread before we could reach a shared understanding of the problems the API is trying to solve. The two concrete concerns they raised were fingerprinting and malicious determination of user usage patterns. Mozilla's position similarly cites the latter as a concern.

Our first mitigation for both issues is that the API is only available behind a permission prompt.

Our second mitigation, which targets fingerprinting, is currently being implemented and will add a small degree of error to the timing of when events are delivered to different origins. When combined with other ongoing efforts to reduce cross-origin communication channels this will prevent event timing from being used to identify users across sites.

Finally, I don't believe that the concerns of behavioral tracking are well-founded. This API provides a higher-resolution indication of the current state but in order to establish patterns the site performing the tracking must be continuously active. Such a site, even without this API, can already create such a profile by observing when the site is visible, when the user interacts with it as well as background events such as system suspend and network changes which can be observed by any page allowed to continuously execute script. The additional information provided by this API does not seem particularly novel to such an attack, but is critical for the real-time messaging applications the API is targetting.

[Chris Harrelson]

On Mon, Aug 16, 2021 at 3:25 PM Reilly Grant <rei...@chromium.org> wrote:

Finally, I don't believe that the concerns of behavioral tracking are well-founded. This API provides a higher-resolution indication of the current state but in order to establish patterns the site performing the tracking must be continuously active. Such a site, even without this API, can already create such a profile by observing when the site is visible, when the user interacts with it as well as background events such as system suspend and network changes which can be observed by any page allowed to continuously execute script. The additional information provided by this API does not seem particularly novel to such an attack, but is critical for the real-time messaging applications the API is targetting.

Another argument made by a Webkit representative was that since sites can already do the things you specified above, they can keep doing that and this API is not necessary. The Webkit representative arbitrarily ended the thread at that point without giving you an opportunity to reply, so for the record could you reply to that argument here?

On Wed, Jul 28, 2021 at 8:29 AM fri...@jeka.info <fri...@jeka.info> wrote:

Thomas Steiner schrieb am Dienstag, 27. Juli 2021 um 11:11:29 UTC+2:

On Mon, Jul 26, 2021 at 10:51 PM Reilly Grant <rei...@chromium.org> wrote:

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/453)

WebKit: Negative (https://lists.webkit.org/pipermail/webkit-dev/2020-October/031565.html)

Mozilla's position gets even more negative now:

https://github.com/mozilla/standards-positions/issues/453#issuecomment-888225596

jj

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/68e36d42-9a51-45a6-92cf-0b4b543092ben%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEmk%3DMZ7W6-pucx95eAZ8K-DBwQEZ%3DSLtewgkZVHts59xsARZw%40mail.gmail.com.

[Reilly Grant]

On Mon, Aug 16, 2021, 7:11 PM Chris Harrelson <chri...@chromium.org> wrote:

On Mon, Aug 16, 2021 at 3:25 PM Reilly Grant <rei...@chromium.org> wrote:

Finally, I don't believe that the concerns of behavioral tracking are well-founded. This API provides a higher-resolution indication of the current state but in order to establish patterns the site performing the tracking must be continuously active. Such a site, even without this API, can already create such a profile by observing when the site is visible, when the user interacts with it as well as background events such as system suspend and network changes which can be observed by any page allowed to continuously execute script. The additional information provided by this API does not seem particularly novel to such an attack, but is critical for the real-time messaging applications the API is targetting.

Another argument made by a Webkit representative was that since sites can already do the things you specified above, they can keep doing that and this API is not necessary. The Webkit representative arbitrarily ended the thread at that point without giving you an opportunity to reply, so for the record could you reply to that argument here?

This API is impossible to polyfill because the definition of "idle" it uses includes input to the system as a whole and not the currently focused window. This is necessary in order to achieve the goal, which is to allow a window that is currently in the background to determine whether or not the user is still actively using the system. As an example, imagine that you were editing a document in one tab while chatting with a coworker in another. Once you switch to editing the document the messaging app cannot observe whether you are still at your computer because it is no longer visible or focused and so gets no input events. You could be using another app or you could've gotten up to make coffee. The feedback from users of messaging applications is that it is annoying when their phone constantly notifies them of new messages when they are still at their desk and using their computer. This API is necessary to provide a signal that can differentiate between these two cases.

[Chris Harrelson]

Thanks for all these responses!

LGTM1

[Thomas Steiner]

For the record, the polyfillable part of the API (that is, detect if the user has the tab focused and is interacting with the document) was requested by developers as a special "mode" one could set when activating idle detection. See https://github.com/WICG/idle-detection/issues/36 for the details. The use case here was not notification deduplication, but collaboration on a shared document.

[Yoav Weiss]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALgRrLkFeQszAdrceNCDU4%2BrO36%3DS_deq3T4iVgDjZ1D%2BbhOmw%40mail.gmail.com.

[Alex Russell]

LGTM3 and apologies for the delay. Part of the reason for the slow reply was an attempt to tackle the arguments that Tantek raised here which I suspect will be used on a loop as this conversation moves forward with respect to other APIs that have similarly narrow-but-important use-cases:

https://github.com/mozilla/standards-positions/issues/453#issuecomment-888225596

Briefly, we should note the structural protections built into the API shape:

• The usual TLS and top-document restrictions are in place for powerful APIs.
• Integration with the Permissions API is provided and use is gated on an async request (per TAG design guidance)

These might not sound like much, but together they create the substrate we need to allow UAs to fully control the UI they present to users about this capability. Scenarios that are enabled by this, based on browser policy, could include any policy regime (from most to least restrictive):

• Blanket denying the capability
• Only allowing the permission to be granted post some user-consent gate and with other gating factors, e.g.:
• Only if installed as a PWA
• Only if the site has a high-enough "reputation"
• Only if the site has garnered enough engagement (e.g. chrome://site-engagement score) on the local device
• Only if the user also grants other permissions more associated with classes of applicsations that have "legitimate" (however that's defined) uses in the eyes of the browser vendor
• Only if developers register their use with the browser (e.g., Chromium Origin Trials)
• Only allowing post user consent
• Potential grants on any page that meets minimum technical criteria

The key point here is that movement between these states won't require an API re-design or for developers to experience breaking changes (in the form of exceptions, although UI treatments may be in want of updating in response to policy changes). All of that argues in opposition to the claim that this is harmful on some ground that cannot be mitigated by sufficiently invested and interested browser vendors. 

Should harms of the sort Tantek raises arise, not only is the UA more in charge of usage reporting when developers adopt the Idle Detection API (as opposed to today's piles of hacks), but semantic transparency provides space for UAs to be more aggressive in tamping down the sorts of fingerprinting-esque behavior we see right now.

For all of those reasons, in this and in other APIs that provide explicit surface for what was previously an implicit hack-based mechanism that was not similarly limited, cabined, and semantically transparent, my vote will be an LGTM. It is bad policy for UAs to be in the business of saying "something bad might happen!" rather than weighing up the real consequences and maintaining a position through APIs to most-probably mitigate harms directly.

Abdication is not a path forward.

Thanks for making this possible, Reilly.

Regards,

Alex

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWF%3DUpG-MZkz8UsUDofF0zQ_w1_-nK6a9H00rhyNzUTcw%40mail.gmail.com.

[Alan Stearns]

I am a bit confused on this point. If the new explicit surface does not also disable the previous hack, I don’t see how adding the new surface (however it might theoretically be limited) is an improvement.

[Alex Russell]

Hey Alan,

Good question. Let me try again. Progress in stable platforms is often a two-step process. Not leaping blindly is an asset, over time.

If we did not have a strategic interest in a platform that is sufficiently capable to meet developer needs, we wouldn't need to be responsible towards developers as well as users. That, however, is not the situation we're in. Winning developers to "our side" (building on the web) requires developing and retaining their trust[1] while making changes that improve life for users. This is why we take deprecations seriously, set usage limits for OTs to prevent "burn in", etc. We don't YOLO changes on our platform because our reputation for responsible evolution underpins nearly everything else we attempt.

So, when UAs begin to introduce policies like the Privacy Sandbox which will begin to take a more sophisticated approach to capping fingerprinting, we need to have semantically transparent APIs in place that developers can migrate towards to the extent that we care about developer success. If we didn't have that balance in mind (and, as always, putting users first), any policy would do.

Because we do not want to lose developers[2] while repair things for users, providing transition windows and alternatives is the responsible thing to do. Many things we do in evolving the web responsibly look like switch-backs rather than direct runs to the summit. That's better because it makes it more likely we'll bring everyone to the summit, not just folks who can re-work the content of their sites every quarter.

Regards

[1]: https://infrequently.org/2020/06/platform-adjacency-theory/

[2]: https://infrequently.org/2021/07/the-core-web-loop/

[Chris Harrelson]

Hi Alan,

Here is how I would put it.

1. This API addresses a real user need. One I encounter almost every day is removing duplicate notifications across devices, and it's very annoying. So there is a user benefit, not just developer benefit, and it is substantial. Per the origin trial feedback:

"Duplicate mobile notifications are a top user complaint for our partner's messaging application. After deploying this Origin Trial their metrics showed that users who granted their site permission to know when they were active on their desktop device received far fewer mobile notifications. Metrics tracking whether users responded to messages however remained steady, showing that this improvement in user experience had no adverse effect on message delivery. In other words, it worked precisely as intended."

2. A fit for purpose API allows for a more reliable user and developer experience. As Reilly also noted, there is no way to polyfill this API.

3. A fit for purpose API also allows user agents to more easily change behavior elsewhere (e.g. throttling, freezing or discarding tabs) without worrying about breaking the idle detection use cases, thereby providing user value, and  rendering the current hacks to track user behavior less effective. It also allows the user agent to assist all sites with better heuristics and signals over time, thereby providing user value.

4. There are mitigations in place for known risks, as outlined by Reilly. If future attacks surface (as they always do in computing these days, regardless of new or old API), there are clear ways to address them because they would by necessity involve the idle detection API, which can be restricted or adjusted to respond.

Chris

On Fri, Aug 20, 2021 at 10:02 AM Alan Stearns <famu...@gmail.com> wrote:

Thanks for the additional background. I still do not see how this “tackles the arguments” I have seen in Mozilla and WebKit’s objections to this feature.

I understand the objections to be something like “the user harm inherent in this feature outweighs the developer benefit.” So it looks to me like the argument is over whether this summit is worth approaching at all, directly or indirectly. To my mind, tacking the arguments presented would either include showing why the developer benefit is being devalued, or providing additional mitigations for user harm (to the current proposal, not theoretical future protections).

I do get the argument that this is a relatively responsible way to work on the feature. But when your vantage point is “this is a relatively responsible way to add user harm to the platform” it is not very convincing.

[Alan Stearns]

Thanks Chris. I find that much more responsive and relevant to the objections I have seen.

[Alan Stearns]

Thanks for the additional background. I still do not see how this “tackles the arguments” I have seen in Mozilla and WebKit’s objections to this feature.

I understand the objections to be something like “the user harm inherent in this feature outweighs the developer benefit.” So it looks to me like the argument is over whether this summit is worth approaching at all, directly or indirectly. To my mind, tacking the arguments presented would either include showing why the developer benefit is being devalued, or providing additional mitigations for user harm (to the current proposal, not theoretical future protections).

I do get the argument that this is a relatively responsible way to work on the feature. But when your vantage point is “this is a relatively responsible way to add user harm to the platform” it is not very convincing.

On Friday, August 20, 2021 at 9:39:07 AM UTC-7 sligh...@chromium.org wrote:

[Alex Russell]

On Fri, Aug 20, 2021 at 10:02 AM Alan Stearns <famu...@gmail.com> wrote:

Thanks for the additional background. I still do not see how this “tackles the arguments” I have seen in Mozilla and WebKit’s objections to this feature.

I understand the objections to be something like “the user harm inherent in this feature outweighs the developer benefit.” So it looks to me like the argument is over whether this summit is worth approaching at all, directly or indirectly. To my mind, tacking the arguments presented would either include showing why the developer benefit is being devalued, or providing additional mitigations for user harm (to the current proposal, not theoretical future protections).

There is an unsubstantiated claim of user harm here, to which Hitchen's Razor applies. I had assumed that was clear, which is why I outlined the space of possibilities, since the status quo won't change much (except to potentially reduce already frustrating user experiences) without other changes that tamp down on the fingerprinting/tracking potential.

If folks want to bring quantification and weigh up impacts, I (and the rest of the OWNERs) welcome that. But polemic without data isn't much of an argument.