Intent to implement: System Keyboard Lock

[Gary Kačmarčík (Кошмарчик)]

Contact emails

Eng: gar...@chromium.org, jamie...@chromium.org, domi...@chromium.org, mgi...@chromium.org

PM: dah...@chromium.org

Spec

Explainer: https://github.com/jondahlke/system-keyboard-lock/blob/master/EXPLAINER.md

WICG Discourse: https://discourse.wicg.io/t/proposal-system-keyboard-lock-api/1594

Draft Spec: https://rawgit.com/garykac/system-keyboard-lock/master/index.html

Summary

While in fullscreen, this API allows the app to request to receive keys that are normally handled by the system or the browser (like Cmd/Alt-Tab or Esc). If granted, the user would be able to exit System Keyboard Lock (and fullscreen) by holding the Esc key for 2 seconds.

Motivation

Some web apps, like remote access apps and games, need access to system keys so that they can handle them in the way that the user expects:

• In a fullscreen connection to a remote computer, the user expects system level keyboard shortcuts to work on the remote machine.

• In games that are ported to, emulated on, or streamed to the web, the Esc key may be used to bring up a context menu or to escape from in-game UI. In these case, the user expects the key to be handled appropriately by the game.

Currently, apps address this problem (in Chrome) by using Chrome App APIs. However, the recent deprecation announcement means that we need equivalents in the open web.

Interoperability and Compatibility Risk

Low/Medium.

This feature is not currently available in any browser, so adding it will not break any existing content.

This feature is a recurring request from web developers (specifically makers of remote access and ssh apps) and users (most often by users of those applications). See related user bugs: 84332, 119881, 162504, 175724, 223118, 436670 and note that the ones marked "Fixed" are because of the (soon-to-be-deprecated) Chrome App APIs.

However, this feature does not appear to be a high priority for other browser vendors, so the risk is MEDIUM that they won't get around to implementing it. This feature was discussed at TPAC during the Web Platform meeting (present: Edge, FF, Yandex), and it was agreed that this was useful although concerns were raised about security (cf. fullscreen). See discussion at the end of https://www.w3.org/2016/09/23-webapps-minutes.html#item11

And finally, note that while this feature is important for the scenarios mentioned above, if it is missing it will not break the entire app, but will instead result in a degraded (but still functional) experience.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Windows, Mac, Linux and ChromeOS.

As currently defined, this feature is a no-op on mobile platforms (although we're investigating what should happen when an external keyboard is connected to a mobile device).

OWP launch tracking bug

https://crbug.com/671774

Link to entry on the feature dashboard

https://www.chromestatus.com/features/5642959835889664

Requesting approval to ship?

Not at this time.

[Philip Jägenstedt]

The explainer says "System keyboard lock is only available when the document is in fullscreen mode, and only from a secure origin". If there's little chance that this would ever work independent of Fullscreen, would it then make sense to do this as an option to requestFullscreen() instead? Maybe element.requestFullscreen({ allowKeyCodes: [27] })?

From https://github.com/jondahlke/system-keyboard-lock/blob/master/EXPLAINER.md#sample-user-experience-flow it looks like there are no new prompts, but just a change in the messaging for how to exit fullscreen. That doesn't seem possible if the request to get esc key events happens after the fullscreen request, which happens in the sample code.

[smaug]

Have you asked feedback about this from security/privacy folks, as was requested @TPAC?


-Olli

On 12/14/2016 12:55 AM, Philip Jägenstedt wrote:
> The explainer says "System keyboard lock is only available when the document is in fullscreen mode, and only from a secure origin". If there's little
> chance that this would ever work independent of Fullscreen, would it then make sense to do this as an option to requestFullscreen() instead? Maybe
> element.requestFullscreen({ allowKeyCodes: [27] })?
>
> From https://github.com/jondahlke/system-keyboard-lock/blob/master/EXPLAINER.md#sample-user-experience-flow it looks like there are no new prompts,
> but just a change in the messaging for how to exit fullscreen. That doesn't seem possible if the request to get esc key events happens after the
> fullscreen request, which happens in the sample code.
>

> On Tue, Dec 13, 2016 at 11:32 PM 'Gary Kačmarčík (Кошмарчик)' via blink-dev <blin...@chromium.org <mailto:blin...@chromium.org>> wrote:
>
> Contact emails
>

> Eng: gar...@chromium.org <mailto:gar...@chromium.org>, jamie...@chromium.org <mailto:jamie...@chromium.org>, domi...@chromium.org
> <mailto:domi...@chromium.org>, mgi...@chromium.org <mailto:mgi...@chromium.org>
>
> PM: dah...@chromium.org <mailto:dah...@chromium.org>

>
>
> Spec
>
> Explainer: https://github.com/jondahlke/system-keyboard-lock/blob/master/EXPLAINER.md
>
> WICG Discourse: https://discourse.wicg.io/t/proposal-system-keyboard-lock-api/1594
>
> Draft Spec: https://rawgit.com/garykac/system-keyboard-lock/master/index.html
>
>
> Summary
>
> While in fullscreen, this API allows the app to request to receive keys that are normally handled by the system or the browser (like Cmd/Alt-Tab
> or Esc). If granted, the user would be able to exit System Keyboard Lock (and fullscreen) by holding the Esc key for 2 seconds.
>
>
> Motivation
>
> Some web apps, like remote access apps and games, need access to system keys so that they can handle them in the way that the user expects:
>

> *

>
> In a fullscreen connection to a remote computer, the user expects system level keyboard shortcuts to work on the remote machine.
>

> *

>
> In games that are ported to, emulated on, or streamed to the web, the Esc key may be used to bring up a context menu or to escape from in-game
> UI. In these case, the user expects the key to be handled appropriately by the game.
>
> Currently, apps address this problem (in Chrome) by using Chrome App APIs. However, the recent deprecation announcement

> <https://blog.chromium.org/2016/08/from-chrome-apps-to-web.html>means that we need equivalents in the open web.

>
>
> Interoperability and Compatibility Risk
>
> Low/Medium.
>
> This feature is not currently available in any browser, so adding it will not break any existing content.
>
> This feature is a recurring request from web developers (specifically makers of remote access and ssh apps) and users (most often by users of

> those applications). See related user bugs: 84332 <https://bugs.chromium.org/p/chromium/issues/detail?id=84332>, 119881
> <https://bugs.chromium.org/p/chromium/issues/detail?id=119881>, 162504 <https://bugs.chromium.org/p/chromium/issues/detail?id=162504>, 175724
> <https://bugs.chromium.org/p/chromium/issues/detail?id=175724>, 223118 <https://bugs.chromium.org/p/chromium/issues/detail?id=223118>, 436670
> <https://bugs.chromium.org/p/chromium/issues/detail?id=436670>and note that the ones marked "Fixed" are because of the (soon-to-be-deprecated)

> Chrome App APIs.
>
> However, this feature does not appear to be a high priority for other browser vendors, so the risk is MEDIUM that they won't get around to
> implementing it. This feature was discussed at TPAC during the Web Platform meeting (present: Edge, FF, Yandex), and it was agreed that this was
> useful although concerns were raised about security (cf. fullscreen). See discussion at the end of
> https://www.w3.org/2016/09/23-webapps-minutes.html#item11
>
> And finally, note that while this feature is important for the scenarios mentioned above, if it is missing it will not break the entire app, but
> will instead result in a degraded (but still functional) experience.
>
>
> Ongoing technical constraints
>
> None.
>
>
> Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
>
> Windows, Mac, Linux and ChromeOS.
>
> As currently defined, this feature is a no-op on mobile platforms (although we're investigating what should happen when an external keyboard is
> connected to a mobile device).
>
>
> OWP launch tracking bug
>
> https://crbug.com/671774
>
>
> Link to entry on the feature dashboard
>
> https://www.chromestatus.com/features/5642959835889664
>
>
> Requesting approval to ship?
>
> Not at this time.
>
>

> --
> You received this message because you are subscribed to the Google Groups "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org
> <mailto:blink-dev+...@chromium.org>.

[Gary Kačmarčík (Кошмарчик)]

On Tue, Dec 13, 2016 at 3:10 PM, smaug <sm...@welho.com> wrote:

Have you asked feedback about this from security/privacy folks, as was requested @TPAC?

We ran this past Chrome's security/privacy folks before TPAC.

During TPAC, my recollection is that you said wanted to run this past Mozilla's security/privacy, and I asked if they could make any comments on the WICG to keep the discussions open.

In retrospect, when I didn't see any comments I *should* have followed up with you so that I could ping people directly. Who would be appropriate people for me to reach out to @mozilla?

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org
<mailto:blink-dev+unsubscribe@chromium.org>.

[Gary Kačmarčík (Кошмарчик)]

On Tue, Dec 13, 2016 at 2:55 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

The explainer says "System keyboard lock is only available when the document is in fullscreen mode, and only from a secure origin". If there's little chance that this would ever work independent of Fullscreen, would it then make sense to do this as an option to requestFullscreen() instead? Maybe element.requestFullscreen({ allowKeyCodes: [27] })?

There are a number of valid use cases where this feature makes sense without fullscreen:

• Remote desktop connection where the client has a large screen (or multiple screens), or has multiple connections and does not want to be in fullscreen.
• Similarly for Secure Shell users, except that they are more likely to have multiple windows connected to different machines

We focused on the only-works-with-fullscreen approach because of feedback that it was better to experiment with the feature in a more restricted environment with the initial release.

Since we may at some point want to relax the fullscreen requirement, we have taken care to keep this API separate from fullscreen.

From https://github.com/jondahlke/system-keyboard-lock/blob/master/EXPLAINER.md#sample-user-experience-flow it looks like there are no new prompts, but just a change in the messaging for how to exit fullscreen. That doesn't seem possible if the request to get esc key events happens after the fullscreen request, which happens in the sample code.

Our intent for messaging is to handle these cases as follows:

If keyboard lock enabled before fullscreen:

requestSystemKeyboardLock();

requestFullscreen();

The fullscreen message would be be updated to "Hold ESC..." before it was shown.

If keyboard lock enabled after fullscreen:

requestFullscreen();

...

requestSystemKeyboardLock();

The fullscreen message would initially be "Press ESC...", but would change to "Hold ESC..." once keyboard lock was called.

If the "Press" was still being displayed, then it would change to "Hold" and the display timer would be reset.

If the "Press" message had already been removed, then a new "Hold" message would appear (with timer).

We expect the common case to be where these 2 requests are made at roughly the same time (before any screen updates), so the user would only see the "Hold" message.

I'll be making that clear in the docs. Thank you for pointing this out.

[Elliott Sprehn]

This is very exciting! I'm curious what the plan is for windowed PWAs? They too need full keyboard access (ex. ssh), but you don't want them in full screen mode.

[Jochen Eisinger]

It's great to see that this feature is only available via secure protocols! Please make sure that you also file a regular chrome launch bug (the OWP launch bug doesn't trigger all the reviews here).

[Rick Byers]

I'm really glad we're pushing on this - thanks Gary, I know there's tough tradeoffs here that'll never satisfy everyone!  Even though most web applications would never want to use this, enabling applications like remote desktop is important for any serious platform.  I like the opt-in-by-code approach.

Given the discussion at TPAC (eg. Chaals concluding "I think it's worth playing with") and on discourse, I'd say there's at least enough non-Google interest in the problem space to justify bringing this into the WICG (to get IP protection, show that there's some involvement at W3C, etc.).  Want to propose to one of the chairs that we transfer the repo to WICG?

Rick

[Vincent Scheib]

I'll voice support for this as well! Including the long term plan of a solution that will work outside of Fullscreen.

Note that Pointer Lock also started restricted to Fullscreen as we were more conservative, then found its way to be orthogonal as well.

[Philip Jägenstedt]

Thanks Gary for elaborating on why this is a separate API for Fullscreen.

I'm happy there won't be a separate permission prompt! Still, can this API implemented if the request can fail? Even if allowed to fail async by return a promise, it wouldn't be possible to wait for keyboard lock to be granted before requesting fullscreen, as by then there will be no user gesture. How might one go about implementing an atomic "enter fullscreen with keyboard lock or do nothing" wrapper around these two APIs?

On naming, I wonder what the significance of "system" is? Fullscreen is also a device-global "resource", and is simply element.requestFullscreen(). In other words, navigator.requestKeyboardLock()?

It will be exciting to see where this goes, personally I could see it as quite useful in remoting to another browser a la BrowserStack and having the remote browser's shortcuts just work :)

[Gary Kačmarčík (Кошмарчик)]

On Wed, Jan 4, 2017 at 5:43 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

I'm happy there won't be a separate permission prompt! Still, can this API implemented if the request can fail?

 

If the request for Keyboard Lock fails, the the user agent will simply not forward any of the special system keys to the web app. Everything else would be unaffected -- the app just won't see the extra keys.

Even if allowed to fail async by return a promise, it wouldn't be possible to wait for keyboard lock to be granted before requesting fullscreen, as by then there will be no user gesture.

True. This sounds like another problem with using "user gesture" as a stand in for "user intent". :-(

How might one go about implementing an atomic "enter fullscreen with keyboard lock or do nothing" wrapper around these two APIs?

This wasn't a use case that we designed for. We focused on gracefully degrading (you don't get the keys that you requested) and we didn't want to tie this to other APIs (so we can possibly separate it later).

But, having said that...

If we changed the keylock API to:

   boolean requestKeyboardLock(...);

then we could use:

   if (requestKeyboardLock(...)) {

      requestFullscreen();

   }

But if we made keylock return a Promise:

   Promise<void> requestKeyboardLock(...);

then I don't think it's possible to have an atomic fullscreen/keylock because of (as you mention) fullscreen's user gesture requirement

If we really want atomic fullscreen (tied to user gesture) + keyboardlock (async), could we update the fullscreen API to allow something like:

   Promise<void> requestFullscreen(Promise<void> preconditions);

and then call:

   requestFullscreen(requestKeyboardLock(...));

Assuming that requestFullscreen verifies the user gesture before blocking on the promise, that seems like it would work. But it would require a change to the fullscreen API.

In any case, it seems like changing requestKeyboardLock to return a Promise seems like a reasonable thing to do (since it allows user agents to pop up a permission dialog if they want to).

On naming, I wonder what the significance of "system" is? Fullscreen is also a device-global "resource", and is simply element.requestFullscreen(). In other words, navigator.requestKeyboardLock()?

It's "system" as in the underlying operating system, as distinct from the browser. We distinguish between "browser" key shortcuts (key events sent to native apps, but handled by the browser, like Ctrl-T and Ctrl-W) and "system" key shortcuts that the browser doesn't normally get (like Alt-Tab).

But it's a subtle distinction and I think that requestKeyboardLock() is a better name.

It will be exciting to see where this goes, personally I could see it as quite useful in remoting to another browser a la BrowserStack and having the remote browser's shortcuts just work :)

Yes, if you're in fullscreen, this is what you expect to happen. E.g., if you are fullscreen-remote-connected to a Mac, you naturally expect Alt-Tab to switch apps on the remote machine. It's actually rather jarring when the local machine jumps in and switches context out from under you.

[Matt Giuca]

On Thu, 12 Jan 2017 at 12:38 'Gary Kačmarčík (Кошмарчик)' via blink-dev <blin...@chromium.org> wrote:

On Wed, Jan 4, 2017 at 5:43 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

I'm happy there won't be a separate permission prompt! Still, can this API implemented if the request can fail?

 

If the request for Keyboard Lock fails, the the user agent will simply not forward any of the special system keys to the web app. Everything else would be unaffected -- the app just won't see the extra keys.

Even if allowed to fail async by return a promise, it wouldn't be possible to wait for keyboard lock to be granted before requesting fullscreen, as by then there will be no user gesture.

True. This sounds like another problem with using "user gesture" as a stand in for "user intent". :-(

How might one go about implementing an atomic "enter fullscreen with keyboard lock or do nothing" wrapper around these two APIs?

This wasn't a use case that we designed for. We focused on gracefully degrading (you don't get the keys that you requested) and we didn't want to tie this to other APIs (so we can possibly separate it later).

But, having said that...

If we changed the keylock API to:

   boolean requestKeyboardLock(...);

then we could use:

   if (requestKeyboardLock(...)) {

      requestFullscreen();

   }

But if we made keylock return a Promise:

   Promise<void> requestKeyboardLock(...);

then I don't think it's possible to have an atomic fullscreen/keylock because of (as you mention) fullscreen's user gesture requirement

I believe the user gesture status is propagated through promises as long as they resolve within a short time. For instance, if you call setTimeout from a user gesture context, the resulting function will still be able to do things that require a user gesture as long as the timeout is small (<1000ms?), but not if the timeout is longer.

[Gary Kačmarčík (Кошмарчик)]

On Wed, Jan 11, 2017 at 5:54 PM, Matt Giuca <mgi...@chromium.org> wrote:

On Thu, 12 Jan 2017 at 12:38 'Gary Kačmarčík (Кошмарчик)' via blink-dev <blin...@chromium.org> wrote:

On Wed, Jan 4, 2017 at 5:43 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

How might one go about implementing an atomic "enter fullscreen with keyboard lock or do nothing" wrapper around these two APIs?

[...]

But if we made keylock return a Promise:

   Promise<void> requestKeyboardLock(...);

then I don't think it's possible to have an atomic fullscreen/keylock because of (as you mention) fullscreen's user gesture requirement

I believe the user gesture status is propagated through promises as long as they resolve within a short time. For instance, if you call setTimeout from a user gesture context, the resulting function will still be able to do things that require a user gesture as long as the timeout is small (<1000ms?), but not if the timeout is longer.

But that would preclude doing any UI (e.g., show a dialog for permission) during the request.

[Matt Giuca]

I'm not sure; it might be the case that the user's response to the dialog would mean the resulting callback / promise resolution is also in a user gesture context? 

[Rick Byers]

I doubt it.  IIRC this "flowing of user gestures" is a pretty Chrome-specific thing right now and is applied inconsistently (eg. I know it does flow along setTimeout but I'm not sure it does along promises).  There's some discussion with the other vendors around trying to simplify our implementation to something more predictable and standardizable.

[Jeffrey Yasskin]

On Wed, Jan 11, 2017 at 5:54 PM, Matt Giuca <mgi...@chromium.org> wrote:

On Thu, 12 Jan 2017 at 12:38 'Gary Kačmarčík (Кошмарчик)' via blink-dev <blin...@chromium.org> wrote:

On Wed, Jan 4, 2017 at 5:43 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

I'm happy there won't be a separate permission prompt! Still, can this API implemented if the request can fail?

 

If the request for Keyboard Lock fails, the the user agent will simply not forward any of the special system keys to the web app. Everything else would be unaffected -- the app just won't see the extra keys.

Even if allowed to fail async by return a promise, it wouldn't be possible to wait for keyboard lock to be granted before requesting fullscreen, as by then there will be no user gesture.

True. This sounds like another problem with using "user gesture" as a stand in for "user intent". :-(

How might one go about implementing an atomic "enter fullscreen with keyboard lock or do nothing" wrapper around these two APIs?

This wasn't a use case that we designed for. We focused on gracefully degrading (you don't get the keys that you requested) and we didn't want to tie this to other APIs (so we can possibly separate it later).

But, having said that...

If we changed the keylock API to:

   boolean requestKeyboardLock(...);

then we could use:

   if (requestKeyboardLock(...)) {

      requestFullscreen();

   }

But if we made keylock return a Promise:

   Promise<void> requestKeyboardLock(...);

then I don't think it's possible to have an atomic fullscreen/keylock because of (as you mention) fullscreen's user gesture requirement

I believe the user gesture status is propagated through promises as long as they resolve within a short time. For instance, if you call setTimeout from a user gesture context, the resulting function will still be able to do things that require a user gesture as long as the timeout is small (<1000ms?), but not if the timeout is longer.

We have a bug open against HTML to figure out how gestures should flow through promises: https://github.com/whatwg/html/issues/1903.