Intent to Implement and Ship: Browser Shortcuts in Fullscreen

98 views
Skip to first unread message

Gary Kacmarcik (Кошмарчик)

unread,
Jan 17, 2017, 7:32:39 PM1/17/17
to Chromium-dev, blink-dev
(resending because previous email was rejected by chromium-dev)

Contact emails

Eng: gar...@chromium.org, zij...@chromium.org, jamie...@chromium.org, domi...@chromium.org, mgi...@chromium.org


Spec

No spec changes.

Fullscreen API spec (see relevant discussion in issue #39)


Summary

Chrome currently has a number of "browser" shortcut keys that are not sent to the web app. For example, the following (related to managing tabs and windows):

  Ctrl/Cmd + (T | W | N)

  Ctrl/Cmd + Shift + (T | W | N)

We propose that all keys normally reserved by the browser (except Escape and/or F11, as appropriate) should be sent to the app when the browser is in fullscreen mode.


In addition, the mouse forward and back buttons (which are used to navigate forward/backward in tab history), should also be sent to the web app when in fullscreen. This is to make the behavior consistent with keyboard forward/back buttons (found on some multimedia keyboards).


Motivation

Some web apps, like remote access apps and games, need access to keys that are taken by the Chrome UI:

  • Ctrl-T, W and N are all common emacs (text editor) commands.

  • In a remote access scenario, the user may wish to use these shortcuts with the remote app.

  • In games, W is a common movement key and modifiers (like Shift or Control) can be used to modify the movement in that direction.

Currently, Chrome apps can address this problem by setting "open as window" which gives access to these browser shortcuts. However, the recent deprecation announcement means that we need another mechanism for making these shortcuts available.


Developer interest: Developers often request the ability to capture browser shortcuts universally. However, permitting this means that sites could disable important user functionality. Our fullscreen-only proposal is a compromise: in fullscreen, it is clearly signalled that pressing Esc will exit, and the fullscreened site is clearly the entire focus of the user's attention..

Related bugs: 25756, 33056, 614145, 638585, 648469, 677680


Note 1: These keys are currently filtered from apps in Chrome for UX/perf reasons. For example, pressing Ctrl-T multiple times to switch tabs can be unbearably slow if each tab needs to be loaded to see if it wants to override the key combo. See 5496. Firefox made their recent behavior change (see below) for similar reasons.


Note 2: How this Intent is related to the recent Intent to Implement for "System Keyboard Lock". The System Keyboard Lock proposal focuses on keys (and shortcuts) that the browser does not normally handle (like Alt-Tab), whereas this proposal focuses on shortcuts that the browser does normally handle (like Ctrl-T).


With this Intent, we are proposing:

Which keys are sent to web app?

In Fullscreen

Not in Fullscreen

By default

Via SysKeyLock

System Keys (e.g., Alt-Tab)

no

yes

no

Browser Keys (e.g., Ctrl-T)

yes

Not needed

no

Esc

no

yes

yes

If we do not go ahead with this Intent, then the System Keyboard Lock API would be needed for Browser keys as well, and sites that required browser keys would need to make use of the System Keyboard Lock API.


Interoperability and Compatibility Risk

Medium, although note that we're not trying to get agreement with this Intent since this is a browser UX issue.

Edge: All these keys can be captured in or out of fullscreen

Firefox: Previously, these keys were sent, but in recent builds they are no longer sent due to lock-up issues in the multiprocess architecture. See 1074971 for change. Also see 380637 for discussion, but skip down to comment #185 if you value your sanity.

Safari: Blocks an even larger set of Cmd-* key combinations from being sent.


Ongoing technical constraints

None.


Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Windows, Mac, Linux and ChromeOS.

As currently defined, this feature is not applicable on mobile platforms.


Tracking bug

https://crbug.com/680737 (for browser keys in fullscreen)

https://crbug.com/680741 (for mouse forward/backward in fullscreen)


Requesting approval to ship?

Yes.



Alexandre Elias

unread,
Jan 17, 2017, 8:37:59 PM1/17/17
to Gary Kacmarcik (Кошмарчик), Chromium-dev, blink-dev
On Tue, Jan 17, 2017 at 4:31 PM, Gary Kacmarcik (Кошмарчик) <gar...@chromium.org> wrote:

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Windows, Mac, Linux and ChromeOS.

As currently defined, this feature is not applicable on mobile platforms.


It seems applicable to Android.  Chrome for Android supports Fullscreen API, and it's possible to connect a physical keyboard via Bluetooth, USB host or proprietary attachment (e.g. Pixel C), and a subset of shortcuts like Ctrl-T is indeed handled.  Could you add that platform to the plan?

I think it's not applicable to Android WebView because WebView does not have browser shortcuts (although it does support fullscreen, so the SysKeyLock proposal may apply to it if only for Escape key handling).

Gary Kacmarcik (Кошмарчик)

unread,
Jan 17, 2017, 9:25:00 PM1/17/17
to Alexandre Elias, Chromium-dev, blink-dev
On Tue, Jan 17, 2017 at 5:36 PM, Alexandre Elias <ael...@chromium.org> wrote:
On Tue, Jan 17, 2017 at 4:31 PM, Gary Kacmarcik (Кошмарчик) <gar...@chromium.org> wrote:

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Windows, Mac, Linux and ChromeOS.

As currently defined, this feature is not applicable on mobile platforms.


It seems applicable to Android.  Chrome for Android supports Fullscreen API, and it's possible to connect a physical keyboard via Bluetooth, USB host or proprietary attachment (e.g. Pixel C), and a subset of shortcuts like Ctrl-T is indeed handled.  Could you add that platform to the plan?

I've updated the tracking bugs for this (and System Keyboard Lock) to indicate that we need to support this on Android with external keyboard.  I'll update the spec as well.


Gary Kacmarcik (Кошмарчик)

unread,
Jan 23, 2017, 7:22:48 PM1/23/17
to Alexandre Elias, Chromium-dev, blink-dev
Ping for further comments.

Note that related bug 119881 ("API to request key events normally reserved for browser shortcuts") is one of Chome's top-50 starred bugs.

Philip Jägenstedt

unread,
Jan 24, 2017, 6:40:24 AM1/24/17
to Gary Kacmarcik (Кошмарчик), Alexandre Elias, Chromium-dev, blink-dev
I think a change like this makes sense, but I'm not clear on the details. Will they key events be sent and have no effect, or do events need to be canceled in order to prevent a new tab from being opened and so on?

Given that this is currently not interoperable and has no spec I don't think it needs to block on any spec discussion, but it would be good to figure out in https://github.com/whatwg/fullscreen/issues/39 if there's anything we can do to get closer to interop here.

Rick Byers

unread,
Jan 24, 2017, 3:30:07 PM1/24/17
to Philip Jägenstedt, Gary Kacmarcik (Кошмарчик), Alexandre Elias, Chromium-dev, blink-dev
Given:
  • It's currently possible to use browser shortcuts inside of fullscreen (eg. Ctrl-T opens a new tab and exits fullscreen) and we presumably don't want to break that when the app has no interest in the key
  • We already have many (most?) shortcut keys which ARE sent as events to the page first (eg. try Ctrl-P, or Ctrl-Shift-I here with the 'preventDefault on keyup' option) 
I'm assuming the design is just that, while in fullscreen, the "exclusive shortcut keys" (is there a term for this) like Ctrl-T just change behavior to trigger as the default action (like other non-exclusive shortcut keys such as Ctrl-P).

Assuming that's indeed the design, LGTM1 from me.  The compat risk seems low, not something we currently try to standardize, and clearly moves the platform forward in being more powerful.

Scott Violet

unread,
Jan 24, 2017, 5:23:56 PM1/24/17
to gar...@chromium.org, Chromium-dev, blink-dev
Rick,

Do you have any metrics as the frequency with which these shortcuts are used while fullscreen? It seems with your proposal they would some times work, and some times the page would process them. That seems confusing, especially control-w, which a malicious type app could then eat and make it harder for you to get out of fullscreen. Have you considered having the page ask and the user grant this capability?

  -Scott

On Tue, Jan 17, 2017 at 4:31 PM, Gary Kacmarcik (Кошмарчик) <gar...@chromium.org> wrote:

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

Dominick Ng

unread,
Jan 24, 2017, 11:39:38 PM1/24/17
to Scott Violet, gar...@chromium.org, Chromium-dev, blink-dev
Rick: your description matches how we would implement it. Browser shortcuts would not be filtered out (as they currently are) before sending the key events to the renderer while in fullscreen. If the site chooses to not preventDefault() them, the events will be sent back to the browser and will trigger the usual actions.

Scott: We definitely considered a permission whilst designing this functionality. But from a security UX perspective:
  • it is difficult to craft an understandable permission request for keyboard lock
  • keyboard lock doesn't send private data, nor is it particularly powerful on its own (unlike other APIs which are guarded by permissions)
  • fullscreen has extremely clear messaging that Esc/F11 is the key used to exit
  • other browsers (currently Edge, and up to recently Firefox) allow these shortcuts to be intercepted in and out of fullscreen
The risk that a malicious site captures Ctrl-W to make it harder to escape from fullscreen doesn't really justify the API friction and demand for a user decision that comes with adding a permission. :)

Chris Harrelson

unread,
Jan 24, 2017, 11:47:35 PM1/24/17
to Dominick Ng, Scott Violet, gar...@chromium.org, Chromium-dev, blink-dev
LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Philip Jägenstedt

unread,
Jan 25, 2017, 2:59:33 AM1/25/17
to Chris Harrelson, Dominick Ng, Scott Violet, gar...@chromium.org, Chromium-dev, blink-dev
LGTM3. I just tested Firefox and Ctrl-T in fullscreen exits fullscreen and opens a new tab, which is also what Chrome does. Making it possible to use those for in-app things makes sense from an API perspective. However, if any security folks have input on this, that would be interesting.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Mike West

unread,
Jan 25, 2017, 3:35:30 AM1/25/17
to Philip Jägenstedt, Emily Schechter, Chris Harrelson, Dominick Ng, Scott Violet, garykac, Chromium-dev, blink-dev
On Wed, Jan 25, 2017 at 8:57 AM, Philip Jägenstedt <foo...@chromium.org> wrote:
LGTM3. I just tested Firefox and Ctrl-T in fullscreen exits fullscreen and opens a new tab, which is also what Chrome does. Making it possible to use those for in-app things makes sense from an API perspective. However, if any security folks have input on this, that would be interesting.

This seems reasonable to me, but let's loop in the security UX folks: Emily, would y'all mind having an opinion on this? :)

-mike
 

Scott Violet

unread,
Jan 25, 2017, 11:18:36 AM1/25/17
to Dominick Ng, gar...@chromium.org, Chromium-dev, blink-dev
Thanks for the details Dominick. I am curious about usage of these keys when fullscreen. Did you try to tracking that?

  -Scott

On Tue, Jan 24, 2017 at 8:38 PM, Dominick Ng <domi...@chromium.org> wrote:

Rick Byers

unread,
Jan 25, 2017, 11:28:52 PM1/25/17
to Scott Violet, Dominick Ng, Gary Kacmarcik (Кошмарчик), Chromium-dev, blink-dev
Scott, I think your concern is an instance of a larger design debate we have off and on: is it better to ensure consistency of user experience or to enable control over the user experience by the application (assuming either is OK from a security perspective).

For example, we have a number of built-in touch gestures, but we ALWAYS give the page first crack at touches that occur over the content area.  This is in stark contrast to Mobile Safari which will "steal" input that the page can't override (much to web developer frustration when it's not what their users expect - eg. Google Photos carousel side-swiping on iPhone).   Or even more simply the same argument could be made for the mouse wheel - USUALLY it scrolls, but other times it does something else like zooms a map on the page.  Occasionally users are frustrated by this inconsistency ("why isn't scrolling working here"), but often the power is used to do what actually feels natural to the user.

Personally I feel our "content not chrome" design philosophy (Chrome is the window manager designed to get out of the way of the application) implies that we should strive to err on the side of giving the content control (for input/focus on the content area), keeping the temptation to "steal" input from the content to the bare minimum necessary for security.  But this is obviously open for debate.

Scott Violet

unread,
Jan 26, 2017, 1:21:02 PM1/26/17
to Rick Byers, Dominick Ng, Gary Kacmarcik (Кошмарчик), Chromium-dev, blink-dev
Thanks for the thoughtful response Rick. I understand your motivation for the change and was mostly just curious to know how many people might be impacted by it.

In my opinion I don't know that there is necessarily a right answer to what should happen with accelerators when fullscreen. For power users familiar with the app then giving the page first crack is definitely expected. On the other hand casual users are more likely to be familiar with and expect accelerators to be processed by chrome first. This is why I suggested offering users the ability to opt-in to this functionality. Again, this is my just opinion, it would be good to involve UX in this decision.

  -Scott

Rick Byers

unread,
Jan 26, 2017, 1:59:57 PM1/26/17
to Scott Violet, Dominick Ng, Gary Kacmarcik (Кошмарчик), Chromium-dev, blink-dev
On Thu, Jan 26, 2017 at 1:19 PM, Scott Violet <s...@chromium.org> wrote:
Thanks for the thoughtful response Rick. I understand your motivation for the change and was mostly just curious to know how many people might be impacted by it.

Thanks Scott.  Note that this isn't "my motivation" - I (and AFAIK the input-dev team generally) haven't really been involved in this feature at all (otherwise I would have recused myself from the approval process)  :-).

In my opinion I don't know that there is necessarily a right answer to what should happen with accelerators when fullscreen. For power users familiar with the app then giving the page first crack is definitely expected. On the other hand casual users are more likely to be familiar with and expect accelerators to be processed by chrome first. This is why I suggested offering users the ability to opt-in to this functionality. Again, this is my just opinion, it would be good to involve UX in this decision.

Yeah whenever there are substantial UX concerns, features should generally go through the internal go/newChromeFeature process (in addition to this web platform process here).  Gary et. al., let us know if there's anything you need from me or other web-platform folks there.

Gary Kacmarcik (Кошмарчик)

unread,
Jan 26, 2017, 9:12:46 PM1/26/17
to Rick Byers, Scott Violet, Dominick Ng, Chromium-dev, blink-dev
On Thu, Jan 26, 2017 at 1:19 PM, Scott Violet <s...@chromium.org> wrote:
In my opinion I don't know that there is necessarily a right answer to what should happen with accelerators when fullscreen. For power users familiar with the app then giving the page first crack is definitely expected. On the other hand casual users are more likely to be familiar with and expect accelerators to be processed by chrome first. This is why I suggested offering users the ability to opt-in to this functionality. Again, this is my just opinion, it would be good to involve UX in this decision.

Actually, with what we've seen so far (admittedly, anecdotal), when the user is in fullscreen mode there is no Chrome chrome so users forget about the browser and *don't* expect browser shortcuts.

It's more extreme with remote access apps: It is incredibly jarring when you're interacting with a remote app and the local browser jumps in and switches you away from your work. When connected to a remote machine in fullscreen, you "forget" about the local machine and become immersed in the remote machine.

So, this isn't a power user vs. casual user issue.

On Thu, Jan 26, 2017 at 10:58 AM, Rick Byers <rby...@chromium.org> wrote:
Yeah whenever there are substantial UX concerns, features should generally go through the internal go/newChromeFeature process (in addition to this web platform process here).  Gary et. al., let us know if there's anything you need from me or other web-platform folks there.

Reply all
Reply to author
Forward
0 new messages