Intent to ship: Android On-Screen Keyboard doesn't resize layout

[David Bokan]

Contact emails

bo...@chromium.org

Spec

No official spec but I have a proposal doc that has more detail.

Summary

This isn't really a new feature, it's more of a change to improve interop with Safari and improve user experience. This follows up on the similar change to the URL bar.

Today, the initial containing block and layout viewport is resized each time the keyboard is shown/hidden. This means the page can change in various ways if the keyboard comes up which can be surprising to users, can do lots of work (increasing latency), and makes our "scroll-input-into-view" behavior difficult to get right in edge-cases (see bugs blocked on crbug.com/404315 for examples). Additionally, bottom position: fixed elements can overlay the input box that the user is typing into. By changing the keyboard to resize only the visual viewport - page layout will be unaffected.

I would also like to stop firing the `resize` event when the keyboard shows up. This may be more controversial as it means authors have no way to tell when the keyboard comes up but this is already the case on Safari (and ChromeOS which this proposal unifies us with). More thorough discussion is in the doc but basically, I'd like to stop overloading the `resize` event and reserve it for changes to the ICB/layout and work with Safari+others to standardise a new event/media query for changes to the visual viewport. I've also added a metric in M58 to measure the amount of time spent inside resize event handlers - results from Canary (grain-of-salt, not many users yet) show the 99th percentile on Android to be 74ms (and 200ms on low-memory devices). My guess is that most resize event handlers probably don't care about keyboard so this is wasted work/performance.

Link to “Intent to Implement” blink-dev discussion

I believe this idea predates the intent process so there was no intent to implement but there has been discussion of this before, notably in this doc (sorry, Googler only link).

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

This proposal affects only Android but unifies behavior with Chrome OS

Demo link

You can try it out by enabling chrome://flags/#enable-osk-overscroll. I've put together http://bokand.github.io/keyboardtest.html to see the differences.

Interoperability and Compatibility Risk

Keyboard resize behavior is completely unspec'd and browsers currently exhibit quite different behavior. This change aligns us more closely with Safari and Edge so interop improves slightly here (though Safari doesn't change the window.innerHeight value and Edge does fire a resize event). See the proposal doc for a behavior matrix.

Compatibility wise, there's some non-trivial risk. A page that tries to keep something fixed to the kayboard (e.g. a formatting bar) will no longer stick the bar to the keyboard (the user can scroll down to see it though). If the page tries to react to the keyboard coming in (e.g. only appearing the bar on keyboard show) it'll no longer know about it because of the lack of resize event. I expect these kinds of patterns to be exceedingly rare as there's no solution for them today in Safari.

Signals from other vendors:

Apple - None but they have a similar model except they don't change window.innerHeight

Firefox - None

Edge/IE - None but they have a similar model except they fire a resize

WebDevs - Unknown but the bug has 44 stars (unclear how much of that is web-dev vs Chrome dev). Positive signals from G+/AMP team in this doc (Google-only).

OWP launch tracking bug

Not OWP but http://crbug.cdlers - results from Canary (grain-of-salt, not many users yet) show the 99th percentile on Android to be 74ms (and 200ms on low-memory devices). My guess is that most resize event handlers probably don't care about keyboard so this is wasted work/performance.om/404315

Entry on the feature dashboard

https://www.chromestatus.com/feature/5656077370654720

[PhistucK]

Just a quick note - I think Safari does let you know about the keyboard showing up, but you have to do something. If I remember - adding an empty element to the page every second or so makes Safari fire resize events (for the top bar moving out on scroll and for the keyboard).

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Dimitri Glazkov]

LGTM1

[Chris Harrelson]

LGTM2

--

[Rick Byers]

IIRC, last we talked about this (with ymalik@) the plan was to ship the ViewportAPI first so that sites making something stick above the keyboard in Chrome (eg. G+ IIRC) had a replacement.

Thoughts?

[David Bokan]

My thinking on this has changed. I'm not confident we could ship the viewport API responsibly in the short term with the uncertainty over what Safari is going to do with their visual viewport implementation. If they stick with their current implementation (which I think is likely), the viewport API is mostly unneeded since visual viewport information is contained in `window.innerWidth|Height and window.scrollX|Y`. The only thing that would really be missing is a resize event for the visual viewport.

Given that the OSK change has immediate benefits and improves interop, I think shipping this now is preferable. This does break the "stick things above the keyboard" use case but I haven't seen any examples of that (the G+ page no longer does it) and for existing content - the breakage should be small, users can still scroll down to see a position: fixed element. The fact that you can't do it on Safari (I haven't tried PhistucK's solution yet, but even if it works...eww) means web developers wouldn't get much from this even if we allowed it. IMO, it's better to align with Safari (and Edge) now and work with them to make this use case viable.

[Rick Byers]

Sorry for the delay (was on vacation - hope other owners weren't blocking my response!  Thanks for the explanation.

However I'm still seeing the difference on G+.  See screenshots of composing a post below with the flag off then on (and then if you scroll the behavior is pretty weird - size snaps smaller and exposes background).  To me this feels like a substantially better user experience with the flag off.  Though Safari has a similar poor user experience, so maybe there's an argument that the cost is worth the benefit in terms of better interop?

Maybe this is just G+ trying to be really smart and not something that's being done elsewhere?  Perhaps it's worth reaching out to them to see if they can do something that feels better than the flag is on?  As always, the experience on one particular popular site isn't a big deal except in that it might highlight a problem that many other developers will struggle with too.

 
​
​

[Elliott Sprehn]

On Fri, Mar 10, 2017 at 11:13 AM, 'David Bokan' via blink-dev <blin...@chromium.org> wrote:

My thinking on this has changed. I'm not confident we could ship the viewport API responsibly in the short term with the uncertainty over what Safari is going to do with their visual viewport implementation. If they stick with their current implementation (which I think is likely), the viewport API is mostly unneeded since visual viewport information is contained in `window.innerWidth|Height and window.scrollX|Y`. The only thing that would really be missing is a resize event for the visual viewport.

Given that the OSK change has immediate benefits and improves interop, I think shipping this now is preferable. This does break the "stick things above the keyboard" use case but I haven't seen any examples of that (the G+ page no longer does it) and for existing content - the breakage should be small, users can still scroll down to see a position: fixed element. The fact that you can't do it on Safari (I haven't tried PhistucK's solution yet, but even if it works...eww) means web developers wouldn't get much from this even if we allowed it. IMO, it's better to align with Safari (and Edge) now and work with them to make this use case viable.

This doesn't sound like a good compromise to me, you're basically saying if you want to build a competitive PWA chat app on android you can't do it on the web, since you can't get a toolbar to be above the keyboard which every modern chat app does (Hangouts, Facebook, etc.)

It's unfortunate that Safari has done something else, but they also don't have a thriving PWA ecosystem with saved to home screen apps that feel native like we do.

It's very important we remain competitive from a UX perspective, and having a toolbar above the keyboard is extremely common UX these days.

[David Bokan]

On Wednesday, March 22, 2017 at 12:06:10 AM UTC-4, Elliott Sprehn wrote:

On Fri, Mar 10, 2017 at 11:13 AM, 'David Bokan' via blink-dev <blin...@chromium.org> wrote:

My thinking on this has changed. I'm not confident we could ship the viewport API responsibly in the short term with the uncertainty over what Safari is going to do with their visual viewport implementation. If they stick with their current implementation (which I think is likely), the viewport API is mostly unneeded since visual viewport information is contained in `window.innerWidth|Height and window.scrollX|Y`. The only thing that would really be missing is a resize event for the visual viewport.

Given that the OSK change has immediate benefits and improves interop, I think shipping this now is preferable. This does break the "stick things above the keyboard" use case but I haven't seen any examples of that (the G+ page no longer does it) and for existing content - the breakage should be small, users can still scroll down to see a position: fixed element. The fact that you can't do it on Safari (I haven't tried PhistucK's solution yet, but even if it works...eww) means web developers wouldn't get much from this even if we allowed it. IMO, it's better to align with Safari (and Edge) now and work with them to make this use case viable.

This doesn't sound like a good compromise to me, you're basically saying if you want to build a competitive PWA chat app on android you can't do it on the web, since you can't get a toolbar to be above the keyboard which every modern chat app does (Hangouts, Facebook, etc.)

It's unfortunate that Safari has done something else, but they also don't have a thriving PWA ecosystem with saved to home screen apps that feel native like we do.

It's very important we remain competitive from a UX perspective, and having a toolbar above the keyboard is extremely common UX these days.

 

Ok, I think I wasn't properly appreciating the PWA use-case. In that case, I think keeping the resize event should be enough to let authors keep a toolbar over the keyboard but there's some other scroll effects I'm not so sure about so I'll postpone the intent for now and do some more research here. Thanks all.

[Reilly Grant]

Have we considered making the on-screen keyboard position available to CSS so that a toolbar can be positioned above the keyboard without changing the layout of the rest of the page?

[David Bokan]

The keyboard will change window.innerHeight, so if we keep firing the resize event, an author could just listen for the resize and add a "position: fixed" toolbar and offset it from the bottom of the page by the difference.

Unfortunately, there's no surefire way to know that the resize is from a keyboard but that's a deeper problem. Android also doesn't have this API which means we use a similar resize heuristic to determine when the OSK is showing. Adding keyboard height (or even presence) in CSS might be over-promising what we can deliver.

[Takayoshi Kochi]

Just to add another viewpoint...

When in the past I was working on W3C IME API (https://www.w3.org/TR/ime-api/ - which once was partially implemented

in Blink but was removed - and the spec was frozen) one use case was to notify web app about IME candidate window

position and size (which usually overlays below caret).  For example, to avoid overlap of Google Suggest and candidate

window on Google Search page.

The API were designed with mostly destop IMEs in mind and for mobile IMEs cases we waited for their UI pattern

to get more mature.

It might be useful to have a general API to detect any occlusion by any non-browser overlay window or virtual keyboard

like intersection observer, though it is my casual thought and I can be wrong.

--

Takayoshi Kochi

[Adam Bradley]

With the Ionic framework, the biggest thing that'd help us out is a standard behavior across all platforms/browsers, which looks like this is what it's addressing. 

Where I'm lost is if the group is in favor, or against firing the resize event. Currently, what's important for us is getting that event so that we can adjust the layout and scroll inputs into place, and to take window measurements a few frames after the resize event fired. Where it becomes a big challenge is trying to figure out if the resize was from an orientation change, the window actually resized, or the keyboard appeared or disappeared. Right now we've got a lot of code around that area to magically figure out what type of event it actually was so the app can adjust accordingly.

Regarding "scrolling inputs into place", we do this mainly for Safari since it'll move the header out of the view when an input is tapped, which isn't what a native app would do: https://adamdbradley.github.io/mobile-ux-issues/auto-scroll.html 

I can see the case for both firing the resize event or not, but either way, in the end what would be most helpful is to have an explicit way of knowing if keyboard was shown or was hidden, however that may happen. This issue looked like it might be addressing that. https://bugs.chromium.org/p/chromium/issues/detail?id=634333#c2 

[Elliott Sprehn]

LinkedIn's chat interface also has a sticky toolbar like this. I don't know what they do in Safari, but it looks good on Android.

[PhistucK]

Safari changed their behavior somewhat in 10.1 (iOS 10.3). Are you aware of those changes? Is Chrome aligned?

Highlighted in their blog post -

Improved Fixed and Sticky Element Positioning

When using pinch-to-zoom, fixed and sticky element positioning has improved behavior using a “visual viewports” approach. Using the visual viewports model, focusing an input field that triggers the on-screen keyboard no longer disables fixed and sticky positioning in Safari on iOS.

☆PhistucK

--

[David Bokan]

Yes, I'm their coordinate system works somewhat differently from Chrome and Edge which I'm trying to reconcile. I'm just installing iOS 10.3 now to look into the details.

[David Bokan]

I think the agreement is that the app needs some way to be able to respond to the keyboard. In that case, I think keeping the resize event will work and you could adjust your layout using window.innerWidth/Height. I'm still trying to determine if that's enough though. The issue I still have is that it might be difficult to prevent scrolling when the keyboard comes up - there needs to be a way to do that. This proposal means we shrink the visual viewport and I don't think there's a way to prevent scroll on the visual viewport. 

The problem with adding an explicit event for keyboard (or even better, a media query) is that Android APIs doesn't give us that information either so Chrome is basically relying on what web pages do today - heuristics. I'm not sure how reliable that signal would be.

--
You received this message because you are subscribed to a topic in the Google Groups "blink-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/blink-dev/Tr43oT4DQoY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to blink-dev+unsubscribe@chromium.org.

[David Bokan]

Update: This is still on the input-dev roadmap and we do still want to do it but it's not a priority right now. I'm not sure when I'll be able to get to it so we can consider this intent shelved for the near future.

[master...@gmail.com]

Any news on this?