Re: Touch-Action + Chrome 36 = unpredictable scrolling

[Rick Byers]

+input-dev (I'm on vacation so others there can also weigh in)

[Note: Dan filed http://crbug.com/399765 to track this, I'll follow up there when we reach consensus on this thread]

On Thu, Jul 31, 2014 at 8:36 PM, Daniel Freedman <dfr...@google.com> wrote:

Hi Rick,

I'm having a very frustrating time with the new touch events behavior on 36.

Sorry for the trouble!  I definitely think the interaction of touch-action: pan-* and our new async touchmove event model needs some more thought.  Eg. see http://crbug.com/391876 for a related issue. 

In particular, touch-action setting touchmove.cancelable seems to not honor the initial direction of scrolling, but will toggle when hitting a scrolling region's boundaries or after a movement has been made in the other direction.

Right, this is a key property of the throttled async touchmove event model (ignoring touch-action entirely).  Whenever the user moves their finger in a direction which cannot scroll, we send unthrottled cancelable touchmove events so that JS can take over and implement their own effect - like pull-to-refresh.  We think of it this way: ideally all touchmove events would be unthrottled and cancellable - the only reason to take that capability away from JavaScript is to reduce the risk of scroll jank during an active scroll.

You can see this on http://www.rbyers.net/touch-action.html.

On the "pan-x" box, drag right, then up and down.

I would assume that up and down would always be throttled to 200ms and be cancelable, and the current behavior is very confusing.

throttled AND cancelable?  Being cancelable implies JS is in control of the event stream, why would you also want/expect the events to be throttled in this case?  We have only two states: 

 1) active scroll: touchmove events are throttled and uncancelable (async)

 2) JS control: touchmove events are unthrottled and cancelable (sync)

The way it's designed (which, like I said, may be something we should change) is that the touch event model and touch-action are almost completely disjoint concepts:

The touch event model says that when we see scrolling occurring we switch to the 'fast' (throttled async) touchmove mode, and when scrolling stops we go back to the full power mode.

Then touch-action acts a filter on the types of scrolls that are created in the first place.  So in a pan-x region any vertical movement does not get included in the scroll gesture.  So if all movement is vertical the touch event model can stay in the "full power" mode (since no scrolling is occurring).

Does this make sense?  Perhaps you can describe what you're trying to achieve or how the model is causing trouble for you and we can brainstorm how things should possibly change.

Rick

  

[Daniel Freedman]

I can see how the current behavior allows for things like customizable over scroll, but is I'd quite a departure from the PointerEvents touch-action.

In my use case, I have a slide-out panel that has a long list of selectable options. It needs to scroll in the y axis, but slide to close in the x axis. However, it is bad UX to allow both operations to happen at once. In the PointerEvents version of touch action, it was guaranteed that only one behavior, slide or scroll, would occur.

It seems like the only way to get the behavior I need would be to make the touchmove handlers more stateful, and ignore touchmove after a single throttled event.

My worry is that this will be super confusing for other devs, because it differs so much from the PointerEvents behavior, but shares the same name. Perhaps some evangelism would help in this regard.

[Rick Byers]

On Fri, Aug 1, 2014 at 9:17 PM, Daniel Freedman <dfr...@chromium.org> wrote:

I can see how the current behavior allows for things like customizable over scroll, but is I'd quite a departure from the PointerEvents touch-action.

I'd argue it's really the touch event model we're using that's a departure from pointer events (where the touchcancel-on-scroll model we used to use was very similar to pointer events).  I think similar confusion can arise without touch-action at all.  For example, when designing for pointer events (or the touchcancel model) it would be natural to design a button which gets despressed touchstart/pointerdown, activates on touchend/pointerup and lifts without activation on touchcancel/pointercancel.  It's effectively relying on events and scrolling being mutually exclusive by expecting events to be cancelled when scrolling starts.  Luckily since Safari has never used this touchcancel-on-scroll model, we didn't actually come across any code that relied on this behavior in Chrome.

So switching from a model where scrolling and event handling are mutually exclusive to one where they're not definitely adds some complexity (but enables additional scenarios possible in native mobile platforms which we feel are essential).

I can see why this confusion often comes up in the context of touch-action: pan-y, but I don't think the essential problem has anything to do with touch-action.  It's inherent in the additional power provided by lifting the scrolling/eventing mutual exclusion restriction.

In my use case, I have a slide-out panel that has a long list of selectable options. It needs to scroll in the y axis, but slide to close in the x axis. However, it is bad UX to allow both operations to happen at once. In the PointerEvents version of touch action, it was guaranteed that only one behavior, slide or scroll, would occur.

Yeah this is a very common case (Matt Gaunt raised similar concerns to me a little while ago - saying that touch-action in M35 seemed more useful to him for, if I recall, precisely this reason).  We should definitely do something to make this easier to understand.  I know both Matt and Jake have worked on such cases.  Perhaps we should publish some sample / library code for making this easy?

Or perhaps this is so common that we should introduce a coupling between touch-action and the touch event model for this.  I think we could have touch-action: pan-y cause the touch event handling code to enter a special mode where touchcancel is sent when movement starts vertically.  I.e. we'd define touch-action to not only act as a gesture filter (as it does in PE), but also introduce a touch event stream disruption too.  It's not clear to me whether we'd also want such a disruption for other touch-action modes like 'pan-x pan-y' and 'manipulation' (probably not).

I didn't initially like this idea because it introduces a coupling between things that are architecturally separate (and to me makes our touch event model more complex/confusing), but if it makes developers lives easier without any loss of real power then maybe it's worth it?  Tim/Jared WDYT?  Off the top of my head I don't think this would be hard to implement - just more coupling between the TouchActionGestureFilter and the TouchEventQueue/InputRouter.

One potential disadvantage to this is that more advanced scenarios (like p2r) wouldn't be able to use touch-action: pan-y, but perhaps that's OK.  +Jorik in case he's got input from his work on hammer.js.

It seems like the only way to get the behavior I need would be to make the touchmove handlers more stateful, and ignore touchmove after a single throttled event.

My worry is that this will be super confusing for other devs, because it differs so much from the PointerEvents behavior, but shares the same name. Perhaps some evangelism would help in this regard.

Matt/Jake: what do you think?  Is this something we can explain with evangelism and a little sample code, or do you think we're better off by extending touch-action to make these scenarios simpler to code?

I'm about to enter Yellowstone park where apparently neither my T-Mobile nor AT&T SIM can get data, so I may not be able to respond until I'm back in the office on Aug 13th.  If you guys can reach consensus on this without me, then we can go ahead and try some changes - perhaps my purist view on this has delayed the right pragmatic solution too long anyway ;-)

Rick

[Jared Duke]

I didn't initially like this idea because it introduces a coupling between things that are architecturally separate (and to me makes our touch event model more complex/confusing), but if it makes developers lives easier without any loss of real power then maybe it's worth it?  Tim/Jared WDYT?  Off the top of my head I don't think this would be hard to implement - just more coupling between the TouchActionGestureFilter and the TouchEventQueue/InputRouter.

I'm not so much worried about implementation complexity as I am conceptual complexity.  Coupling touch-action state with touch event dispatch does sound convenient, but it's yet another branch in our already branchy model. I'd like to hear from more web developers that actually use these API's for a living. As it stands, I've seen far too many developers fail to understand something as basic as touch-action: none to trust my own views on what is simple and useful. =/

[Timothy Dresser]

Or perhaps this is so common that we should introduce a coupling between touch-action and the touch event model for this.  I think we could have touch-action: pan-y cause the touch event handling code to enter a special mode where touchcancel is sent when movement starts vertically.

As many developers never figured out our previous, much simpler touchcancel behavior, I'm worried that this more complicated touchcancel behavior would create more trouble than its worth.

We should keep in mind that there are cases where web developers want to use native scrolling on one axis, and javascript scrolling on the other.

[Matt Gaunt]

I personally don't understand the real advantage of touch-action as it stands in todays Chrome.

It offers me little since I have to handle the behaviour I want in javascript regardless of the touch-action being set, the very first implementation (which had the touchcancel behaviour) was useful since the browser actually took some responsibility and simplified my JS code.

Jake ran into this problem this morning and wanted this behaviour of a touchcancel when the gesture is the defined touch-action and receives touch events otherwise.

[Jake Archibald]

Here's the use-case I had:

https://jsbin.com/zawadi/1/quiet

In the above, I use the first touchmove in combination with touchstart to decide whether to manually handle side-swipe, or let the UA do its normal scrolling. I thought touch-action would help me out here, but it was actually a hindrance.

If I apply touch-action: pan-y; to the columns, the UA decides whether a particular touch gesture should result in scrolling & throttled touchmoves, or no scrolling and per-frame touchmoves. Unfortunately I don't get to find out which happened, so I'm still using my own JS to decide whether to take control & side-swipe or not. That makes touch-action: pan-y; not only redundant, but creates situations where my JS & the UA handling differ, such as my code decides to let scrolling through, but the UA decides to not scroll.

I'm not very experienced with touch events, but it'd be more useful to be if it behaved like this:

touch-action becomes your opt-in for more specific touch event behaviour. Assuming touch-action: pan-y;

• Fire touchstart
• Does this look like a panning action? If so:
• Fire touchcancel
• Allow this touch point to control scrolling
• No further touch events for this touch point
• Else:
• No not use this touch point to control scrolling
• Fire touchmove per-movement throttled to raf
• Fire touchend on touch end

This means I can use touchcancel to know the UA is in charge of this action, and touchmove to know I'm in charge.

Maybe that breaks a bunch of other use cases.

[Jake Archibald]

Also, should it be possible for a second touch to cancel an in-progress scroll? It isn't at the moment: https://jsbin.com/xoteq/1/edit?html,js,output

[Rick Byers]

Thanks for the input Jake.  So, it sounds like you're saying you'd expect the behavior that I proposed earlier on this thread.  Since you, Matt and Dan have all independently said that's what you expected to happen, that's pretty strong evidence that we should consider changing it (see Jared's concern that he'd like to hear from more web developers).

This would reintroduce some limited use of touchcancel on scroll (something we eliminated in M35 as confusing / different from other browsers).  If we're going to reintroduce this, then the question is when exactly would it happen.

 

In particular, if pan-x causes a touchcancel to be sent on horizontal scrolling, and pan-y causes a touchcancel to be sent on vertical scrolling, then what would you expect to happen for "pan-x pan-y" elements?  

On Mon, Aug 18, 2014 at 8:49 AM, Jake Archibald <jakear...@chromium.org> wrote:

Also, should it be possible for a second touch to cancel an in-progress scroll? It isn't at the moment: https://jsbin.com/xoteq/1/edit?html,js,output

For touchmove events (as done by your code) - no. during active scrolling cancelling a touchmove can NEVER stall the scroll (because those events are marked cancelable=false for performance reasons).  The fact that you've added a 2nd finger doesn't change this (eg. we don't want pinch updates to be janky either).  In fact your code will generate the following warning in the console that I added to try to help explain this:

Ignored attempt to cancel a touchmove event with cancelable=false, for example because scrolling is in progress and cannot be interrupted.

BUT the question is more interesting if you tried to do this on touchstart events. We debated this a bit here (see the comments on the ScrollUpdate and PinchBegin lines) and weren't sure of the ideal behavior.  Perhaps cancelling the touchstart for a 2nd finger while a scroll is in progress should stop the scroll (in addition to preventing the pinch).  What are you trying to achieve?

[Matt Gaunt]

I always felt the touchcancel behaviour was hard to follow because it was based on a timeout which was linked to preventDefault and even calling that would sometime fail if the browser was busy for too long.

--

Matt Gaunt

Developer Advocate @ Google

tw: @gauntface

G+: plus.google.com/+MattGaunt

Blog: https://gauntface.com/blog

[Jake Archibald]

On Mon, Aug 18, 2014 at 4:08 PM, Rick Byers <rby...@chromium.org> wrote:

This would reintroduce some limited use of touchcancel on scroll (something we eliminated in M35 as confusing / different from other browsers).  If we're going to reintroduce this, then the question is when exactly would it happen.

 

In particular, if pan-x causes a touchcancel to be sent on horizontal scrolling, and pan-y causes a touchcancel to be sent on vertical scrolling, then what would you expect to happen for "pan-x pan-y" elements?  

pan-x pan-y feels like an opt-out of touchmove events. You'd get a touchcancel after every touchstart.

On Mon, Aug 18, 2014 at 8:49 AM, Jake Archibald <jakear...@chromium.org> wrote:

Also, should it be possible for a second touch to cancel an in-progress scroll? It isn't at the moment: https://jsbin.com/xoteq/1/edit?html,js,output

For touchmove events (as done by your code) - no. during active scrolling cancelling a touchmove can NEVER stall the scroll (because those events are marked cancelable=false for performance reasons).  The fact that you've added a 2nd finger doesn't change this (eg. we don't want pinch updates to be janky either).  In fact your code will generate the following warning in the console that I added to try to help explain this:

Ignored attempt to cancel a touchmove event with cancelable=false, for example because scrolling is in progress and cannot be interrupted.

BUT the question is more interesting if you tried to do this on touchstart events. We debated this a bit here (see the comments on the ScrollUpdate and PinchBegin lines) and weren't sure of the ideal behavior.  Perhaps cancelling the touchstart for a 2nd finger while a scroll is in progress should stop the scroll (in addition to preventing the pinch).  What are you trying to achieve?

I was thinking of pinch-zoom or 2-finger rotate. Being able to cancel on the 2nd touchstart is good enough.

I found the demo in one of your comments https://code.google.com/p/chromium/issues/detail?id=240735#c47

[Rick Byers]

The touchcancel on ACK timeout is a completely separate thing that is unfortunately still present.  We're talking here about getting a touchcancel when you start scrolling.

[Rick Byers]

Yeah, so this doesn't work today (the 2nd touchstart is cancelable=false too), but I think it probably should.  Otherwise you can't implement custom pinch/rotate gestures that work after scrolling has started.  Perhaps cancelling the 2nd touchstart should prevent pinch and also return you to sync mode which means that scrolling can be interrupted.  Jared/Tim what do you guys think?

This would, of course, be an additional source of potential jank if someone want's to start pinch-zoom after scrolling has begun.  But in the common case of zooming before exceeding the slop threshold it would be no different.  I'm not sure if the small additional power is worth the perf impact here.

[Jared Duke]

Is there a custom pinch/rotate gesture template out there that works seamlessly with native scrolling (on any platform/browser combination)?  If there's precedent and/or some compelling real world scenarios here that are otherwise impossible, then we should absolutely support it.

My worry going forward is that, as we poke and prod the current touch dispatch model, we'll end up with an indecipherable web of state-coupled corner cases, nuances and gotchas.  If these changes move us toward an ideal model, for some consistent and well-defined ideal, that's great.

Do we have data on how touch-action affects IE 11's dispatch of touch events? Or even what their current touch dispatch model is (now that they're shipping a form of touch events with 8.1)?

[Rick Byers]

On Mon, Aug 18, 2014 at 12:56 PM, Jared Duke <jdd...@chromium.org> wrote:

Is there a custom pinch/rotate gesture template out there that works seamlessly with native scrolling (on any platform/browser combination)?  If there's precedent and/or some compelling real world scenarios here that are otherwise impossible, then we should absolutely support it.

 

The closest example I can come up with in a few minutes is the Android GMail application in the conversation view.  There you absolutely want native scrolling (built-in fling, etc.) but a custom pinch experience.  In particular pinching zooms the content of the messages, changing positions of the message headings without zooming them.  This would be impossible to implement on the web today I think (without re-implementing all of scrolling).

My worry going forward is that, as we poke and prod the current touch dispatch model, we'll end up with an indecipherable web of state-coupled corner cases, nuances and gotchas.  If these changes move us toward an ideal model, for some consistent and well-defined ideal, that's great.

Agreed.  I've been considering the Android and iOS models as roughly "ideal" here - low level touch events that always fire, but can be plugged into composable higher level behaviors.  Of course "ideal" would mean that the main thread is so responsive that we could use a simple sync touchmove model and not worry (much) about jank.  IMHO we should aim to be as close as we can to the simpler/powerful sync-touchmove model while retaining 60fps performance in the vast majority of cases.  The line here is of course highly controversial and continues to be our primary point of design debate.

Do we have data on how touch-action affects IE 11's dispatch of touch events? Or even what their current touch dispatch model is (now that they're shipping a form of touch events with 8.1)?

Yes, they claim they copied Chrome's behavior before we started experimenting with touchmove absorption etc.   So this is touchcancel on scroll - just like pointer events, which is obviously much simpler for them (cancelling pointer events but not touch events could get really hairy).

On Mon, Aug 18, 2014 at 9:10 AM, Rick Byers <rby...@chromium.org> wrote:

On Mon, Aug 18, 2014 at 11:48 AM, Jake Archibald <jakear...@chromium.org> wrote:

On Mon, Aug 18, 2014 at 4:08 PM, Rick Byers <rby...@chromium.org> wrote:

This would reintroduce some limited use of touchcancel on scroll (something we eliminated in M35 as confusing / different from other browsers).  If we're going to reintroduce this, then the question is when exactly would it happen.

 

In particular, if pan-x causes a touchcancel to be sent on horizontal scrolling, and pan-y causes a touchcancel to be sent on vertical scrolling, then what would you expect to happen for "pan-x pan-y" elements?  

pan-x pan-y feels like an opt-out of touchmove events. You'd get a touchcancel after every touchstart.

If we did this, then I think we'd absolutely want to make the 2nd-finger touchstart synchronous as we're discussing above.  Otherwise there's no way to disable pinch-zoom in some region without disabling native scrolling.  Doing both seems reasonable to me though.

What then about 'touch-action: manipulation'?  Today that's the same as 'pan-x pan-y' except it adds pinch-zoom.

Note that it we go this route of coupling touch-action with the touch event dispatch model somehow, then we should really try to get this into some spec somewhere.  I argued that the touch-action portion of the PE spec fully defined our behavior because it was orthogonal (after a minor edit) to the touch dispatch model.  I think this is fine (probably even desirable) - just more work.

[Jake Archibald]

On Mon, Aug 18, 2014 at 6:26 PM, Rick Byers <rby...@chromium.org> wrote:

On Mon, Aug 18, 2014 at 9:10 AM, Rick Byers <rby...@chromium.org> wrote:

On Mon, Aug 18, 2014 at 11:48 AM, Jake Archibald <jakear...@chromium.org> wrote:

pan-x pan-y feels like an opt-out of touchmove events. You'd get a touchcancel after every touchstart.

If we did this, then I think we'd absolutely want to make the 2nd-finger touchstart synchronous as we're discussing above.  Otherwise there's no way to disable pinch-zoom in some region without disabling native scrolling.  Doing both seems reasonable to me though.

What then about 'touch-action: manipulation'?  Today that's the same as 'pan-x pan-y' except it adds pinch-zoom.

Hmm, I guess with 'pan-x pan-y' you get touchmoves when there's > 1 touch point, but not if there's 'manipulation'. I'm kinda picking ideas out of the air here so probably wrong.

 

Note that it we go this route of coupling touch-action with the touch event dispatch model somehow

We're already doing that right? If you've got 'pan-y' and you touch-swipe horizontally we fire touchmove more frequently than we would without 'pan-y'.

 

, then we should really try to get this into some spec somewhere.  I argued that the touch-action portion of the PE spec fully defined our behavior because it was orthogonal (after a minor edit) to the touch dispatch model.  I think this is fine (probably even desirable) - just more work.

It'd be brilliant to bring the touch events spec up to the quality of pointer events in terms of details. Such as the ordering of equivalent mouse events, when touchcancel is fired, how frequently touchmove is fired, how it interacts with touch-action etc etc. As you say, just more work.

[Rick Byers]

On Tue, Aug 19, 2014 at 9:59 AM, Jake Archibald <jakear...@chromium.org> wrote:

On Mon, Aug 18, 2014 at 6:26 PM, Rick Byers <rby...@chromium.org> wrote:

On Mon, Aug 18, 2014 at 9:10 AM, Rick Byers <rby...@chromium.org> wrote:

On Mon, Aug 18, 2014 at 11:48 AM, Jake Archibald <jakear...@chromium.org> wrote:

pan-x pan-y feels like an opt-out of touchmove events. You'd get a touchcancel after every touchstart.

If we did this, then I think we'd absolutely want to make the 2nd-finger touchstart synchronous as we're discussing above.  Otherwise there's no way to disable pinch-zoom in some region without disabling native scrolling.  Doing both seems reasonable to me though.

What then about 'touch-action: manipulation'?  Today that's the same as 'pan-x pan-y' except it adds pinch-zoom.

Hmm, I guess with 'pan-x pan-y' you get touchmoves when there's > 1 touch point, but not if there's 'manipulation'. I'm kinda picking ideas out of the air here so probably wrong.

We've been advocating using 'manipulation' as a way to selectively disable double-tap (and hence the click delay) without affecting anything else.  I think we'd like to preserve that property.  So I think my preference would be to say that the touchcancel on scroll applies only with 'pan-x', 'pan-y' and (maybe) 'pan-x pan-y'.  Otherwise we leave things as they are today.  It's a little inconsistent though :-(

Note that it we go this route of coupling touch-action with the touch event dispatch model somehow

We're already doing that right? If you've got 'pan-y' and you touch-swipe horizontally we fire touchmove more frequently than we would without 'pan-y'.

 Not directly, no.  Touch-action says what gestures are allowed (so using pan-y suppresses the horizontal scroll gestures), and then the touch event model can change behavior based on those gestures.  I.e. the touch event model can be described in terms of "when scrolling occurs" without explicit reference to touch-action.  With this change we'd have to more directly couple them.  Perhaps that's just semantics / implementation though - practically the effect is pretty similar.

 

 

, then we should really try to get this into some spec somewhere.  I argued that the touch-action portion of the PE spec fully defined our behavior because it was orthogonal (after a minor edit) to the touch dispatch model.  I think this is fine (probably even desirable) - just more work.

It'd be brilliant to bring the touch events spec up to the quality of pointer events in terms of details. Such as the ordering of equivalent mouse events, when touchcancel is fired, how frequently touchmove is fired, how it interacts with touch-action etc etc. As you say, just more work.

Yeah, I'd love to do that too.  The main question now is whether anyone else wants to work with us on evolving the touch event spec.  At the moment the answer is no.

[Matt Gaunt]

I hate to say this, but generally I struggle with a lot of the reasoning behind touch event behaviors in Chrome and this is largely an overhang of developing native Android apps.

I really just wanted to get all the touch events, not mucked around with by Chrome, and have the ability to prevent native behaviour at any point for my own purposes, otherwise still listen.

I understand we can't do that for performance reasons, but with things like throttling - it's hard to see a use for the throttled events. With the touchcancel of touchevents when I don't call preventDefault means I can't just listen and override later and runs the risks of the time out (although I'm hearing less about this).

If the above were fixed, the use of touch-events would (IMO) be easier, more sane and more predictable and then touch-actions make sense since I can define the native behaviour, listen to all touch events and filter what behaviour is done (native or custom or both).

I think the above requests in this thread, feels like a middle ground between handing over some of the work to native and doing some of the work in JS only when needed since currently we fall into a weird middle ground of not quite doing one or the other.

*Fin Ramble*

[Jake Archibald]

I wonder if there's a common solution that works with "scroll" too, a way to opt-in to sync events without disrupting UA behaviour at the cost of performance.

I agree with Matt that there's not much use in throttled touchmove events that I'm aware of. But also I don't have any use cases for cancelling a touchmove that isn't the first of a particular touch.

[Rick Byers]

On Wed, Aug 20, 2014 at 6:29 AM, Jake Archibald <jakear...@chromium.org> wrote:

I wonder if there's a common solution that works with "scroll" too, a way to opt-in to sync events without disrupting UA behaviour at the cost of performance.

Having some way to explicitly opt-in to sync touchmove events would be nice and would perhaps give us more freedom for the default behavior.  We've been throwing around some ideas around this for awhile, eg. scroll-delay. 

I agree with Matt that there's not much use in throttled touchmove events that I'm aware of. But also I don't have any use cases for cancelling a touchmove that isn't the first of a particular touch.

Yeah our original plan didn't have throttled touchmove events at all ("touchmove absorption"), but we found that broke some websites.  Eg. imagine you have a button which depresses on touchstart, aborts on touchmove beyond some magic distance, and activates on touchend if not yet aborted.  You're effectively relying on being able to track the finger to some degree while scrolling.

FWIW I don't think the "throttled" aspect of these events is terribly important, what matters is that they're async and so uncancelable.  We wanted to start conservative, but I could see removing the throttle if that made people's lives easier (from a perf perspective they're no worse than the async scroll events we already fire once per scroll frame).  We'd need some mechanism to ensure they don't pile up (since we don't have the back-pressure of ACK messages for async events), but we could come up with something (and should probably do that anyway even when they're throttled).