Intent to Implement: Pointer Events

[Rick Byers]

Contact emails

rby...@chromium.org, mus...@chromium.org

Summary

The Pointer Events API is a low-level input API for mouse, touch and stylus introduced by IE.  Pointer Events extends the MouseEvent model while offering a replacement for all uses of Mouse and Touch events.  Based on the feedback we’ve received, and the productive collaboration in the Pointer Events working group, I now believe we should implement this API in Blink.

Motivation

Last year we announced that, despite our involvement in the Pointer Events standard, we were going to focus on incremental improvements to existing APIs (like Touch Events), rather than implement Pointer Events in Chromium.  Since then we’ve received a steady stream of feedback from web developers, framework authors, and other browser vendors indicating that they see Pointer Events as a highly valuable addition to the platform.  Since we’re committed to a web platform which evolves collaboratively through open discussion and data from real-world development, we need to take this feedback very seriously.

During this same period, many of our outstanding technical concerns with the API have made progress towards being addressed.  Enabling rich touch effects which interact with scrolling now has an official avenue for exploration within the W3C, and improvements to touch-action richness are planned within the PEWG.  Mouse event compatibility issues with existing models for touch are being addressed (and planned for a future version of the PE spec).  The PEWG has been responsive to feedback and we enjoy the productive and collaborative relationship we have with the the other members including representative from the IE, jQuery, Mozilla and Opera teams.

Pointer Events offers some technical advantages over the existing use of Touch Events and Mouse Events.  Most notably, pointer event listeners never block scrolling, and so replacing all touch event handlers with pointer event handlers will address the main longstanding source of scroll-start jank we see on Android (irrespective of whatever scheduler improvements we’re able to make to better prioritize input handling).

Compatibility Risk

IE: Public support (shipping since IE10)

Firefox: Public support (implemented behind a flag)

Safari: Publicly opposed  

Web developers: Mostly positive public support, including jQuery and Dojo

Pointer Events has pretty broad support, but we have some outstanding concerns with the existing API which probably cannot be addressed without a non-trivial breaking change (see below for details).  The key outstanding challenge is to try to identify a design which addresses our concerns without seriously breaking websites which support the existing API.  We will of course work closely with the other vendors to try to find a design we can all agree to implement consistently.

We don’t expect Touch Events to go away in the foreseeable future, but with IE adding support for Touch Events, they’ve shown that it’s possible to support both models simultaneously without serious web compat impact.  Jacob Rossi on the IE team has been very helpful (through his investment in the Touch Events Community Group), helping us understand their experience with Touch Events and the interaction with Pointer Events.  I’m confident that we can come to a consensus on a model for pointer, touch and mouse events that is highly interoperable between IE, Chrome and Firefox, while remaining compatible with Safari’s implementation of touch and mouse events.

Ongoing technical constraints

Pointer Events as currently defined requires a hit-test on every pointermove (as is the case for mousemove, but not touchmove).  This imposes a performance cost on the engine which the major native mobile platforms and browsers don’t have.  I intend to work with the PEWG to identify some (probably breaking) API changes to allow us to avoid this cost for touch by default.  This will be challenging to do without substantial compat pain, but I’m optimistic some solution can be reached to enable us to support Pointer Events without committing to this performance constraint.

Also implementing the desired performance properties of Pointer Events means we must be able to do accurate touch-action hit-testing on the impl thread.  This is work we planned to do anyway for ‘scroll-blocks-on: none’, and is expected to be easy once CC operates in terms of display lists instead of layers.

Spec

www.w3.org/TR/pointerevents/

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

OWP launch tracking bug?

http://crbug.com/196799

Link to entry on the feature dashboard

https://www.chromestatus.com/features/4504699138998272

Requesting approval to ship?

No.  Implementation is expected to take some time.

[Jacob Rossi]

This is exciting!  Obviously, we fully support you here. Rick and I had the opportunity to discuss some of the technical constraints and ideas on how we can make some improvements to advance the API further, in particular WRT implicit vs explicit capture.  We look forward to working on those ideas together. 

 

-Jacob

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[st...@opendirective.com]

That's very encouraging news indeed. Thank you.

Steve Lee
OpenDirective.com

[dani...@gmail.com]

Three cheers for unified interaction events! Huge! A big thank you from all web devs to everyone involved in making this happen.

- Daniel

[andrea.g...@gmail.com]

about this:

> it’s possible to support both models simultaneously without serious web compat impact. 

turns out, when IE has problem with PoitnerEvents it reflects into Touch Events too. This page shows a demo which has been filed as bug and so far ignored:

http://www.3site.eu/ie/

You tap and hold in mobile IE and after a little while it will trigger pointerup by its own.

Absolutely not reproducible with any Touch Events based device **but** if you use these instead where supported in IE, the bug is absolutely the same: it will trigger by its own touchend without a reason.

It's not clear if this happens in every mobile phone but at least you have a page to test against PointerEvents and behaviors VS touch events.

I still have mixed feelings about this last-minute change of direction but I believe for the sake of Web Standards, and considering the disappointing state of Touch Events proposal, this is the best thing to do. I wish Apple would follow ASAP too.

Thanks and Best Regards

[andrea.g...@gmail.com]

about this:

> it’s possible to support both models simultaneously without serious web compat impact. 

[Jacob Rossi]

> about this:

> > it’s possible to support both models simultaneously without serious web compat impact. 

> turns out, when IE has problem with PoitnerEvents it reflects into Touch Events too. This page shows a demo which has been filed as bug and so far ignored:
http://www.3site.eu/ie/

> You tap and hold in mobile IE and after a little while it will trigger pointerup by its own.

> Absolutely not reproducible with any Touch Events based device **but** if you use these instead where supported in IE, the bug is absolutely the same: it will trigger by its own touchend without a reason.

> It's not clear if this happens in every mobile phone but at least you have a page to test against PointerEvents and behaviors VS touch events.

This is an unfortunate bug that has to do with the hosting mechanism for the rendering engine on Windows Phone 8.1 and below. The system gesture recognizer on WP was not configurable in such a way that we could fix this under that architecture.  In Windows 10, we’ve changed our hosting mechanism and this is no longer an issue.  The issue was purely implementation-specific and there’s nothing inherent in Pointer Events that should suggest implementing it should affect an existing Touch Events implementation.

In fact, we’ve been quite pleased thus far with our results of supporting both. The only real issues continue to be problems with enabling TE on more devices—we still see lots of issues with TE on machines that have additional inputs other than touch . But perhaps supporting Pointer Events interoperably gives justification to not worry about enabling TE everywhere.

-Jacob

 

[Andrea Giammarchi]

Thanks for the clarification, I believe it should be somehow mentioned in the related bugs too?

https://social.technet.microsoft.com/Forums/ie/en-US/0e939601-c14c-4637-b0c8-08277e850900/ie11-mobile-windows-phone-81-touchend-fired-without-user-action?forum=ieitprocurrentver

https://connect.microsoft.com/IE/feedback/details/955065/mouseup-touchend-pointerup-triggers-unexpectedly-in-ie11-windows-phone-8-1

https://connect.microsoft.com/IE/feedback/details/1076515/mouseup-pointerup-events-are-fired-automatically-when-mouse-or-pointerdown-are-held-windows-phone-8-1-ie11

I thought it was some `gesturehold` or `holdvisial` behavior messing up pointers too.

Best Regards

[Philip Rogers]

The issues brought up in previous pointer-events threads resonate with me but I've come around to think this API is worth implementing from the developer productivity angle. A unified model helps developers get past the initial hurtles of getting pages off the ground.

Can you describe concrete plans for addressing the capture-by-default issue?

[Rick Byers]

On Fri, Mar 27, 2015 at 6:45 PM, Philip Rogers <p...@chromium.org> wrote:

The issues brought up in previous pointer-events threads resonate with me but I've come around to think this API is worth implementing from the developer productivity angle. A unified model helps developers get past the initial hurtles of getting pages off the ground.

Thanks Philip!

Can you describe concrete plans for addressing the capture-by-default issue?

It's still early, but see this public-pointer-events thread for some details on a particular proposal (which is a better place to debate the design than blink-dev anyway - feel free to chime in on that thread).   The big challenge here will be finding a solution which isn't too breaking to existing content.  I think we should start with the simplest / cleanest solution (eg. what I've proposed there) and measure the compat impact, then go from there. 

[Ojan Vafai]

On Wed, Mar 25, 2015 at 9:08 AM Rick Byers <rby...@chromium.org> wrote:

Ongoing technical constraints

Pointer Events as currently defined requires a hit-test on every pointermove (as is the case for mousemove, but not touchmove).  This imposes a performance cost on the engine which the major native mobile platforms and browsers don’t have.  I intend to work with the PEWG to identify some (probably breaking) API changes to allow us to avoid this cost for touch by default.  This will be challenging to do without substantial compat pain, but I’m optimistic some solution can be reached to enable us to support Pointer Events without committing to this performance constraint.

I just want to call this out as I think this is very important to our performance goals. We've found that the hit-test itself is a surprisingly non-trivial percentage of per-frame CPU time on reasonably powered mobile devices. A large percentage of use-cases, particularly on mobile, do not need hit-tests on every move. I strongly believe that the default should be to capture on pointerdown. How you opt-in to the non-capturing behavior is, as you point out, challenging, but I'm hopeful we can find a reasonably satisfying solution.

On a related note, it's my understanding that both Android and iOS do not do hit-tests on every touch move. So, requiring it for the web would put the web at a distinct performance disadvantage.

[Philip Jägenstedt]

The WebKit feedback is from 2012, have you talked to anyone working on WebKit about what they'll do if every other engine supports pointer events?

From that email: "Combining the very different models of touch and mouse is a bad idea. They are distinct and for anything beyond the trivial, you really need to treat them separately to get a good outcome. The basic premise of pointer events seems to be over-abstraction at the expense of the user experience."

Is there anything to this criticism? PointerEvent.pointerType allows the distinction to be made, but is it the norm or the exception to want to handle touch and mouse in a unified way?

Philip

[Andrea Giammarchi]

That sentence is still valid in my opinion, Pointer Events simply moved the problem inside the listener, instead of having separated concerns outside the same.

With Pointer Events we have now to understand internally, instead of using an appropriate event name separating concerns, the nature of the event: was it a mouse, a touch, a pen, or an empty string that no developer knows at that point how to fallback or handle?

touchmove would never happen if at least a touchstart happened before, so now it's  a developer problem if pointermove can be triggered at any time, the developer needs to handle this too.

Considering that DOM handlers can also be Objects, and not just function, the ability to create classes of gestures and pointers per type is lost, since every event needs potentially an `if/else if/else if/else` switch inside each event to correctly understand what to do: meaning also more performance lost in user-land.

As summary: I've been working with PointerEvents for the entire IE10 and beginning of IE11 era, and all I needed was a working and consistent implementation of Touch events there too.

There are very few things if not just one that Pointer Event does better, but the API itself makes things more complicated from a developer prospective, without providing any real/concrete advantage.

Custom gestures are also easier to implement via Touch thanks to the TouchList ... in PoitnerEvents we need to somehow handle pointers ID too.

That being said, and even if I cannot agree more with that 2012 answer, Touch territory is apparently doomed so, as Web developer, I'd sign anytime / anywhere to have a consistent API across all platforms, even if that will end up being this PointerEvents thing that at least is there, and it's already deployed somewhere.

Just my 2 cents from a 6 years Mobile Web developer

[Rick Byers]

On Fri, Apr 17, 2015 at 3:00 PM, Philip Jägenstedt <phi...@opera.com> wrote:

The WebKit feedback is from 2012, have you talked to anyone working on WebKit about what they'll do if every other engine supports pointer events?

I've talked regularly (privately) with Safari engineers about this, but I'm hesitant to put words into their mouth here.  High level, my impression is that what the engineers want or don't want to do here matters less than what the Apple lawyers want (and I won't pretend to try to know how they think).

From that email: "Combining the very different models of touch and mouse is a bad idea. They are distinct and for anything beyond the trivial, you really need to treat them separately to get a good outcome. The basic premise of pointer events seems to be over-abstraction at the expense of the user experience."

Is there anything to this criticism? PointerEvent.pointerType allows the distinction to be made, but is it the norm or the exception to want to handle touch and mouse in a unified way?

There's certainly some validity to that argument - often you do really need to design separately.  But I'd argue it's more common on the web to want a similar implementation.  Eg. the Apple watch site (like many websites) has an image carousel that you swipe or drag to move between images.  Internally they've implemented this with event handlers called 'onPointerDown', 'onPointerMove' which are hooked up to both touch and mouse events.  They attempt to do the exact same thing for touch and mouse but unfortunately their logic for trying to unify the two events types has a bug (depends on DOM attributes being incorrectly on the instance instead of the prototype chain).  So it seems that at least some of Apple's own web developers disagree with the position of the Safari team here.

Here's a synthetic example where I tried to write a drag-and-drop UI (like the Google+ circle editor) as simply as possible with the different possible combinations of event types.  Using mouse+touch events was dramatically more difficult than using pointer events.

 

For me the difference is that when you do want to unify logic between touch and mouse it's quite difficult to do so correctly using touch events and mouse events (especially on devices that have both touch and mouse input).  However if you're using pointer events and want to have completely separate handling for the different pointer types, then that's quite easy (eg. a trivial library could allow you to easily register for touch-only pointer events).

But this is a complex debate and I don't expect everyone to agree.  The important thing is that we've got a LOT of developers (including major frameworks like jQuery and Dojo, and representatives from IE, Firefox and Opera) saying this is the API they want - we need to take that feedback seriously.  Providing pointer events won't take away the option to use touch and mouse events if that's really what developers prefer.

Philip

 

On Fri, Apr 17, 2015 at 10:29 AM, Andrea Giammarchi <andrea.g...@gmail.com> wrote:

That sentence is still valid in my opinion, Pointer Events simply moved the problem inside the listener, instead of having separated concerns outside the same.

With Pointer Events we have now to understand internally, instead of using an appropriate event name separating concerns, the nature of the event: was it a mouse, a touch, a pen, or an empty string that no developer knows at that point how to fallback or handle?

touchmove would never happen if at least a touchstart happened before, so now it's  a developer problem if pointermove can be triggered at any time, the developer needs to handle this too.

True.  But it's a trivial couple lines of code if you want to have a way of ignoring all hover events.  If I could do it over again I would have argued for separate pointerhover and pointerdrag events, but since that's so trivial to provide in a library I don't think it's worth the cost at this point to change the spec.  It's possible we could incrementally add that though if we felt it was important.  I'm happy to discuss your feedback at public-poi...@w3.org.  

Considering that DOM handlers can also be Objects, and not just function, the ability to create classes of gestures and pointers per type is lost, since every event needs potentially an `if/else if/else if/else` switch inside each event to correctly understand what to do: meaning also more performance lost in user-land.

Again this is trivial to build a wrapper to accommodate when it's the pattern you want, right? 

As summary: I've been working with PointerEvents for the entire IE10 and beginning of IE11 era, and all I needed was a working and consistent implementation of Touch events there too.

Luckily you're getting that too (IE 11 mobile supports touch events, and they plan to extend it to desktop).  If you're happy with that, then no need to change.  We'll be sure not to break any touch events behavior when implementing pointer events.

There are very few things if not just one that Pointer Event does better, but the API itself makes things more complicated from a developer prospective, without providing any real/concrete advantage.

There's obviously lots of disagreement on this point.  For example, not blocking the start of threaded scrolling is something I see as a HUGE performance advantage of pointer events.

Custom gestures are also easier to implement via Touch thanks to the TouchList ... in PoitnerEvents we need to somehow handle pointers ID too.

I know opinions differ on this.  Pointer Events certainly favors single-point interaction (eg. it's much easier to write code that only wants to consider single-finger interactions thanks to isPrimary and the lack of touch lists).  But multi-touch gesture recognition is always going to be non-trivial and somewhat specialized, I'm fine with that being the purview of a few popular libraries / frameworks.  Such library owners I talk to (eg. Jorik from Hammer.js) say they are happy with the design of pointer events.

There is one bit I think is missing from the spec here that I've been meaning to bring up with the working group.  I've just filed an issue on the spec here to track it for PE level 2.

That being said, and even if I cannot agree more with that 2012 answer, Touch territory is apparently doomed so, as Web developer, I'd sign anytime / anywhere to have a consistent API across all platforms, even if that will end up being this PointerEvents thing that at least is there, and it's already deployed somewhere.

This point is key.  It's precisely because we highly value a healthy interoperabilty ecosystem (vendors and frameworks working together to make improve web APIs) that we reversed our opinion here.  We can bikeshed forever on minor technical API differences (especially unimportant when libraries/frameworks can easily provide they sugar they prefer), but if we don't even have a forum effectively building consensus between the vendors then there's no point.

Just my 2 cents from a 6 years Mobile Web developer

Thanks for your feedback, I appreciate it!

[Andrea Giammarchi]

Somewhere someone will end up saying how better was drag-and-drop on mobile ... which is really not common pattern for phones, beside some icon movement, rather a desktop pattern and maybe tablet.

Yes, drag-and-drop is the only thing I think is simplified via Pointer Events, for the rest, during those days, this solved all interactions I needed:

https://github.com/WebReflection/ie-touch#ie-touch

The blocking scrolling part I'm not sure I understand the benefit, I cannot meaningfully do anything if an underlying panel is scrolling under my finger unless it's a game, but again I managed to create via touch interfaces that were scrolling horizontally and also vertically within the page (beside iOS7 that was freezing all the things there ...)

About gestures: not only these are quite straight forward to implement via gesture events in iOS, even during Meego time there were Webkit developers involved in bringing in them as transformation events and I think PonterEvents without meaningful/improved PointerGestures are a half baked thing so I hope these will be shipped together and hopefully, gestures speaking, improved too.

Last, but not least, polyfilling PointerEvents on old Android 2.X devices has never been an option due overhead and *really* slower performance.

Like you said, Touch will stay, so I don't mind feature detecting 'navigator.pointerevents' first and `'ontouchend' in window` after.

Anyway, like I've said, I can't wait to see this shipped everywhere for the peace of all developers.

Best Regards and have a nice week end

[Daniel Buchner]

In almost every circumstance, I've seen a desire from devs to have most mouse/touch events unified. Furthermore, nothing precludes adding separate listeners in cases where separate evaluation (using the type prop) is necessary. I don't see much validity in the argument myself, but welcome other thoughts/comments.

- Daniel

[Andrea Giammarchi]

> I've seen a desire from devs to have most mouse/touch events unified

This is the part I have strong doubts, not on what they are saying, but on the real-world scenarios they are trying to solve.

Developers have been told that PointerEvents is a unique interface for events and I am pretty sure most of them believes PointerEvents will automagically solve all problems for them.

I've worked with touch for years and MSPointerEvents for as long as these existed and the overhead is not as trivial as it looks on paper.

Nobody swipes with a mouse, it's not natural, everyone clicks ... nobody scroll pressing around, that's a Touch thing, not a mouse one. Drag and drop is not friendly via fingers and small screens, I don't even know why we have it, if not as some inherited interaction-pattern legacy from Desktop.

You'll never pinch or spread with a mouse or a pen, and you'll rarely hold. You pointerdown, pointermove, and pointerup to select text with a mouse, not to swipe screen and change desktop or gallery ... I'm looking forward to see which library will win because die-hard vanilla JavaScript developers that ate the "PointerEvents all in one miracle propaganda" will have hard time.

Regardless, and I'm very done with my boring arguments, I'm still all in for cross-platform solutions. If these are not ideal, as log as these work consistently, we can build better, simplified, meaningful API on top. At which cost though, we dont' know yet ... but I'm sure by the time PointerEvents will be everywhere, the default Hardware will be 16X better than the one we had 2 years ago.

Best Regards

[Rick Byers]

I want to give a quick update on the implementation status and potential launch plan for Pointer Events in blink.  tl;dr is that we're starting a finch trial where it'll be enabled for a tiny subset of Chrome users in order to collect the compat data we need to attempt to resolve the outstanding design questions with the W3C working group.

As mentioned at BlinkOn, Mustaq and Navid recently hit a big milestone - getting PE to be feature complete and largely web compatible, working with the PEWG to resolve dozens of spec issues in the process.  As of Chrome 53 it's enabled with "--enable-experimental-web-platform-features" and lately several of us have been doing our regular browsing with it enabled without much issue.  You can still enable/disable it explicitly via chrome://flags/#enable-pointer-events and if you find issues that reproduce only with it enabled, please file a bug using this template.

However, as discussed in the intent to implement (below) there are still some outstanding design debates.  In particular, our implementation uses an "implicit capture" model for touch that avoids doing hit-tests on every pointmove when pointerType=touch (for both performance and UX reasons).  But we lack consensus in the W3C on if (or how) the spec should change for this (and so the interop risk is still too high to propose shipping this design now).  Making progress in resolving this debate will require concrete web compat data which can only really be obtained through real-world usage (simulating human-like input on real sites and detecting breakage is notoriously hard).  So in order to unblock progress we're using Finch to enable pointer events on a tiny experimental subset of Chrome users (eg. all canary and dev channel users for a period of time unless we encounter non-trivial breakage).  Since this is only a small fraction of Chrome users, we don't believe there's any risk of web developers depending on a particular variant of support from Chrome existing (no more than the existing risk of developers coding for the design that's available behind flags).  Officially "shipping" pointer events in blink is still blocked on coming to a consensus in the W3C on the design we can all agree to.

Let me know if you have any questions or concerns with this plan.  And please be on the lookout for input-related site bugs that could be due to pointer events being enabled (file bugs here, or at least just add Blink>Input and they'll get to the input-dev triage queue).  You can quickly tell if pointer events is enabled for you with this pretty demo page (or my ugly eventTest).

Thanks!

   Rick