Intent to Ship: rAF Aligned Mouse Input

317 views
Skip to first unread message

Dave Tapuska

unread,
May 1, 2017, 10:41:43 AM5/1/17
to blink-dev
dtap...@chromium.org None. The life-cycle of when these events should fire is not in any specification.
Align continuous mouse events (mousemove, mousewheel) so that they occur close to the rAF as possible. Historical events can be accessed via getCoalescedEvents (see https://w3c.github.io/pointerevents/extension.html#extensions-to-the-pointerevent-interface)

Note: rAF aligned touch events are already shipped in M59. Continuous events coalesce together; ie. the newer event usually overrides the effect of the first one. Most often these continuous mouse events cause work that is not necessary, hit tests, and style updates. If you get two occurring inside the same rAF call then the first work is useless because it didn't actually update the display.

Note that some drawing apps require all the details of all the points that occurred. These are accessible via the historical event API.

We have conducted a Variation trial on Canary and Dev releases since March 21st 2017 and no reported issues have been filed with regard to the adjusted timing of these events.
Firefox: Mixed signals (see https://www.w3.org/Bugs/Public/show_bug.cgi?id=28876) Edge: No public signals Safari: No public signals Web developers: Mixed signals (some positive
https://groups.google.com/a/chromium.org/forum/#!searchin/input-dev/rAF$20Aligned/input-dev/Aqdge_cj060/DwaDu1o9AQAJ, some negative
https://bugs.chromium.org/p/chromium/issues/detail?id=642368&desc=2#c35) Drawing Apps might have reduced fidelity with fast moving mice. Can be corrected by slowing the gesture or updating the app to read the historical events for even better fidelity that existed in the past.
None
Yes https://crbug.com/703344 https://www.chromestatus.com/features/6192761317359616

Chris Harrelson

unread,
May 3, 2017, 8:31:28 PM5/3/17
to Dave Tapuska, blink-dev
Hi Dave, some comments below.

On Mon, May 1, 2017 at 7:41 AM, Dave Tapuska <dtap...@chromium.org> wrote:
dtap...@chromium.org None. The life-cycle of when these events should fire is not in any specification.

I see that this was filed and discussed in 2015. Was there an attempt to write it up and spec it precisely as part of the pointer events spec since then?
 
Align continuous mouse events (mousemove, mousewheel) so that they occur close to the rAF as possible. Historical events can be accessed via getCoalescedEvents (see https://w3c.github.io/pointerevents/extension.html#extensions-to-the-pointerevent-interface)

Note: rAF aligned touch events are already shipped in M59. 
Continuous events coalesce together; ie. the newer event usually overrides the effect of the first one. Most often these continuous mouse events cause work that is not necessary, hit tests, and style updates. If you get two occurring inside the same rAF call then the first work is useless because it didn't actually update the display.

Note that some drawing apps require all the details of all the points that occurred. These are accessible via the historical event API.

We have conducted a Variation trial on Canary and Dev releases since March 21st 2017 and no reported issues have been filed with regard to the adjusted timing of these events.
Firefox: Mixed signals (see https://www.w3.org/Bugs/Public/show_bug.cgi?id=28876)

Did they say they might adopt our model if it works out? As it stands, this intent would increase interoperability problems compared to other browsers, correct? In addition to speed/quantity of events, getCoalescedEvents will be only on Chrome I assume?
 
Edge: No public signals Safari: No public signals Web developers: Mixed signals (some positive
https://groups.google.com/a/chromium.org/forum/#!searchin/input-dev/rAF$20Aligned/input-dev/Aqdge_cj060/DwaDu1o9AQAJ, some negative
https://bugs.chromium.org/p/chromium/issues/detail?id=642368&desc=2#c35) Drawing Apps might have reduced fidelity with fast moving mice. Can be corrected by slowing the gesture or updating the app to read the historical events for even better fidelity that existed in the past.
None
Yes https://crbug.com/703344 https://www.chromestatus.com/features/6192761317359616

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Dave Tapuska

unread,
May 4, 2017, 9:52:26 AM5/4/17
to Chris Harrelson, blink-dev
> I see that this was filed and discussed in 2015. Was there an attempt to write it up and spec it precisely as part of the pointer events spec since then?

No there wasn't an attempt to formally spec it.  Do you have an example from CSS/HTML that defines the document lifecycle should occur? I can see us augmenting the HTML spec to add another step here indicating to dispatch any pending input at step 1.  And then augmenting the UI Events spec indicating that UA may choose to wait to dispatch a native mousemove/wheel event by placing it in a pending queue that is flushed during rAF. But a UA must maintain order of all events.

> Firefox: Mixed signals

I've pinged the thread to get updated responses from Microsoft and FireFox

Rick Byers

unread,
May 4, 2017, 3:55:04 PM5/4/17
to Dave Tapuska, Chris Harrelson, blink-dev, ato...@chromium.org, Domenic Denicola
Note that the interoperability risk here is part of a larger issue about the rendering processing model not being fully specified or interoperable.  I'm not sure to what extent we're actively driving specification / interop work here. Aleks / Domenic, anything you can add about the current state, plans or importance?

Having event timing depend entirely on the underlying OS event timing (as it does today for mouse) already isn't really interoperable.  For example, Android and iOS each moved to vsync-aligned input events at different points in their development years ago, changing the timing of touch events in browsers on those platforms.  I haven't heard of that posing an interop problem in practice for anyone.  Hopefully moving to a precisely defined (and specifiable) timing model rather than depending on OS details will ultimately decrease long-term interop risk.  Note you can use https://rbyers.github.io/event-timing.html to see how exactly input events (and coalesced events) line up with the timing of RAF on different browsers - there isn't a ton of interoperability here today across OSes and different input types.

I think the main risk is compat risk around the need to use getCoalescedEvents.  Today you can get a kind of OK mouse/stylus drawing experience in Chrome and other browsers without getCoalescedEvents, but it won't be great (typically will have some dropped frames, and will typically fail to keep up with 240hz+ input devices).  But this change will make that worse.  I expect we will hear of a couple drawing apps where people complain that the curves aren't as smooth anymore.  But the solution (using getCoalescedEvents) will be much better than the current status quoa.  Here's a demo - the bigger dots are the actual input events and (if the browser you're on support it), the smaller dots are the coalesced events.  Add ~20ms of jank via the "extra work" field.

So my judgement in terms of our compat principles is:

Page views impacted: not something we can really /measure I think, but anecdotally somewhat low but non-trivial - just art/drawing scenarios

Performance: positive (though hard numbers would help justify this) - mousemove events can drive a lot of work on some sites, doing them at a bad time/rate can be really bad for perf.  Dave do you have some traces or other data from specific sites in mouse-drag scenarios?  Dragging-animation scenarios (which this will help) are very common, especially in comparison to the art scenarios that will be hurt.

Severity of breakage:  low - should be largely cosmetic (drawn curves with ~60 points per second instead of eg. an unreliable 120).  Users will move their input device more slowly to compensate when necessary.

Ease of adaptation: moderately positive - getCoalescedEvents API was designed/shipped to easily enable this transition for the rare use cases that need to opt-in.  This requires the use of pointer events, but pointer events themselves are designed to be incrementally usable inside of code designed for MouseEvent (i.e. getCoalescedEvents returns an array of PointerEvent objects, each of which is also a valid MouseEvent object with a new 'type' value - 'pointermove' instead of 'mousemove').

Interoperability: some real risk on the desktop/mouse-specific front.  This will mean Chrome is different from AFAIK every other desktop browser.

Standards conformance: neutral now (unspecified) but positive long-term because this is something we COULD specify (as opposed to leaving it an impl detail)

Outreach: none yet AFAIK, but probably important

Personally I think we should just ship this along with a snazzy blog post showing how to leverage getCoalescedEvents (while continuing to work independently on getting the rendering model spec'd and interoperable).  But I've been pretty close to this issue over the years so should recuse myself from API owners voting.

Rick



Dave Tapuska

unread,
May 9, 2017, 5:16:51 PM5/9/17
to Rick Byers, Chris Harrelson, blink-dev, ato...@chromium.org, Domenic Denicola
On the topic of performance.

I don't have specific data from drag events.
But I do have some awesome performance results: Canary on Windows performed 32% less hit tests which was roughly 35% time savings in the total duration of hit tests we performed. Although I can't express actual numbers there is lots of CPU time saved.

dave.

Chris Harrelson

unread,
May 9, 2017, 5:31:23 PM5/9/17
to Dave Tapuska, blink-dev
On Thu, May 4, 2017 at 6:52 AM, 'Dave Tapuska' via blink-dev <blin...@chromium.org> wrote:
> I see that this was filed and discussed in 2015. Was there an attempt to write it up and spec it precisely as part of the pointer events spec since then?

No there wasn't an attempt to formally spec it.  Do you have an example from CSS/HTML that defines the document lifecycle should occur? I can see us augmenting the HTML spec to add another step here indicating to dispatch any pending input at step 1.  And then augmenting the UI Events spec indicating that UA may choose to wait to dispatch a native mousemove/wheel event by placing it in a pending queue that is flushed during rAF. But a UA must maintain order of all events.

Possibly. What about also modifying the pointer event spec to say something about all this?

Rick Byers

unread,
May 10, 2017, 11:19:02 PM5/10/17
to Chris Harrelson, Dave Tapuska, blink-dev
On Tue, May 9, 2017 at 5:30 PM, Chris Harrelson <chri...@chromium.org> wrote:

On Thu, May 4, 2017 at 6:52 AM, 'Dave Tapuska' via blink-dev <blin...@chromium.org> wrote:
> I see that this was filed and discussed in 2015. Was there an attempt to write it up and spec it precisely as part of the pointer events spec since then?

No there wasn't an attempt to formally spec it.  Do you have an example from CSS/HTML that defines the document lifecycle should occur? I can see us augmenting the HTML spec to add another step here indicating to dispatch any pending input at step 1.  And then augmenting the UI Events spec indicating that UA may choose to wait to dispatch a native mousemove/wheel event by placing it in a pending queue that is flushed during rAF. But a UA must maintain order of all events.

Possibly. What about also modifying the pointer event spec to say something about all this?

The spec intentionally leaves event timing undefined today since there isn't interoperability yet (it's typically up to the underlying OS).  We likely wouldn't want to put this into the Level 2 spec since it's in the process of getting ready for CR and we won't have two implementations.  But we could try to put something into our incubation document (along with the related getCoalescedEvents API), though I'm not sure what the hook into the HTML spec would look like exactly.

But I don't think it would be correct to change just the pointer events behavior and not also the behavior of mouse/touch events.  So I'm not sure trying to put this into any one (or all 3) of the different pointer-based-input specs is really the right place...

Dave Tapuska

unread,
May 16, 2017, 4:18:26 PM5/16/17
to Rick Byers, Chris Harrelson, blink-dev
I've made some modifications to the event-loop explainer document to hopefully clarify some of the designs that we made. I hope that this will help with the concerns that you had Chris.

dave.

Chris Harrelson

unread,
May 16, 2017, 4:31:42 PM5/16/17
to Dave Tapuska, Rick Byers, blink-dev
I think this is great and enough to move forward with this intent. I hope we can soon make progress making an official spec of the event loop sometime soon.

LGTM1

Jochen Eisinger

unread,
May 22, 2017, 7:12:28 AM5/22/17
to Chris Harrelson, Dave Tapuska, Rick Byers, blink-dev
lgtm2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.




--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw9SybU70a4uM6WsevFzJgrB7JfqRqcCXYh%3DJkkv6Xpi%2Bg%40mail.gmail.com.

TAMURA, Kent

unread,
May 22, 2017, 11:10:33 PM5/22/17
to Jochen Eisinger, Chris Harrelson, Dave Tapuska, Rick Byers, blink-dev
LGTM3


lgtm2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.




--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw9SybU70a4uM6WsevFzJgrB7JfqRqcCXYh%3DJkkv6Xpi%2Bg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALjhuidt%3D649g4%2BmoSqcwONHw7YuzQ9cD84GRSUucbCeBAjGAQ%40mail.gmail.com.



--
TAMURA Kent
Software Engineer, Google


juj...@gmail.com

unread,
Aug 28, 2017, 10:44:59 AM8/28/17
to blink-dev, joc...@chromium.org, chri...@chromium.org, dtap...@chromium.org, rby...@chromium.org
Just saw this thread, and I feel bad for not being introduced to this earlier. There is a problem with a .getCoalescedEvents() API that it bundles N input events together from between the last actual sent event to the current event. This means that all events that fell "in between" will suffer from an added latency, even when the application wanted to opt out and receive input events at a high refresh rate/low latency.

Another annoying thing is that in order to refactor an application to opt in to high refresh rate events, one may need to refactor its input handling to start firing N internal events for each actual received input Event, one for each element in the .getCoalescedEvents() list. This may be clumsy for some.

Would it be possible to change the API of .getCoalescedEvents() to instead be a foo.addEventListener('rapidmousemove') (high refresh rate/low latency) rather than a foo.addEventListener('mousemove')? (or just add a new API in parallel, if this already shipped?) That way there would not exist a latency impact for applications that do need the high refresh rate, and applications would have much easier time to opting in to high refresh rate input - just add five characters more to code, rather than to refactor each fired input Event to internally route to N mouse move events?
LGTM3


lgtm2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.




--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw9SybU70a4uM6WsevFzJgrB7JfqRqcCXYh%3DJkkv6Xpi%2Bg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

Dave Tapuska

unread,
Aug 28, 2017, 10:53:59 AM8/28/17
to juj...@gmail.com, blink-dev, Jochen Eisinger, Chris Harrelson, Rick Byers
This is being discussed in the pointer events working group. I'd love to hear what application you have that doesn't work based on the document lifecycle. We've had a few ideas about low latency canvas.

One thought is that you might have a legacy application that needs individual events to do some type of drawing based on individual events but that is still related to the document lifecycle (so you shouldn't experience any visual output latency to the user). And you'd actually be more performant if you handled all the events at one but again it might be logically difficult for you to adapt to that.


To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
 To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/36869216-2995-40d2-9171-f8926725b42c%40chromium.org.

Jukka Jylänki

unread,
Aug 28, 2017, 10:58:43 AM8/28/17
to Dave Tapuska, blink-dev, Jochen Eisinger, Chris Harrelson, Rick Byers
Thanks for the quick reply and link! The main use case I am thinking
is much the same as in https://github.com/whatwg/html/issues/2659.
Wrote some thoughts to Firefox bugzilla about this:
https://bugzilla.mozilla.org/show_bug.cgi?id=1392876#c7
>> email to blink-dev+...@chromium.org.
Reply all
Reply to author
Forward
0 new messages