Intent to Implement and Ship: Fractional coordinates in MouseEvent

[Mustaq Ahmed]

Contact emails

mus...@chromium.org

Spec

CSSOM View Module extension to MouseEvents: https://drafts.csswg.org/cssom-view/#extensions-to-the-mouseevent-interface

For PointerEvent which inherits MouseEvent, the community group reached consensus here: https://github.com/w3c/pointerevents/issues/107

Summary

Update the types of all coordinates in MouseEvents from long to double to make the coordinates more precise for PointerEvents on high-DPI devices.

Motivation

We updated TouchEvent coordinates few years ago to report fractional CSS px units, to avoid possible loss in precision (caused by truncating to integers) particularly in modern mobile hardware where multiple device pixels correspond to one CSS px unit.  Around that time we deprioritized a similar update in MouseEvents because of the lack of a clear benefit:

 https://lists.w3.org/Archives/Public/www-style/2015Feb/0193.html

Now that we are shipping PointerEvents (in M55) which is speced through a healthy collaboration of most major browsers, the benefit to update MouseEvents is now obvious: PointerEvents would inherit the fractional coordinates in MouseEvents, and web developers would be able to migrate from TouchEvents to PointerEvents without any precision loss in coordinates.  The PointerEvents CG recently reached consensus about it.

Interoperability and Compatibility Risk

Since JavaScript represents both longs and doubles as Numbers, most existing code should work fine (and even show immediate smoothness benefits in typical coding patterns).  There is a slight risk of breakages for the following hypothetical use cases:

A. Code that relies on exact comparison of (integer) coordinates, or uses coordinates as array indices.

B. Code that constructs MouseEvents in JS with fractional coordinates yet expects truncated values (e.g. expects “new MouseEvent('mousemove', {clientX: 1.5}).clientX” to be 1 instead of 1.5).

If we encounter significant breakages, we can alternatively truncate the coordinates in all browser-fired MouseEvents to integers while maintaining fractional coordinates in all PointerEvents.  Edge currently follows this approach.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

OWP launch tracking bug

crbug.com/660057

Link to feature dashboard entry

https://www.chromestatus.com/feature/6169687914184704

Requesting approval to ship?

Yes.

[Vincent Scheib]

This should also hopefully increase resolution while under Pointer Lock, which was one of the earlier pointer lock requests that was difficult to practically satisfy.

[Rick Byers]

On Fri, Oct 28, 2016 at 3:34 PM, Vincent Scheib <sch...@chromium.org> wrote:

This should also hopefully increase resolution while under Pointer Lock, which was one of the earlier pointer lock requests that was difficult to practically satisfy.

On Fri, Oct 28, 2016 at 8:02 AM, Mustaq Ahmed <mus...@chromium.org> wrote:

Contact emails

mus...@chromium.org

Spec

CSSOM View Module extension to MouseEvents: https://drafts.csswg.org/cssom-view/#extensions-to-the-mouseevent-interface

For PointerEvent which inherits MouseEvent, the community group reached consensus here: https://github.com/w3c/pointerevents/issues/107

Summary

Update the types of all coordinates in MouseEvents from long to double to make the coordinates more precise for PointerEvents on high-DPI devices.

Motivation

We updated TouchEvent coordinates few years ago to report fractional CSS px units, to avoid possible loss in precision (caused by truncating to integers) particularly in modern mobile hardware where multiple device pixels correspond to one CSS px unit.  Around that time we deprioritized a similar update in MouseEvents because of the lack of a clear benefit:

 https://lists.w3.org/Archives/Public/www-style/2015Feb/0193.html

Now that we are shipping PointerEvents (in M55) which is speced through a healthy collaboration of most major browsers, the benefit to update MouseEvents is now obvious: PointerEvents would inherit the fractional coordinates in MouseEvents, and web developers would be able to migrate from TouchEvents to PointerEvents without any precision loss in coordinates.  The PointerEvents CG recently reached consensus about it.

I agree this is really important.  Some scenarios (eg. art, slow dragging of any slider, etc.) will show a regression in smoothness on mobile in M55 if they upgrade TouchEvent code to use PointerEvents instead.

Interoperability and Compatibility Risk

Since JavaScript represents both longs and doubles as Numbers, most existing code should work fine (and even show immediate smoothness benefits in typical coding patterns).  There is a slight risk of breakages for the following hypothetical use cases:

A. Code that relies on exact comparison of (integer) coordinates, or uses coordinates as array indices.

B. Code that constructs MouseEvents in JS with fractional coordinates yet expects truncated values (e.g. expects “new MouseEvent('mousemove', {clientX: 1.5}).clientX” to be 1 instead of 1.5).

If we encounter significant breakages, we can alternatively truncate the coordinates in all browser-fired MouseEvents to integers while maintaining fractional coordinates in all PointerEvents.  Edge currently follows this approach.

I suspect that you're going to need to do this.  There's a TON of code out there that deals with mouse events, and could be subtly broken.  For example, when making some CSS OM properties fractional we came across some sites that used size values as array indices. I could be wrong though - there's a chance we'd get lucky.  But I think the risk is high enough that if you want to try going all the way now, I think you should try to figure out some way to do some up-front compat analysis to try to quantify the level of breakage (eg. do some synthetic input on the top 100k sites using the telemetry cluster and look for exceptions that occur only when fractional co-ordinates are enabled).

If, however, you want to start by only permitting fractions to show up ONLY in these cases:

1) Mouse/Pointer/Wheel events created in JS where fractional values are supplied explicitly to a constructor (not init*Event methods)

2) Pointer events with pointerType=touch

Then I think the compat impact is likely low enough (and corresponding benefit high enough) that we should just give it a try and keep our eye out for reports of compat issues via the normal launch process.

So LGTM1 to ship from me if you decide to scope this intent down for now to not affect any browser-generated mouse*, click, wheel events.

[Rick Byers]

On Fri, Oct 28, 2016 at 3:34 PM, Vincent Scheib <sch...@chromium.org> wrote:

This should also hopefully increase resolution while under Pointer Lock, which was one of the earlier pointer lock requests that was difficult to practically satisfy.

BTW this totally sounds like something we should try once we've confirmed that the basic IDL change is web-compatible.  But again I'd suggest doing a separate intent / CL for it (since the compat risk and alternative options is really separate).  Worst case and that turns out not to be sufficiently web compatible, then perhaps we could add an option to the Pointer Lock API which explicitly opts into fractional co-ordinates.

[Majid Valipour]

Interoperability and Compatibility Risk

Since JavaScript represents both longs and doubles as Numbers, most existing code should work fine (and even show immediate smoothness benefits in typical coding patterns).  There is a slight risk of breakages for the following hypothetical use cases:

A. Code that relies on exact comparison of (integer) coordinates, or uses coordinates as array indices.

B. Code that constructs MouseEvents in JS with fractional coordinates yet expects truncated values (e.g. expects “new MouseEvent('mousemove', {clientX: 1.5}).clientX” to be 1 instead of 1.5).

Another class of breakages may come from any server-side logic that tries to parse these values as integer. I am not sure how prevalent this may be but something to keep in mind. 

Majid

[PhistucK]

On Wed, Nov 2, 2016 at 4:02 PM, Rick Byers <rby...@chromium.org> wrote:

2) Pointer events with pointerType=touch

Does Edge restrict fractional coordinates to pointerType=touch?

☆PhistucK

[Mustaq Ahmed]

No, PointerEvent coordinates in Edge are fractional for all pointer types.

[PhistucK]

So, Rick, why would you suggest that only pointerType=touch will have fractional coordinates?

☆PhistucK

[Rick Byers]

On Wed, Nov 2, 2016 at 2:29 PM, PhistucK <phis...@gmail.com> wrote:

So, Rick, why would you suggest that only pointerType=touch will have fractional coordinates?

Ah, good question.  That part of my argument is not really motivated by web compat concerns (which is what this intent should focus on) but implementation details.  It's actually a lot of work to plumb fractional co-ordinates through for mouse, and not nearly as important as the touch case (since high-dpi is more common with touch, and the smoothness of dragging is more psychologically important for direct-manipulation input devices).

I don't have any objection to working towards exposing fractional co-ordinates on PointerEvent for mouse input.  I'm fine including approval for that change in this intent, as long as we decouple the actual implementation work (don't risk holding up the touch scenario because the mouse scenario isn't done yet).

[PhistucK]

I would not want you to get to the MouseEvent state where it is very risky to convert to fractional, for the non-touch PointerEvent coordinates...

☆PhistucK

[Mustaq Ahmed]

As per the above discussion, the intent will include the following changes (and non-changes):

- PointerEvents of type touch: We will expose fractional coordinates in M56 since this would ease migrating from TouchEvents to PointerEvents for any use case that needs high-precision.

- PointerEvents of type mouse: We will defer fractional coordinates for now only because of the amount of work involved. We want to do it in future to match Edge but it's not our priority for M56.

- MouseEvents, WheelEvents, DragEvents, PointerLock: We will minimize risks by not changing anything for the events we fire, i.e., all fired events will have integer coordinates. (We will try to switch to fractional coordinates in future through a separate intent/CL plus some compat analysis as Rick suggested above. The only visible change will be the events (a) created in JS through event constructors & not through the init*() methods, AND (b) with fractional coordinates in *Init parameters---this is a rare case with very little risk.

Do I need anything else here? 

[Rick Byers]

On Thu, Nov 3, 2016 at 11:37 AM, Mustaq Ahmed <mus...@google.com> wrote:

As per the above discussion, the intent will include the following changes (and non-changes):

- PointerEvents of type touch: We will expose fractional coordinates in M56 since this would ease migrating from TouchEvents to PointerEvents for any use case that needs high-precision.

- PointerEvents of type mouse: We will defer fractional coordinates for now only because of the amount of work involved. We want to do it in future to match Edge but it's not our priority for M56.

- MouseEvents, WheelEvents, DragEvents, PointerLock: We will minimize risks by not changing anything for the events we fire, i.e., all fired events will have integer coordinates. (We will try to switch to fractional coordinates in future through a separate intent/CL plus some compat analysis as Rick suggested above. The only visible change will be the events (a) created in JS through event constructors & not through the init*() methods, AND (b) with fractional coordinates in *Init parameters---this is a rare case with very little risk. 

Do I need anything else here? 

Sounds perfect to me, thanks!  Still LGTM1

[Chris Harrelson]

LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Jochen Eisinger]

lgtm3

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.