Intent to Implement: PointerEvent.getPredictedEvents

[eir...@chromium.org]

Contact emails

eir...@chromium.org.com, nzol...@chromium.org

Explainer

https://docs.google.com/document/d/15NKd4s5BSEPXQZ1IJnm9zNzz7hltIRw0YOMlN5WoOzA/edit#

DemoPage: eirage.github.io/Prediction/test

Spec

https://w3c.github.io/pointerevents/extension.html#extensions-to-the-pointerevent-interface

Summary

An API to return a few(~3 for now) predicted events of the received events in javascript. This API enables apps like drawing app to draw the predicted line in advance, so users feel less latency.

Motivation

One of the existing gaps between native apps and web apps is the latency of the input. This is particularly more visible in drawing apps. A way to make users feel like the lines they are drawing appear in the realtime is to predict where user will draw next and they pre-draw that point as well. These APIs are provided by different platforms, and web platform is lacking such an API.

Interoperability and Compatibility Risk

This is a new API so there’ll be no compatibility change.

Edge: No signals

Firefox: No signals

Safari: No signals

Web developers: No signals

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/5765569655603200

Requesting approval to ship?

No

[Jochen Eisinger]

It looks like the predicted future events increase the information about past events as well. I was checking the spec whether there's a privacy consideration section, but didn't find any?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/56400959-294c-47d6-ac98-1e247908882d%40chromium.org.

[Ella Ge]

We are using the last few events to do the prediction, so presumably developer can do their own predictions base on the last event information (for example I believe google keep are doing prediction when drawing).

The only extra information I can think about is the very first event entering an iframe. The speed/direction might be exposed by predicted events (although it's not the case for our implementation as our prediction are done after oopif) I don't think this is a issue, because it's very unlikely the frame only get ONE event, and it can only calculate the speed/direction from predicted event.

[Rick Byers]

It may be worth noting that iOS, Windows and Android all have APIs for this already.

Ella, to what extent are we doing our own prediction, vs. exposing the underlying OS prediction where it's available? On the one hand, if we're just doing our own prediction based on past points, then presumably applications could also do that themselves (still some value in having it built-in of course). If we can, for some reason, actually do a better prediction than the app (such as by leveraging information in the OS / firmware not otherwise available to the app), then that's adding a new capability available to native apps.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADo_Rti9sciWKgY5nLAuAytJ1PohhtpkHvKhbthPHjJQTEtmrw%40mail.gmail.com.

[Jochen Eisinger]

sorry for not replying for so long.

From the explainer, it looked to me like we'd only expose events at certain points to the site (maybe aligned to rAF?), but use all events for the predictions, i.e. in a situation like below where the direction changed between two events, the prediction would give you more information than just extrapolating from the two reported events:

[Rick Byers]

We already ship the getCoalescedEvents API which directly exposes this additional information. So if there's a privacy concern here, I think it's for a previous intent, not this one ;-)

Rick

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALjhuieHmvp87QFdfCh1ynD%2Br2s8P%3D_FFymT33-0qH1mpvzTWA%40mail.gmail.com.

[Jochen Eisinger]

Ah, if those events are already exposed, then there's indeed no new info leaked.

So the only use case for this API is when the user moves the mouse or some other pointer from outside the tab to inside the tab?

[Ella Ge]

jochen@:  Actually the use case should be when user moving the mouse within the web page. 

Rick: right now we are doing our own prediction based on past points. There are longer term plan that we can use more information from the OS to do better prediction. but currently we just using very straight forward prediction algorithm.

We'd like to adding this functionality so that developer can take benefit from this addition api instead of writing their own.

[Jochen Eisinger]

Ok, that makes me wonder whether this is the right thing to invest time into? From a layered platform POV, all information is already available to the website. From an ecosystem POV, nobody signaled that they'd miss this kind of API?

[Navid Zolghadr]

Jochen, this API is most useful for drawing apps. We have some signals and requests for this already from some teams to make stylus drawing feel more like interacting with paper.

Regarding the quality of the prediction you are correct. If UA only uses the information that is available to the website itself the it is not adding much benefit. Although that is what we do right now as the first draft we will be taking advantage of things from the platform prediction if existed in our prediction models in the future as Rick pointed out. But how UA is implementing this API is their concern and the hope of this API is that UA prediction model will be using information more that what is exposed to the website already to make a better prediction.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALjhuidqUgSe90u-tUi8WR4BjrFdusDzYX9Hzw5WM8vDS3hVQQ%40mail.gmail.com.