Intent to ship: PointerEvent.getPredictedEvents API

[Ella Ge]

Contact emails

eir...@chromium.org.com, nzol...@chromium.org

Explainer

https://docs.google.com/document/d/15NKd4s5BSEPXQZ1IJnm9zNzz7hltIRw0YOMlN5WoOzA/edit#

Spec

https://w3c.github.io/pointerevents/extension.html#extensions-to-the-pointerevent-interface

Summary

An API to return a few predicted events of the received pointerevents. This API enables apps like drawing app to draw the predicted line in advance, so users feel less latency.

The finch result for the prediction accuracy shows we predict pretty good result in 95% and ok result in 99%.

Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/CgFyxYikn6A/a7U2FRqJCgAJ;context-place=forum/blink-dev

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Demo link

Try the following test pages on chrome canary with chrome://flags/#enable-experimental-web-platform-features turned on.

eirage.github.io/PredictionTest

eirage.github.io/PredictionDemo (A slow motion video from this test page, red line is the predicted curve)

Risks

Interoperability and Compatibility

This is a new API so there’ll be no compatibility change.

Edge: No signals

Firefox: No signals

Safari: No signals

Web developers: No signals

Requesting approval to ship?

Yes

Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

external/wpt/pointerevents/extension/pointerevent_predicted_events_attributes-manual.html is the added test. It’s manual test only now because there is no main thread event queue for layout tests (crbug.com/770028).

Entry on the feature dashboard

https://www.chromestatus.com/feature/5765569655603200

--

Ella Ge

[Daniel Bratell]

Hi! 

We (Chris, Yoav and I) just discussed this in the Blink API Owner meeting and I have a couple of questions.

A goal with the changes we do is that they move the web as a whole forward and then an important part is that we have input from the web standards community and other browser vendors. One of the missing steps here is the TAG review. Did anyone file for such a review or is it not needed for some reason?

Another part is that the intent says "no signals" from other vendors and from web developers. What kind of outreach has been done to solicit their feedback? I understood from Chris that the situation might be a bit better than it looks below?

/Daniel

https://www.chromium.org/blink/launching-features

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGbAJuHLrOwaH8VCg_j1WNF9w5Roo81JW_aJwiDa4L2MEY5P-w%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Bo Cupp]

> Another part is that the intent says "no signals" from other vendors and from web developers. What kind of outreach has been done to solicit their feedback?

With regards to the above, I'm a dev lead on the Edge team and have been following this work.  The pen as an input mechanism is important to Microsoft and some users are sensitive to the stroke lagging behind the pen.  Native apps have access to the Windows Ink stack which uses all sorts of cleverness to reduce latency, but the web is behind in this area.  To make the experience in web apps competitive with native apps we're interested in this specification and others...

• This predicted pointer event spec (to reduce percieved latency)
• Pointer events in workers combined with offscreen canvas (to improve responsives)
• Low latency canvas (to further reduce latency by bypassing the OS compositor)
• Raw pointer move events (to further reduce latency by drawing the latest input for the next frame to be presented)

In fairness for predicted pointer events if all the algorithms are doing is considering previous pointer events to make a prediction then this could be done in JS.  The benefit of the API as I see it is to provide additional insight from hardware when available or aligning with other prediction models used by the browser or OS components that are providing algorithms for prediction already.

TLDR; Edge is interested in shipping this API

Thanks!

Bo

Sent from Outlook

---

From: Daniel Bratell <bra...@opera.com>
Sent: Thursday, February 14, 2019 12:43 PM
To: blink-dev; Ella Ge
Cc: Navid Zolghadr; Bo Cupp
Subject: Re: [blink-dev] Intent to ship: PointerEvent.getPredictedEvents API

 

[rhal...@google.com]

Hi,

Did this feature have a privacy review?

[Daniel Bratell]

I don't think there is a need for a privacy review for this since it doesn't expose any new information. If I understand correctly, the predicted events *may* use extra information as the angle of the pen, the pressure or similar, but what it predicts will normally just be exposed as actual information a couple of milliseconds later.

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/ec810dcb-55ed-4b74-8e0b-331bbb1253e0%40chromium.org.

[rhal...@google.com]

OK, thanks. I agree.

[Alex Russell]

Hey all,

The OWNERS met today and this intent was discussed again. Would like to see a TAG review for this feature and to understand what efforts have been made to reach out to other vendors for input.

Thanks

[Ella Ge]

Hi Alex,

The TAG review is in progress: https://github.com/w3ctag/design-reviews/issues/346

For reach out to other vendors, 

MS has shown interest in this and has been following this work (see Bo's reply above);

Firefox has been working with us in the specification as well and they are also aware but there is still no timeline on their side for implementation

Thanks

[Alex Russell]

Thanks, Ella!

The background is very helpful. TAG review can take some time (which is one reason we now encourage folks to file in the I2I template, rather than at I2S). Given the difficult nature of assessing the fitness of the design today (no signals from web developers), have you considered running an Origin Trial? OT might allow you to make a strong case with developer feedback that this feature is important to apps.

Regards

[Ella Ge]

Hi all,

We finally get the approval from Tag review. We would like to bring up the this i2s again and request API owners' opinion on shipping this feature.

Thanks,

Ella

 

[Daniel Bratell]

LGTM1

Happy to see it coming through the, in this case, long process!

The tag review covers both this and pointerrawupdate so I'll update that thread too.

The documented state of support from WebKit and Mozilla is a bit thin, but maybe you can link to their implementation bugs or if they don't exist, file one each for them?

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADo_RtifTj6w5kg_L4Wh9J7tpsugj16_BRD7yHj6M9eqXV5CxA%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

[Chris Harrelson]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/op.z3zxwusurbppqq%40cicero2.linkoping.osa.

[Ella Ge]

On Wed, Jun 26, 2019 at 11:28 AM Chris Harrelson <chri...@chromium.org> wrote:

LGTM2

On Wed, Jun 26, 2019 at 7:44 AM Daniel Bratell <bra...@opera.com> wrote:

LGTM1

Happy to see it coming through the, in this case, long process!

The tag review covers both this and pointerrawupdate so I'll update that thread too.

The documented state of support from WebKit and Mozilla is a bit thin, but maybe you can link to their implementation bugs or if they don't exist, file one each for them?

 

Here is a bug for Mozilla https://bugzilla.mozilla.org/show_bug.cgi?id=1550461

[Ella Ge]

One more vote needed >.<

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADo_RtifTj6w5kg_L4Wh9J7tpsugj16_BRD7yHj6M9eqXV5CxA%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Yoav Weiss]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADo_RtifTj6w5kg_L4Wh9J7tpsugj16_BRD7yHj6M9eqXV5CxA%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/op.z3zxwusurbppqq%40cicero2.linkoping.osa.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0eb78acf-e74c-4512-9dce-02a876b84d9f%40chromium.org.