Intent to Experiment: Handwriting Recognition API

[Jiewei Qian]

Contact emails

q...@chromium.org, hong...@chromium.org, mgi...@chromium.org

Explainer

https://github.com/WICG/handwriting-recognition/blob/main/explainer.md

Specification

None

Design docs

https://github.com/WICG/handwriting-recognition/blob/main/explainer.md

Summary

An API for web applications to make use of advanced handwriting recognition services (e.g. those on operating systems) to recognize text from handwriting drawings (inks) in real time. In this context, handwriting drawing means the temporal and positional information used to describe a human handwriting process.

Blink component

Blink

TAG review

https://github.com/w3ctag/design-reviews/issues/591

TAG review status

Pending

Risks

Interoperability and Compatibility

Different browsers (and operating systems) will expose different underlying implementations and produce different outputs given the same input. We think this is acceptable given the implementation is machine-learning based, and it is not feasible or desirable to precisely specify the expected output for a given input in a standard. In the past, we have taken this approach for the Web Speech and Shape Detection APIs.

Gecko: No signal

WebKit: No signal

Web developers: No signals

Goals for experimentation

- General API design. Determine if the API design satisfies the App developer's use cases - Specifically, we want to see if the current API design provides sufficient information (in the result) to enable web developers to make use of this feature.

Reason this experiment is being extended

N/A

Ongoing technical constraints

None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

Initially we will implement this feature on Chrome OS, since it has handwriting recognition service ready for use. We will need to do additional work to integrate with other platforms. Handwriting recognition capabilities already exist on all other platforms. We will do other I2E / I2S for implementing this on other platforms, due to the different security and privacy implications.

Is this feature fully tested by web-platform-tests?

No

Tracking bug

https://crbug.com/1166910

Launch bug

https://crbug.com/1177374

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5263213807534080

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/VXUq1UY4m7Y

This intent message was generated by Chrome Platform Status.

[Yoav Weiss]

What is the timeline for experimentation?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANFkJJmGJVmkrO1xAeDNc9i9%2Bp5z%2BacpRPDAD8doxYnFGN37%3Dg%40mail.gmail.com.

[Jiewei Qian]

We plan to run origin tral from M91 to M94.

[Mike West]

LGTM to experiment from M91 through M94, with two notes:

* Please ask other vendors for signals regarding this API, as described in https://bit.ly/blink-signals.

* I'd like to understand the testing strategy here. Will you extend the web driver APIs to make it possible to mock the underlying platform's handwriting recognizer, for example?

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANFkJJkoOQX_bJ6p_cnuay3aoz0VL2Fyvi6VnS5e-Ar%2BGEZuaw%40mail.gmail.com.

[Jiewei Qian]

Hi Mike,

> Vendor signals

I just requested Gecko and WebKit for their signals.

> Test

We have a web test that mocks the Mojo interface for blink (https://chromium-review.googlesource.com/c/chromium/src/+/2758532).

For Chrome OS, we have a Tast test that exercises the entire code path (from JavaScript calls all the way to Chrome OS on-device handwriting recognizer). https://chromium-review.googlesource.com/c/chromiumos/platform/tast-tests/+/2751318 . This test uses collected handwriting data (English letter ABC) to verify the recognizer works reasonably (e.g. returns "ABC" as a prediction result).

We haven't decided on how to test this on other platforms yet. Ideally, we implement some form of end-to-end testing. Mocking platform specific API might not be a good idea. Maybe we will do the same thing (assuming the recognizer library is reasonable, and doesn't return strange values for an obvious / example handwriting).

[Jiewei Qian]

Hi,

Due to technical reasons, we plan to run an origin trial in M91 on a limited set of devices that can support it.

We plan to expand the origin trial coverage to all Chrome OS devices in M92, and run a Chrome-OS-wide origin trial from M92 to M94.