Intent to Ship: Handwriting Recognition API

[Jiewei Qian]

Contact emails

q...@chromium.org, hong...@chromium.org, mgi...@chromium.org

Explainer

https://github.com/WICG/handwriting-recognition/blob/main/explainer.md

Specification

https://wicg.github.io/handwriting-recognition/ [in review]

Design docs

https://github.com/WICG/handwriting-recognition/blob/main/explainer.md

Summary

An API for web applications to make use of advanced handwriting recognition services (e.g. those on operating systems) to recognize text from handwriting drawings (inks) in real time. In this context, handwriting drawing means the temporal and positional information used to describe a human handwriting process.

Blink component

Blink>Handwriting

TAG review

https://github.com/w3ctag/design-reviews/issues/591

TAG review status

Issues addressed

Link to origin trial feedback summary

https://docs.google.com/document/d/1BGd3WiwiqBA1JQaG-r0bc5No0f0KkqW13EhO1YeQocA/edit

Risks

Interoperability and Compatibility

Different browsers (and operating systems) will expose different underlying implementations and produce different outputs given the same input. We think this is acceptable given the implementation is machine-learning based, and it is not feasible or desirable to precisely specify the expected output for a given input in a standard. In the past, we have taken this approach for the Web Speech and Shape Detection APIs.

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/507) Feedback requested, no reply yet.

WebKit: No signal (https://lists.webkit.org/pipermail/webkit-dev/2021-March/031762.html) Feedback requested, no reply yet.

Web developers: Positive (https://github.com/WICG/handwriting-recognition/issues/21) Twitter: https://twitter.com/ChromiumDev/status/1428640785810051074

Other signals:

Debuggability

N/A. Existing DevTools tooling works for this feature.

Is this feature fully tested by web-platform-tests?

No. Full coverage relies on platform-specific implementation. WPT only tests failure cases.

Flag name

Requires code in //chrome?

False

Tracking bug

https://crbug.com/1207667

Launch bug

https://crbug.com/1206121

Estimated milestones

OriginTrial desktop last	97
OriginTrial desktop first	91

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5263213807534080

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/VXUq1UY4m7Y

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/o8RqlFwZItQ/m/vgHLF_pHBQAJ

This intent message was generated by Chrome Platform Status.

[Glen Robertson]

Given that the Handwriting Recognition OT ends with M97, and M98 branches today, it seems like it will be difficult to ship in M98 - a merge request to enable the feature is unlikely to be approved.

Blink Owners: Would it be acceptable to extend the OT for 1 additional milestone (end in M98) and ship in M99?

[Chris Harrelson]

I think it's ok. LGTM. Thanks for doing your best to ship a high-quality implementation!

Note that this will be a total of 8 milestones, including M98, but as was mentioned in a previous intent, there was also a breaking API change in M95, plus evidence of experimental feedback.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPV%2BSg-5XE4cpDbwT69_T4u2CPN18O-9y_8nAW%2Bq-XYpwtjn3A%40mail.gmail.com.

[Chris Harrelson]

Coming back to the intent-to-ship part: do you consider it ready for approval? Or are there more spec or other issues to iron out before we should consider approval?

[Jiewei Qian]

The spec is currently being reviewed by the spec mentor. Otherwise we think the feature is ready for review.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw9XKeNj0Vznbv7bz_O%2BqvstWn89GOaO1xrzbe2XnDB8%2Bw%40mail.gmail.com.

[Jiewei Qian]

Some context for security and privacy considerations (the reviewers suggest me including them here):

• This intent is for shipping on Chrome OS.
• We'll send separate I2S and security/privacy reviews for shipping on other platforms because their platform-specific recognizers might have different implications for privacy.
• Chrome OS handwriting recognizers are stateless, and won't update themselves based on user behavior.
• The list of available Chrome OS handwriting recognizers are hard-coded. So this API won't expose information about the user's language settings and preferences.

[Chris Harrelson]

LGTM1 from Alex came in a non-threaded message:

"LGTM.

Per conversation in the API OWNERs, it'd be great to have a list in this thread of the bits that will be closed-source and potentially diverge by platform. Thanks!"

LGTM2 from me, but please reply with the bits Alex mentioned thanks.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANFkJJ%3DajkSxJ4XYiN%2BQ9viW8TsgSVjf09tPnrQh2DtGjkoEPw%40mail.gmail.com.

[Yoav Weiss]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_Z6%2BANqj2yV1gJtYMZXPd1GCixB%2BM4SijwMOH0t0diQQ%40mail.gmail.com.

[Jiewei Qian]

We expect the following to potentially be closed source:

• The OS-specific implementation of the handwriting recognizer and its models. In other words, how to convert a list of points to some recognized text. Example: the implementation of Microsoft Windows Ink, and Windows Ink’s English recognizer model.

We expect the following to potentially diverge by platform:

• The handwriting recognizer implementation being used (depending on availability). Example: Uses Windows Ink on Microsoft Windows, Use MLService on Chrome OS, use Apple PencilKit on macOS.

• Language fallback rules (depending on OS-specific models). Example: If the OS model recognizes both “zh-Hans” and “zh-Hant” at the same time, the Web recognizer supports “zh” language tag. If the model only supports “zh-Hans”, the Web recognizer doesn’t support “zh” language tag (because “zh” should also cover “zh-Hant”).

• Availability of languages (depending on availability of OS-specific models). Example: Windows Ink doesn’t support az-Latn (Arabic in Latin alphabet), but Google MLKit supports this language.

The factors contributing to divergence:

• Recognizer models could be large (the model for a single language can exceed 20MB), thus impractical to ship them with the browser

• Allowing the use of proprietary recognizers (which could outperform open-source ones)

• Allowing browsers and OSs to compete on recognizer quality

Thanks all!