Intent to Ship: Handwriting Recognition API

221 views
Skip to first unread message

Jiewei Qian

unread,
Dec 8, 2021, 9:57:40 PM12/8/21
to blink-dev, gle...@chromium.org

Contact emails

q...@chromium.org, hong...@chromium.org, mgi...@chromium.org


Explainer

https://github.com/WICG/handwriting-recognition/blob/main/explainer.md


Specification

https://wicg.github.io/handwriting-recognition/ [in review]


Design docs


https://github.com/WICG/handwriting-recognition/blob/main/explainer.md


Summary

An API for web applications to make use of advanced handwriting recognition services (e.g. those on operating systems) to recognize text from handwriting drawings (inks) in real time. In this context, handwriting drawing means the temporal and positional information used to describe a human handwriting process.



Blink component

Blink>Handwriting


TAG review

https://github.com/w3ctag/design-reviews/issues/591


TAG review status

Issues addressed


Link to origin trial feedback summary

https://docs.google.com/document/d/1BGd3WiwiqBA1JQaG-r0bc5No0f0KkqW13EhO1YeQocA/edit


Risks



Interoperability and Compatibility

Different browsers (and operating systems) will expose different underlying implementations and produce different outputs given the same input. We think this is acceptable given the implementation is machine-learning based, and it is not feasible or desirable to precisely specify the expected output for a given input in a standard. In the past, we have taken this approach for the Web Speech and Shape Detection APIs.



Gecko: No signal (https://github.com/mozilla/standards-positions/issues/507) Feedback requested, no reply yet.


WebKit: No signal (https://lists.webkit.org/pipermail/webkit-dev/2021-March/031762.html) Feedback requested, no reply yet.


Web developers: Positive (https://github.com/WICG/handwriting-recognition/issues/21) Twitter: https://twitter.com/ChromiumDev/status/1428640785810051074


Other signals:



Debuggability

N/A. Existing DevTools tooling works for this feature.



Is this feature fully tested by web-platform-tests?

No. Full coverage relies on platform-specific implementation. WPT only tests failure cases.


Flag name



Requires code in //chrome?

False


Tracking bug

https://crbug.com/1207667


Launch bug

https://crbug.com/1206121


Estimated milestones

OriginTrial desktop last

97

OriginTrial desktop first

91





Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5263213807534080


Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/VXUq1UY4m7Y

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/o8RqlFwZItQ/m/vgHLF_pHBQAJ



This intent message was generated by Chrome Platform Status.


Glen Robertson

unread,
Dec 9, 2021, 12:07:09 AM12/9/21
to Jiewei Qian, blink-dev
Given that the Handwriting Recognition OT ends with M97, and M98 branches today, it seems like it will be difficult to ship in M98 - a merge request to enable the feature is unlikely to be approved.

Blink Owners: Would it be acceptable to extend the OT for 1 additional milestone (end in M98) and ship in M99?

Chris Harrelson

unread,
Dec 10, 2021, 12:50:58 AM12/10/21
to Glen Robertson, Jiewei Qian, blink-dev
I think it's ok. LGTM. Thanks for doing your best to ship a high-quality implementation!

Note that this will be a total of 8 milestones, including M98, but as was mentioned in a previous intent, there was also a breaking API change in M95, plus evidence of experimental feedback.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPV%2BSg-5XE4cpDbwT69_T4u2CPN18O-9y_8nAW%2Bq-XYpwtjn3A%40mail.gmail.com.

Chris Harrelson

unread,
Dec 15, 2021, 11:57:53 AM12/15/21
to Glen Robertson, Jiewei Qian, blink-dev
Coming back to the intent-to-ship part: do you consider it ready for approval? Or are there more spec or other issues to iron out before we should consider approval?

Jiewei Qian

unread,
Dec 20, 2021, 11:38:42 PM12/20/21
to Chris Harrelson, Glen Robertson, blink-dev
The spec is currently being reviewed by the spec mentor. Otherwise we think the feature is ready for review.

Jiewei Qian

unread,
Dec 22, 2021, 8:02:28 PM12/22/21
to Chris Harrelson, Glen Robertson, blink-dev
Some context for security and privacy considerations (the reviewers suggest me including them here):
  • This intent is for shipping on Chrome OS.
  • We'll send separate I2S and security/privacy reviews for shipping on other platforms because their platform-specific recognizers might have different implications for privacy.
  • Chrome OS handwriting recognizers are stateless, and won't update themselves based on user behavior.
  • The list of available Chrome OS handwriting recognizers are hard-coded. So this API won't expose information about the user's language settings and preferences.

Chris Harrelson

unread,
Jan 5, 2022, 12:19:12 PMJan 5
to Jiewei Qian, Glen Robertson, blink-dev
LGTM1 from Alex came in a non-threaded message:

"LGTM.

Per conversation in the API OWNERs, it'd be great to have a list in this thread of the bits that will be closed-source and potentially diverge by platform. Thanks!"

LGTM2 from me, but please reply with the bits Alex mentioned thanks.

Yoav Weiss

unread,
Jan 5, 2022, 12:23:47 PMJan 5
to Chris Harrelson, Jiewei Qian, Glen Robertson, blink-dev

Jiewei Qian

unread,
Jan 6, 2022, 12:32:43 AMJan 6
to Yoav Weiss, Chris Harrelson, Glen Robertson, blink-dev

We expect the following to potentially be closed source:

  • The OS-specific implementation of the handwriting recognizer and its models. In other words, how to convert a list of points to some recognized text. Example: the implementation of Microsoft Windows Ink, and Windows Ink’s English recognizer model.


We expect the following to potentially diverge by platform:

  • The handwriting recognizer implementation being used (depending on availability). Example: Uses Windows Ink on Microsoft Windows, Use MLService on Chrome OS, use Apple PencilKit on macOS.

  • Language fallback rules (depending on OS-specific models). Example: If the OS model recognizes both “zh-Hans” and “zh-Hant” at the same time, the Web recognizer supports “zh” language tag. If the model only supports “zh-Hans”, the Web recognizer doesn’t support “zh” language tag (because “zh” should also cover “zh-Hant”).

  • Availability of languages (depending on availability of OS-specific models). Example: Windows Ink doesn’t support az-Latn (Arabic in Latin alphabet), but Google MLKit supports this language.


The factors contributing to divergence:

  • Recognizer models could be large (the model for a single language can exceed 20MB), thus impractical to ship them with the browser

  • Allowing the use of proprietary recognizers (which could outperform open-source ones)

  • Allowing browsers and OSs to compete on recognizer quality


Thanks all!
Reply all
Reply to author
Forward
0 new messages