Intent to Ship: EditContext API

[Daniel Clark]

Contact emails

dan...@microsoft.com, sni...@microsoft.com, shi...@microsoft.com

Explainer

https://github.com/w3c/edit-context/blob/gh-pages/explainer.md

Specification

https://w3c.github.io/edit-context

Design docs

https://github.com/w3c/edit-context/blob/gh-pages/dev-design.md

Summary

The EditContext API simplifies the process of integrating a web app with advanced text input methods such as IME Compositions and speech recognition, and unlocks new capabilities for web-based editors.

 

Blink component

Blink>Editing

Search tags

editing, contenteditable, input, rawinput, ime

TAG review

Completed (Resolution: satisifed) at https://github.com/w3ctag/design-reviews/issues/416

TAG review status

Issues addressed

Chromium Trial Name

EditContext

Link to origin trial feedback summary

https://github.com/w3c/edit-context/

Origin Trial documentation link

https://github.com/w3c/edit-context/blob/gh-pages/explainer.md

In the Origin Trial the Google Docs team used EditContext to receive IME input and position the IME window for Docs, replacing the current approach of manually positioning a hidden contenteditable element over the document when composing text. The new EditContext approach is more performant and supports a wider range of IME interactions.

 

We received similar feedback from Adobe, who are also using EditContext to replace a hidden text input element for triggering the IME.

 

Risks

Interoperability and Compatibility

There are no known interop or compat risks.

Gecko: Under consideration (https://github.com/mozilla/standards-positions/issues/199)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/243)

Web developers: Strongly positive Positive feedback from Word online, Adobe and Figma, Google Docs

Other signals:

Ergonomics

None.

Activation

Developers interested in this feature will typically have their own polyfill for text input using hidden textarea or contenteditable elements. Feature detecting and using new API to avoid side effects of previous approaches is intended to be easily adoptable.

Security

No particular security risks. See https://github.com/w3c/edit-context/blob/gh-pages/security-privacy.md.

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None.

Debuggability

Existing DevTools features should be sufficient for debugging EditContext.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes. This is a core web platform feature that is not limited to any particular underlying platform.

Is this feature fully tested by web-platform-tests?

Yes.

Tests are available at https://wpt.fyi/results/editing/edit-context?label=experimental&label=master&aligned Note that some composition scenarios are not yet testable in WPT due to a dependency on content_shell-only test APIs. Work is underway to add functionality for mocking IME input in WPTs such that these tests can be moved to WPT.

Flag name on chrome://flags

edit-context

Finch feature name

EditContext

Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=999184

Measurement

The UseCounter WebFeature::kEditContext tracks instantiation of EditContext.

Availability expectation

We expect other browser vendors to be interested in implementing this feature, though we cannot comment on specific timelines.

Adoption expectation

Feature will be used by Google Docs upon launch in Chrome.

Adoption plan

We are already working with the Docs team as a partner in the feature's Origin Trial, where they have implemented composition using EditContext.

Non-OSS dependencies

Does the feature depend on any code or APIs outside the Chromium open source repository and its open-source dependencies to function?

None.

Estimated milestones

Shipping on desktop	121
OriginTrial desktop last	120
OriginTrial desktop first	116

 

OriginTrial Android last	120
OriginTrial Android first	116

 

OriginTrial webView last	120
OriginTrial webView first	116

 

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

Open spec issues can be found here: https://github.com/w3c/edit-context/issues We expect these issues to be resolved in a forward-compatible way and/or to only affect rare corner-cases. Many of these discuss potential additions to the feature that will be considered based on ongoing developer feedback as EditContext is adopted more widely.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5041440373604352

Links to previous Intent discussions

Intent to Implement: https://groups.google.com/u/1/a/chromium.org/g/blink-dev/c/OHqvPx9mFww/m/1za_qdEHDwAJ

Intent to Experiment: https://groups.google.com/u/1/a/chromium.org/g/blink-dev/c/QZQrESwcK3o/m/k3pfYBcRBAAJ 

[Gregg Tavares]

Not a decider but one that sees the IME on many sites that try to roll their own text input.

This sounds like a "if you do all of these 30 things perfectly, then maybe your site will work with most IME issues but you won't know unless you get someone experienced with IME users to test for you" solution

Vs. some other solution which is "do nothing and it just works".  The current "do nothing and it just works" is, use <input type="text"> or <textarea> or contenteditable.

Is this API just giving developers lots of rope to hang themselves?

Also, how does it align with other browsers? For example the explainer shows a sequence of events

Event	EventTarget	key code	event.text
keydown	focused element	'S'
compositionstart	active EditContext
textupdate	active EditContext		'S'
textformatupdate	active EditContext
keyup	focused element	'S'
keydown	focused element	'U'
textupdate	active EditContext		'す'
textformatupdate	active EditContext
keyup	focused element	'U'
keydown	focused element	'Space'
textupdate	active EditContext		'巣'
textformatupdate	active EditContext
compositionend	active EditContext
keyup	focused element	'Space'

That seems non-intuitive to me. I get a keydown first, at which point my app reacts when it wasn't supposed to as the key was meant for the IME. 

Also, this table doesn't seem to match Firefox for example, pressing 's' when in Japanese input mode pops up the IME and in firefox it produces event.key = 'Process', not event.key = 's' which at least makes more sense since the input should be going to the IME, not the page.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/MW4PR00MB1654118011BA087BA5058967C5A1A%40MW4PR00MB1654.namprd00.prod.outlook.com.

[Daniel Clark]

EditContext is not meant to be an interchangeable replacement for <input type=”text”>, contenteditable, etc, and most sites that want to receive simple text input will want to continue using the existing set of editing features.

 

The target user of EditContext is one that has already reimplemented a lot of the editing stack, such that the browser’s built-in editing functionality is more of a hindrance than a help – the typical case here is something like Google Docs (where the entire editor view is reimplemented in a <canvas>). EditContext replaces hacks that sites like these often have to resort to such as hidden contenteditable elements that are floated around the page to position the IME window.

 

A site that just wants to receive text input without building out their own fully-featured editing experience can and should continue using the existing “batteries-included” tools like <textarea> or contenteditable.

 

The keydown event coming before compositionstart seems to be consistent with the existing contenteditable behavior in both Chromium and Firefox.  While EditContext changes how some editing-related events are fired, some of the existing orderings like this were left in place for consistency’s sake when there wasn’t a strong reason to change them.

 

The keydownevent.key interop difference is a good one to note, but I think it should be resolved orthogonally to EditContext. Since that behavior difference is present for both EditContext and contenteditable, the ideal outcome would be to bring this behavior in line across browsers for all editable fields.  It looks like there are some stale issues in the EditingWG in that area, e.g. this one from before Gecko started firing keydown/keyup events during composition; maybe this should be taken back up by the WG to try to drive further interoperability in the area. If we end up making a change there it would apply both to EditContext and to other types of editable fields.

 

-- Dan

 

From: Gregg Tavares <gm...@chromium.org>
Sent: Monday, October 30, 2023 10:19 PM
To: Daniel Clark <dan...@microsoft.com>
Cc: blink-dev <blin...@chromium.org>; Alex Keng <shi...@microsoft.com>; Anupam Snigdha <sni...@microsoft.com>; ko...@chromium.org
Subject: Re: [blink-dev] Intent to Ship: EditContext API

 

You don't often get email from gm...@chromium.org. Learn why this is important

[Gregg Tavares]

I don't think the keydown issue should be ignored until after shipping this API. I think it's essential for this API to actually function.

Consider an app that responds to Control-Shift-J for say "jump to definition". 

keydown = key = 'Control'

keydown = key = 'Shift' (CtrlKey = true)

keydown = key = 'K' (CtrlKey = true, Shift = true)  And the app would trigger it's jump-to-definition command  But what should have happened is the IME switched to Japanese mode (on Mac) and nothing happened in the app.

It seems like this API is supposed to solve these issues but if keydown happens first then it seems like it fails at it's actual goal.

The same issue but for different keys exists because of the difference in Firefox vs Chrome where in Chrome the app will mistakenly respond to keys it shouldn't and in Firefox it won't since Firefox hides those keys as 'Process'.

Other questions:  

(*) Are we sure this API matches all platforms? It appears to be only implemented on Windows so far. The concern being, without implementing on other platforms before shipping, we may run into design issues that require changes to the API. I'm not remotely an expert in IME APIs but I know in my own domain, GPUs, if we didn't actually implement across APIs we'd have that issue. 

I wanted to test some things but I don't currently have access to a windows device and the sample didn't seem to work on my Mac with the EditContext API enabled.

(*) How is recoversion handled in the canvas example? In a normal text editing area the user can select a portion of the text and pick "reconvert" either from menus or via hotkeys 

 

[Daniel Clark]

To the extent that firing keydown before compositionstart is a problem for apps, that’s an issue that equally affects contenteditable. Changing the order may be a good improvement, but it will be a breaking change for the web that should be carefully considered and hopefully done in tandem with Firefox and Webkit so that browsers can arrive at matching behavior. I think tying EditContext to this is not necessary, as EditContext aims to solve a variety of problems around receiving text input aside from how keydown specifically is handled. Since EditContext does not make any changes around the order of keydown relative to other events, shipping EditContext should not add any additional complications to changing that event order if the Editing Working group decides to go that route.

 

The API is supported on Mac as well. If the sample you tried didn’t work it may be because we failed to update it in response to breaking changes. This sample is up-to-date and should serve to show simple text input and composition: https://magic-organic-yam.glitch.me/. Note that a lot of basic editing functionality isn’t implemented here, and the text formatting for compositions is with highlights rather than underlines -- this was built for testing the API, not real-world editing.

 

As part of the Origin Trial, developers at Google Docs and Adobe included Mac in their testing and reported several Mac-specific issues that we’ve since fixed. If you’d like to try with the Docs implementation, let me know and I can ask about getting you opted in to that rollout.

 

For reconversions, when a page is using EditContext the same UI will still be available to the user via the menu or hotkeys. The page keeps the platform informed about which text is selected by the user by calling EditContext.updateSelection(). When the user triggers a reconversion, Blink will send the page another TextUpdateEvent indicating to it that it should change the text in the specified range to the new value.

[Rick Byers]

It certainly sounds to me that there's additional work to be done here, but I agree with Daniel that changing event order is not something we can do lightly. Gregg, could you please file an issue on the spec so the discussion can continue there?

Given the positive feedback we have from major real-world deployments in OT and the 3+ years of design work that's gone into this API via the editing WG, I personally don't think this concern is sufficient to block shipping at this stage. Editing is going to continue to be an area to work on improving rationality and interop and shipping this API now seems like the right pragmatic next step to me. Since fundamentally fixing this issue is going to need breaking changes anyway, and EditContext is most likely to be eagerly adopted just by a few large applications, I'm reasonably confident that we can continue to improve over time as driven by real-world deployment experience. LGTM1

Rick

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH7PR00MB1637386167B3A830D5D06D0AC5A7A%40PH7PR00MB1637.namprd00.prod.outlook.com.

[Alex Russell]

+1 to the evidence from OT being persuasive.

LGTM2

Contact emails

dan...@microsoft.com, sni...@microsoft.com, shihken@microsoft.com

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/MW4PR00MB1654118011BA087BA5058967C5A1A%40MW4PR00MB1654.namprd00.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Mike Taylor]

LGTM3

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6707471d-651b-482d-9047-a5fb0f493680n%40chromium.org.

[Daniel Bratell]

LGTM4 (yes, a bonus LGTM since I felt ninja'd by Mike and I think this will be a nice addition to the web platform)

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/4c6eed01-7341-41d9-87ee-fae9876a4752%40chromium.org.

[Gregg Tavares]

Changing the event order seems like something you'd be opting into since this API has not shipped yet.

Don't use the API, get the existing order, Do use the API, get the more useful order.

I'm skeptical this API actually fixes the real issues. The examples are incomplete and therefore unconvincing (to me)

Maybe I can find a way into the docs trial and test. If I find issues will that change shipping vs not shipping? If it's going to ship regardless of what I find then there's no reason for me to check.

Contact emails

dan...@microsoft.com, sni...@microsoft.com, shi...@microsoft.com

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/MW4PR00MB1654118011BA087BA5058967C5A1A%40MW4PR00MB1654.namprd00.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH7PR00MB1637386167B3A830D5D06D0AC5A7A%40PH7PR00MB1637.namprd00.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6707471d-651b-482d-9047-a5fb0f493680n%40chromium.org.

[Rick Byers]

On Wed, Nov 8, 2023 at 5:56 PM Gregg Tavares <gm...@chromium.org> wrote:

Changing the event order seems like something you'd be opting into since this API has not shipped yet.

Don't use the API, get the existing order, Do use the API, get the more useful order.

Web compat is almost always more complicated than that. Many real websites are a collection of scripts from a variety of authors and often served by different origins (eg. think of RUM analytics providers). So it would be super weird and confusing if simply using a new API in one script caused DOM event order to change for all other scripts on the page (and what about 3p iframes?). I'd expect this to cause adoption-blocking issues for EditContext since anyone starting to use the API couldn't reason about how doing so might break unrelated scripts on the same page. More pragmatic, I think, would be a document policy a page can opt into which changes the event order, not coupled to any other API. That could potentially be a good thing to do independently.

I'm skeptical this API actually fixes the real issues. The examples are incomplete and therefore unconvincing (to me)

Maybe I can find a way into the docs trial and test. If I find issues will that change shipping vs not shipping? If it's going to ship regardless of what I find then there's no reason for me to check.

Given that this feature is designed for major platforms like MS Office, Google Docs and Adobe tools, I think it's their feedback which I'd want to most rely on for a shipping decision. I certainly don't trust my own understanding of this API over that of experts working on products like that. Dan, do you have public OT feedback somewhere? The OT feedback link in the e-mail is just to the repo. Or maybe we could take this off list and reach out to one of our partners to ask specifically how they feel about this concern? Regardless, we should be tracking the debate in detail on a github issue, and coming back to blink-dev with a summary when it's concluded.

[Ian Kilpatrick]

Providing a little more context for the benefit of this API - the complexity that editors (Google Docs, Adobe tools, MS Office, etc) have to deal with for providing a good user experience is substantial and multiple engineer years worth of work to get close to correct. They typically have to create a hidden contenteditable element then trap + mutate all sorts of inputs to correctly (across different OSes, user locales, user input methods) to provide an experience inline with what the user expects - then render all that to an entirely different surface (canvas, svg, parallel dom, etc). Even with this amount of effort there are typically lots of issues/bugs due to this approach. This API reduces the complexity this style of editors have to deal with substantially.

I personally hope we'll see more high quality editors from smaller teams as a result of this API given for most web developers providing a good experience to most users is an insurmountable task.

Ian

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY_pZEbTFxFCSMO_zUJ8%2B3mrGT5zH%2B-wg1kNF-VjBS5BHA%40mail.gmail.com.

[Daniel Clark]

I filed an issue here to track the keydown/compositionstart ordering question: https://github.com/w3c/uievents/issues/360.

 

We’ve received the OT feedback over email so I don’t have a public link to share. Aside from specific bugs which we fixed, the feedback has been positive; Docs and Adobe were able to use EditContext to replace hidden contenteditable element hacks during composition. I’ll ask them to weigh in on whether this event ordering question has been a problem for them.

 

-- Dan

 

From: Ian Kilpatrick <ikilp...@chromium.org>
Sent: Thursday, November 9, 2023 10:07 AM
To: Rick Byers <rby...@chromium.org>
Cc: Gregg Tavares <gm...@chromium.org>; Daniel Bratell <brat...@gmail.com>; Mike Taylor <mike...@chromium.org>; Alex Russell <sligh...@chromium.org>; blink-dev <blin...@chromium.org>; Alex Keng <shi...@microsoft.com>; Anupam Snigdha <sni...@microsoft.com>; Koji Ishii <ko...@chromium.org>; Daniel Clark <dan...@microsoft.com>
Subject: Re: [blink-dev] Intent to Ship: EditContext API

 

You don't often get email from ikilp...@chromium.org. Learn why this is important

[Yordan Yordanov]

dictionary TextUpdateEventInit {
    unsigned long updateRangeStart;
    unsigned long updateRangeEnd;
    DOMString text;
    unsigned long selectionStart;
    unsigned long selectionEnd;
};

[Exposed=Window]
interface TextUpdateEvent : Event {
    constructor(optional TextUpdateEventInit options = {});
    readonly attribute unsigned long updateRangeStart;
    readonly attribute unsigned long updateRangeEnd;
    readonly attribute DOMString text;
    readonly attribute unsigned long selectionStart;
    readonly attribute unsigned long selectionEnd;
};

dictionary TextFormatUpdateEventInit {
    unsigned long formatRangeStart;
    unsigned long formatRangeEnd;
    DOMString underlineColor;
    DOMString backgroundColor;
    DOMString suggestionHighlightColor;
    DOMString textColor;
    DOMString underlineThickness;
    DOMString underlineStyle;
};

[Exposed=Window]
interface TextFormatUpdateEvent : Event {
    constructor(optional TextFormatUpdateEventInit options = {});
    readonly attribute unsigned long formatRangeStart;
    readonly attribute unsigned long formatRangeEnd;
    readonly attribute DOMString underlineColor;
    readonly attribute DOMString backgroundColor;
    readonly attribute DOMString suggestionHighlightColor;
    readonly attribute DOMString textColor;
    readonly attribute DOMString underlineThickness;
    readonly attribute DOMString underlineStyle;
};

dictionary EditContextInit {
    DOMString text;
    unsigned long selectionStart;
    unsigned long selectionEnd;
};

/// @event name="textupdate", type="TextUpdateEvent"
/// @event name="textformatupdate", type="TextFormatUpdateEvent"
/// @event name="compositionstart", type="CompositionEvent"
/// @event name="compositionend", type="CompositionEvent"
[Exposed=Window]
interface EditContext : EventTarget {
    constructor(optional EditContextInit options = {});

    void updateSelection(unsigned long start, unsigned long end);
    void updateBounds(DOMRect controlBounds, DOMRect selectionBounds);
    void updateText(unsigned long start, unsigned long end, DOMString newText);

    attribute DOMString text;
    attribute unsigned long selectionStart;
    attribute unsigned long selectionEnd;

    // Event handler attributes
    attribute EventHandler ontextupdate;
    attribute EventHandler ontextformatupdate;
    attribute EventHandler oncompositionstart;
    attribute EventHandler oncompositionend;
};

[Gregg Tavares]

On Fri, Nov 10, 2023 at 1:31 AM Rick Byers <rby...@chromium.org> wrote:

On Wed, Nov 8, 2023 at 5:56 PM Gregg Tavares <gm...@chromium.org> wrote:

Changing the event order seems like something you'd be opting into since this API has not shipped yet.

Don't use the API, get the existing order, Do use the API, get the more useful order.

Web compat is almost always more complicated than that. Many real websites are a collection of scripts from a variety of authors and often served by different origins (eg. think of RUM analytics providers). So it would be super weird and confusing if simply using a new API in one script caused DOM event order to change for all other scripts on the page (and what about 3p iframes?). I'd expect this to cause adoption-blocking issues for EditContext since anyone starting to use the API couldn't reason about how doing so might break unrelated scripts on the same page. More pragmatic, I think, would be a document policy a page can opt into which changes the event order, not coupled to any other API. That could potentially be a good thing to do independently.

It seems like opting into the EditContext API could also solve this issue by adding new events. A page using EditContext API would know to ignore keydown/keyup and use the new events and old libraries wouldn't see them. 

 

I'm skeptical this API actually fixes the real issues. The examples are incomplete and therefore unconvincing (to me)

Maybe I can find a way into the docs trial and test. If I find issues will that change shipping vs not shipping? If it's going to ship regardless of what I find then there's no reason for me to check.

Given that this feature is designed for major platforms like MS Office, Google Docs and Adobe tools, I think it's their feedback which I'd want to most rely on for a shipping decision. I certainly don't trust my own understanding of this API over that of experts working on products like that. Dan, do you have public OT feedback somewhere? The OT feedback link in the e-mail is just to the repo. Or maybe we could take this off list and reach out to one of our partners to ask specifically how they feel about this concern? Regardless, we should be tracking the debate in detail on a github issue, and coming back to blink-dev with a summary when it's concluded.

Unfortunately I have less confidence in those major platforms. I know this API is meant to fix their issues but IMO they would never have shipped if they were aware of the issues in the first place. Google Docs does not support emoji and Japanese input, and probably all IME input, has been broken since Canvas editing shipped. I could be wrong but, it seems likely the people in charge were not aware of the issues and blindly forged ahead because breaking the product for 100s of millions of users doesn't seem like a choice anyone would have made deliberately. So that's arguably evidence they are not experts in this issue. I'm not saying I am an expert either, but I am at least aware enough to be able to test some of it.

[Igor Kopylov]

Speaking for Google Docs, we are working right now on using EditContext to fix these kinds of IME input problems. From the experience so far, it makes our code simpler and addresses a number of longstanding issues, like emoji and accent input. I’m certainly happy to follow up on specific issues. Regarding canvas rendering - that launch didn’t change anything about how text input works. As Ian mentioned, text input has been going to an offscreen element separate from the rendered view, at least since the current version of Google Docs launched in 2010. IME support has been a challenge that whole time, and we see EditContext as a way to finally address those issues directly. Again, happy to follow up off-thread about specific IME issues and to see if they are getting addressed.

-Igor