Virtual Keyboard API proposals

[Mounir Lamouri]

Hi,

Vivien and I have been trying to design an API to allow writing a
virtual keyboard in HTML. This API is very different from the IME API
from Google [1] that aims to re-use the system's IME in a web page. We
want an API to implement the system IME as a web app.

** Naive approach **

We began to draft an API that was a naive approach trying to solve the
two technical issues we had:
- the VKB app wasn't able to know when the focus was changing in other
apps and what kind of element it was;
- the VKB app wasn't able to send trusted key events such as they are
considered by the other apps as user' inputs.

To fix that, we had the idea to create a navigator.vkb object that would
get 'focuschange' events when the focus would be change in any app. The
event would carry information like the element name, type, value, etc.
In addition, navigator.vkb would have a sendKey() method to send a key
event as trusted.

However, that solution wasn't enough because it is not handling
composition events [2] and we clearly need that. For some languages but
also for suggestions that you can see in some phone's virtual keyboard.
Also, that solution wasn't taking into account elements that had to be
handled without text typing like some input types (date, for example) or
select elements.

** Handling special cases **

It's actually quite easy to handle special cases, we just need specific
functions for them:
- .setValue(DOMString value)
This function takes in parameter a string that would be set to the
focused element if it's an input element. This has to be used for
special situations where the value had to be chosen amongst a list
(type=month) or a widget (type=date, time, etc.). If the value passed in
parameter isn't valid (in the term of HTML5 Forms Validation), the value
will simply be ignored by the input element.
- .setSelectedOption(long idx)
This function takes in parameter the option index that has to be
selected on the focused select element. If the focused element isn't a
select element, it will be ignored. As if the index isn't valid.
- .setSelectedOptions(Array<int> idxs)
Same as before but for select elements that allow multiple selected options.

** Solution 1: composition handled by the VKB **

The main issue with composition is when should it be started? Should we
start a composition when selecting the middle of a word, for example? As
far as we understood, there is no reason why we couldn't. So we thought
we could let the app decide whether or not begin a composition event.
To do that, we need to pass the selectionStart and selectionEnd values
to the VKB but also inform it when the selection changes inside the
currently focused element with a new event named 'selectionchange'. That
event could carry some information like element's value, new selection
values, etc.

Basically the selection handling could be done with three methods:
- .startComposition(int begin, int end)
Parameters will define the text that is affected by the composition.
Calling that method would fire 'compositionstart' with the appropriate
values in |data| to the focused element.
- .updateComposition(DOMString text)
This method should be used after calling startComposition() (calling it
in other situations would fire) and would update the text being
composed. For example, updateComposition('F') should be called if the
text field was empty and the user typed 'F'. updateComposition('Fo')
should be called if the user then typed 'o'. Calling that method would
send a 'compositionupdate' with the appropriate values in |data|, an
'input' event and maybe 'keydown', 'keypress' and 'keyup' events.
- .endComposition(DOMString text)
This method should be called when the composition has to stop. Whether
because a delimiter has been typed or because a suggested word has been
selected. Whatever the reason is, the text should be passed as a
parameter. It will send a 'compositionend' event with the appropriate
values in |data| and like for updateComposition() will fire other DOM
events.

In addition of those methods, we still need a sendKey() event that would
be used for delimiters, backspace and numbers but could also be used
when the VKB doesn't want to use composition. Calling sendKey() between
startComposition() and endComposition() calls would throw.

** Solution 2: composition handled by the platform **

This is very similar to solution 1 but without startComposition() and
endComposition(). Instead of those method, there would be
'compositionstart' and 'compositionend' events fired by the platform to
the VKB. When not in composition mode, the keyboard would have to call
sendKey() when a key would be pressed and, when informed that it should
now go to composition mode, updateComposition() would have to be used
exactly like in solution 1. The text being composed would be passed in
the 'compositionstart' event.
With that solution, sendKey() could whether create a new composition or
not, depending on the platform's decision but calling sendKey() between
'compositionstart' and 'compositionend' events would throw an exception.

** Feedback **

We are waiting for feedback to know which solution should be preferred
if any sound sensible.
Personally, I prefer the first one because it's more flexible and let
the VKB app to really be implemented as the developer wants.

[1] http://dvcs.w3.org/hg/ime-api/raw-file/default/Overview.html
[2]
http://dev.w3.org/2006/webapi/DOM-Level-3-Events/html/DOM3-Events.html#events-compositionevents

Cheers,
--
Mounir

[JOSE MANUEL CANTERA FONSECA]

El 25/04/12 21:42, "Mounir Lamouri" <mou...@lamouri.fr> escribió:

>Hi,
>
>Vivien and I have been trying to design an API to allow writing a
>virtual keyboard in HTML. This API is very different from the IME API
>from Google [1] that aims to re-use the system's IME in a web page. We
>want an API to implement the system IME as a web app.
>
>** Naive approach **
>
>We began to draft an API that was a naive approach trying to solve the
>two technical issues we had:
>- the VKB app wasn't able to know when the focus was changing in other
>apps and what kind of element it was;
>- the VKB app wasn't able to send trusted key events such as they are
>considered by the other apps as user' inputs.
>
>To fix that, we had the idea to create a navigator.vkb object that would
>get 'focuschange' events when the focus would be change in any app. The
>event would carry information like the element name, type, value, etc.
>In addition, navigator.vkb would have a sendKey() method to send a key
>event as trusted.

Today we were talking to Vivien about a typical use case of virtual
keyboards, which consists on a pair of buttons that allow to put the focus
on the next / previous element in a form (see for instance the Google
Accounts login form in a browser). Thus, probably we would need another
method on navigator.vkb to put the focus on the next element (in
accordance to 'tabindex', for example).

In addition there is also the <go> key that I think is similar to pressing
the enter key, thus probably that case can be solved by using the sendKey
method.

>dev-webapi mailing list
>dev-w...@lists.mozilla.org
>https://lists.mozilla.org/listinfo/dev-webapi
>



Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

[Philipp von Weitershausen]

On Wed, Apr 25, 2012 at 12:42 PM, Mounir Lamouri <mou...@lamouri.fr> wrote:
> ** Solution 1: composition handled by the VKB **
>
> The main issue with composition is when should it be started? Should we
> start a composition when selecting the middle of a word, for example?

The trivial answer is "yes", I think, but I think the question is a bit flawed.

As far as I understand Composition Events, they are sent whenever text
composition is going on. If composition happens through a traditional
keyboard, they are sent along with the keyboard events, e.g. for dead
keys or compound letters [1]. This would apply anywhere, including the
middle of the word.

[1] http://dev.w3.org/2006/webapi/DOM-Level-3-Events/html/DOM3-Events.html#keys-DeadKeys

> As far
> as we understood, there is no reason why we couldn't. So we thought we could
> let the app decide whether or not begin a composition event.

> To do that, we need to pass the selectionStart and selectionEnd values to
> the VKB but also inform it when the selection changes inside the currently
> focused element with a new event named 'selectionchange'. That event could
> carry some information like element's value, new selection values, etc.
>
> Basically the selection handling could be done with three methods:
>  - .startComposition(int begin, int end)
> Parameters will define the text that is affected by the composition. Calling
> that method would fire 'compositionstart' with the appropriate values in
> |data| to the focused element.

I assume with "appropriate values" you're referring to this excerpt
from the spec:

"Some implemenations may populate the data attribute of the compositionstart
event with the text currently selected in the document (for editing and
replacement); otherwise, the value of the data attribute must be the
empty string."

Is that what |begin| and |end| are for? I guess the keyboard will know
from the Selection Events.

>  - .updateComposition(DOMString text)
> This method should be used after calling startComposition() (calling it in
> other situations would fire) and would update the text being composed. For
> example, updateComposition('F') should be called if the text field was empty
> and the user typed 'F'. updateComposition('Fo') should be called if the user
> then typed 'o'. Calling that method would send a 'compositionupdate' with
> the appropriate values in |data|, an 'input' event and maybe 'keydown',
> 'keypress' and 'keyup' events.

My initial reaction here is that this is a bit backwards. With a
regular keyboard, it works like this: a key press triggers Keyboard
Events, and depending on the key clicked, etc., Composition Events are
dispatched as well. I think we should retain this when we're
virtualizing a keyboard. IOW, have a method for "compositeSendKey()"
as a short-circuit for when a traditional "key" is pressed.

I guess what I'm saying is: we should be very explicit about when we
virtualize a keyboard. The VKB should have to perform the exact steps
required by the spec (emulate key presses, send Composition Events,
etc.). Emulating key presses after the fact will be messy (and lying!)
when we have non-keyboard-based input methods (see below.)

> ** Solution 2: composition handled by the platform **
>
> This is very similar to solution 1 but without startComposition() and
> endComposition(). Instead of those method, there would be 'compositionstart'
> and 'compositionend' events fired by the platform to the VKB. When not in
> composition mode, the keyboard would have to call sendKey() when a key would
> be pressed and, when informed that it should now go to composition mode,
> updateComposition() would have to be used exactly like in solution 1. The
> text being composed would be passed in the 'compositionstart' event.
> With that solution, sendKey() could whether create a new composition or not,
> depending on the platform's decision but calling sendKey() between
> 'compositionstart' and 'compositionend' events would throw an exception.

This last sentence seems to be missing a word. And even so, I'm not
sure what the platform's decision is and how it can be made. I feel
like any kind of decision should be deferred to the VKB.

Also, I don't like a key-centric approach at all as it would make
methods like handwriting recognition, voice input, etc. have to
conform to this particular API.

So, solution #1 with explicit keyboard emulation features has my vote.


P.S.: Something somewhat related to IME is copy & paste support, at
least the bits about focus and selection and placing (text) data into
a form field. Might be good to design the VKB API in a way so that it
could also serve for that use case (with an explicit clipboard API
provided by the User Agent, of course). See also
https://bugzilla.mozilla.org/show_bug.cgi?id=747798.

[Timothy Chien]

Mounir, Vivien,

Thank you for drafting the APIs. Few comments here

1) I think you are mixing the composition events and the text selection
here. Composition are pending characters in Asian IME or alphabet input
using dead key/modifier key.

For example, on a Mac with English keyboard layout, I can type alt+`
then E to type È. "alt+`" will trigger composition mode and send
composition events. The text/character that is currently in composition
("`" here) will be shown in input with underline. Asian IME use the same
UI extensively to input text.

Your selection dealing solutions looks good to me as long as it's
s/composition/selection/g. Unless you are really serious about sending
the selected text back to the composition mode (that's really weird,
considering text in composition is actually non-exist in the actual
input.value, at least not yet).

Then we would need another set of functions to deal with the real
composition and composition events.

2) I would imagine the sendValue() function will be used to provide the
spell check and auto-suggest feature. Would there be another method to
specify the position of the cursor *after* updating the value? For
example, if the user typed "t" "o" "m" "o" "r" and ended up accepting
the suggestion "tomorrow", I would want to put the cursor after the "w"
instead of the first "r".

3) key events is actually quite tricky with the keyCode and charCode.
Fun thing to try here: http://jsfiddle.net/timdream/YDGgk/

For keyup and keydown events the charCode is always zero, and keyCode
represent the hardware key pressed (equal to the lowercase ASCII value
for alphabetical keys). For keypress event the keyCode is zero, and
charCode is the Unicode value of the character. And there are other
things like shiftKey ctrlKey ...

Basically all parameter needed in event.initKeyEvent() would need to be
able to controlled by the virtual keyboard.

= My proposal =

For problems above, I think alternatively we should separate the events
from value. For input elements there should be three useable functions:

- .sendKeyEvent(KeyboardEvent event|CompositionEvent event)
Event can be key events and composition events created by
document.createElement().
Just like firing a generated key event on a input, this function does
not update the value.

- .sendValue(DOMString value)
same function Mounir proposed

- .setComposition(DOMString value)
update the underlined composition shown on UI

- .setCursorPosition(long idx)
set the cursor position. This can be a property instead of a function if
we decided to expose all info of the current focused element as
properties instead of in focuschange event.

For select elements function suggested by Mounir will work:

- .setSelectedOption(long idx)
- .setSelectedOptions(Array<int> idxs)

Separate input value update with events can complicated the virtual
keyboard itself, but I think this will make the API more generalized and
minimal.


Tim

[Timothy Chien]

FYI, web developers expect to controlling the behavior of virtual
keyboard with various attributes on the input element. We would need to
decide which attribute we want to expose to the virtual keyboard app.

https://www.w3.org/Bugs/Public/show_bug.cgi?id=12885
WF3: Provide a way for the author to specify what kind of keyboard
behaviour to use for text inputs on multimodal keyboards (e.g. show the
numeric keyboard, capitalise the first character by default, etc)

http://developer.apple.com/library/ios/documentation/StringsTextFonts/Conceptual/TextAndWebiPhoneOS/KeyboardManagement/KeyboardManagement.html#//apple_ref/doc/uid/TP40009542-CH5-SW1
check "Configuring the Keyboard for Web Views"


Tim

[Mounir Lamouri]

On 04/26/2012 01:36 AM, Timothy Chien wrote:
> FYI, web developers expect to controlling the behavior of virtual
> keyboard with various attributes on the input element. We would need to
> decide which attribute we want to expose to the virtual keyboard app.
>
> https://www.w3.org/Bugs/Public/show_bug.cgi?id=12885
> WF3: Provide a way for the author to specify what kind of keyboard
> behaviour to use for text inputs on multimodal keyboards (e.g. show the
> numeric keyboard, capitalise the first character by default, etc)

We are planning to implement that in Gecko. See bug 746142.

> http://developer.apple.com/library/ios/documentation/StringsTextFonts/Conceptual/TextAndWebiPhoneOS/KeyboardManagement/KeyboardManagement.html#//apple_ref/doc/uid/TP40009542-CH5-SW1
>
> check "Configuring the Keyboard for Web Views"

I wouldn't pay much attention to Apple's documentation. They never
pushed those attributes to any standard and very likely don't care about
that.
FWIW, @autocapitalization is going to be part of @inputmode and
@autocorrect is *very* specific to Apple's virtual keyboard.

Cheers,
--
Mounir