Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app

Showing 1-28 of 28 messages
Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Yuan Xulei 11/8/12 11:20 PM
Hi all,

In order to enhance the virtual keyboard for more commonly used feature
on the Android and the iOS platforms, and also to support the 3rd-party
keyboard app after v1, we(Rudy, Tim and I) are trying to design a new
mozKeyboard API.

I created a MoPad to list and discuss our proposal:
  https://etherpad.mozilla.org/gaia-mozkeyboard-v3

The draft API of the our proposal are listed bellow, which is similar to
that of the android.

== KeyboardManager ==

When user focuses an editable text field which requires the soft
keyboard, an "imestatechange" event will be fired.

The keyboard app should listen to the  "inputmethodstatechane" events by
  the KeyboardManager#oninputmethodstatechane attribute. Then it check
the KeyboardManager#connection attribute. If the
KeyboardManager#connection  is not null, it's time to start input or
else finish input.

The full interface of KeyboardManager:

partial interface Navigator {
readonly attribute KeyboardManager mozKeyboard;
};

interface KeyboardManager {
     // Show a (system) menu for all input method that allow user to select
     void showInputMethodPicker();
     // Switch directly to the next input method
     void switchToNextInputMethod();
     // Hide and show the soft keyboard
     void hideSoftKeyboard();
     void showSoftKeyboard();
     // Not null when start input
readonly attribute KeyboardConnection? connection;
     // User begins or finishes editing a field.
attribute Function oninputmethodstatechane;
     // User moves the cursor, changes the selection, or alers the
composing text length
attribute Function onselectionchange;
};

== KeyboardConnection  ==

The KeyboardManager#connection  is an interface of KeyboardConnection
that  defines the communication channel from the keyboard to the app
that  is receiving user input. We  use the KeyboardConnection  to read
and edit the value of the input field.

interface KeyboardConnection {
     // Send a keyevent to the input field that is currently attached.
It exists in the current interface
     void sendKey(in long keyCode, in long charCode);
     // Commit text to the input field to current cursor position and
clear composing text.
     void commitText(DOMString text);
     // Set the value of the current input field.  It exists in the
current interface and is used for "special situations where the value
had to be chosen amongst a list  (type=month) or a widget (type=date,
time, etc.)".
     // Check
https://groups.google.com/group/mozilla.dev.webapi/browse_thread/thread/56cdfe1c6bfd34dcfor
details.
     void setValue(DOMString text);
     // Get "length" characters of text before the current cursor position.
DOMString getTextBeforeCursor(in long length);
     // Get "length" characters of text after the current cursor position.
DOMString getTextAfterCursor(in long length);
     // Delete text around the cursor
     void deleteSurroundingText(in long beforeLength, in long afterLength);
     //  Set the composing text around the current cursor position as
the parameter "text"
     void setComposingText(DOMString text);
     // Clear composing text
     void finishComposingText();
     // The start and stop position of the selection. The start position
is the current cursor position. To set the cursor position, set the
start and end position tthe same value.
attribute long selectionStart;
attribute long selectionEnd;
readonly attribute DOMString selectedText;
     // The type of the input field, including text, number, password,
url, tel and email.
readonly attribute DOMString inputType;
     // Get the return key type, which determines the title of the key
displayed in the keyboard and action performed when it is pressed. The
value could be "done", "next", "go", "search" and "send".
     // If the value is "next", when pressing the enter key, the focus
will be move to the next input field.
readonly attribute DOMString returnKeyType;
     // Used to specifiy a customized label for the return key.
attribute DOMString returnKeyLabel;
     // The capitalization mode, which could be one of these types:
none, words, sentences  and characters.
readonly attribute DOMString capitalizationMode;
};

== KeyboardSelectionChangeEvent ==

To moniter the changes of the input field, the keyboard app could listen
to the  "selectionchange" events by  the
KeyboardManager#onselectionchange attribute.
An object of KeyboardSelectionChangeEvent will be passed when a
"selectionchange" event is fired.

interface KeyboardSelectionChangeEvent: Event {
readonly attribute long oldSelectionStart;
readonly attribute long oldSelectionEnd;
readonly attribute long newSelectionStart;
readonly attribute long newSelectionEnd;
readonly attribute long composingTextStart;
readonly attribute long composingTextEnd;
};

sample code:

// Called when the user starts or finishes editing an input field
navigator.mozKeyboard.oninputmethodstatechane= function(event) {
     // Get the KeyboardConnection instance
     var conn = navigator.mozKeyboard.connection;
     if (!conn) {
         // finishes editing
return;
     }
     // Send a string to user's input field
     if (conn.inputType == "text") {
conn.commitText("Hello world");
     } else if (conn.inputType == "url") {
conn.commitText("http://www.mozilla.org");
     } else {
conn.commitText("123");
     }
};


Please let me know if you have any questions.
Thanks.

Regards,

Yuan Xulei
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Ben Francis 11/16/12 9:20 AM
Hi,

I was at a Career Fair at Cambridge University yesterday and was talking to
an engineer from SwiftKey (swiftkey.net), a London-based startup who make
an alternative keyboard for Android.

Their advice was not to make our keyboard API anything like Android's!

I have pointed them towards this thread to give them an opportunity to
provide further feedback.

Ben

On Fri, Nov 9, 2012 at 7:20 AM, Yuan Xulei <xy...@mozilla.com> wrote:

> Hi all,
>
> In order to enhance the virtual keyboard for more commonly used feature on
> the Android and the iOS platforms, and also to support the 3rd-party
> keyboard app after v1, we(Rudy, Tim and I) are trying to design a new
> mozKeyboard API.
>
> I created a MoPad to list and discuss our proposal:
>  https://etherpad.mozilla.org/**gaia-mozkeyboard-v3<https://etherpad.mozilla.org/gaia-mozkeyboard-v3>
>
> The draft API of the our proposal are listed bellow, which is similar to
> that of the android.
>
> == KeyboardManager ==
>
> When user focuses an editable text field which requires the soft keyboard,
> an "imestatechange" event will be fired.
>
> The keyboard app should listen to the  "inputmethodstatechane" events by
>  the KeyboardManager#**oninputmethodstatechane attribute. Then it check
>     // Check https://groups.google.com/**group/mozilla.dev.webapi/**
> browse_thread/thread/**56cdfe1c6bfd34dcfor<https://groups.google.com/group/mozilla.dev.webapi/browse_thread/thread/56cdfe1c6bfd34dcfor>details.
> to the  "selectionchange" events by  the KeyboardManager#**onselectionchange
> attribute.
> An object of KeyboardSelectionChangeEvent will be passed when a
> "selectionchange" event is fired.
>
> interface KeyboardSelectionChangeEvent: Event {
> readonly attribute long oldSelectionStart;
> readonly attribute long oldSelectionEnd;
> readonly attribute long newSelectionStart;
> readonly attribute long newSelectionEnd;
> readonly attribute long composingTextStart;
> readonly attribute long composingTextEnd;
> };
>
> sample code:
>
> // Called when the user starts or finishes editing an input field
> navigator.mozKeyboard.**oninputmethodstatechane= function(event) {
>     // Get the KeyboardConnection instance
>     var conn = navigator.mozKeyboard.**connection;
>     if (!conn) {
>         // finishes editing
> return;
>     }
>     // Send a string to user's input field
>     if (conn.inputType == "text") {
> conn.commitText("Hello world");
>     } else if (conn.inputType == "url") {
> conn.commitText("http://www.**mozilla.org <http://www.mozilla.org>");
>     } else {
> conn.commitText("123");
>     }
> };
>
>
> Please let me know if you have any questions.
> Thanks.
>
> Regards,
>
> Yuan Xulei
> ______________________________**_________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-webapi<https://lists.mozilla.org/listinfo/dev-webapi>
>



--
Ben Francis
http://tola.me.uk
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 11/19/12 1:20 AM
Hi Ben,

Thanks for your advices, but could please help to provide more details
on why not to be like android.

We borrowed some API from the android because it fits our needs and has
been tested by 3-rd IMEs on android.  Also it will be easier for the
developer to learn the IME framework and port the existing android IMEs
to Firefox OS.

If there exists limitation and flaws with the draft API, please feel
free to  point it out and make it better.

Regards,

Yuan Xulei

On 11/17/2012 01:20 AM, Ben Francis wrote:
> Hi,
>
> I was at a Career Fair at Cambridge University yesterday and was
> talking to an engineer from SwiftKey (swiftkey.net
> <http://swiftkey.net>), a London-based startup who make an alternative
> keyboard for Android.
>
> Their advice was not to make our keyboard API anything like Android's!
>
> I have pointed them towards this thread to give them an opportunity to
> provide further feedback.
>
> Ben
>
> On Fri, Nov 9, 2012 at 7:20 AM, Yuan Xulei <xy...@mozilla.com
> <mailto:xy...@mozilla.com>> wrote:
>
>     Hi all,
>
>     In order to enhance the virtual keyboard for more commonly used
>     feature on the Android and the iOS platforms, and also to support
>     the 3rd-party keyboard app after v1, we(Rudy, Tim and I) are
>     trying to design a new mozKeyboard API.
>
>     I created a MoPad to list and discuss our proposal:
>     https://etherpad.mozilla.org/gaia-mozkeyboard-v3
>
>     The draft API of the our proposal are listed bellow, which is
>     similar to that of the android.
>
>     == KeyboardManager ==
>
>     When user focuses an editable text field which requires the soft
>     keyboard, an "imestatechange" event will be fired.
>
>     The keyboard app should listen to the  "inputmethodstatechane"
>     events by  the KeyboardManager#oninputmethodstatechane attribute.
>     KeyboardManager#onselectionchange attribute.
>     An object of KeyboardSelectionChangeEvent will be passed when a
>     "selectionchange" event is fired.
>
>     interface KeyboardSelectionChangeEvent: Event {
>     readonly attribute long oldSelectionStart;
>     readonly attribute long oldSelectionEnd;
>     readonly attribute long newSelectionStart;
>     readonly attribute long newSelectionEnd;
>     readonly attribute long composingTextStart;
>     readonly attribute long composingTextEnd;
>     };
>
>     sample code:
>
>     // Called when the user starts or finishes editing an input field
>     navigator.mozKeyboard.oninputmethodstatechane= function(event) {
>         // Get the KeyboardConnection instance
>         var conn = navigator.mozKeyboard.connection;
>         if (!conn) {
>             // finishes editing
>     return;
>         }
>         // Send a string to user's input field
>         if (conn.inputType == "text") {
>     conn.commitText("Hello world");
>         } else if (conn.inputType == "url") {
>     conn.commitText("http://www.mozilla.org");
>         } else {
>     conn.commitText("123");
>         }
>     };
>
>
>     Please let me know if you have any questions.
>     Thanks.
>
>     Regards,
>
>     Yuan Xulei
>     _______________________________________________
>     dev-webapi mailing list
>     dev-w...@lists.mozilla.org <mailto:dev-w...@lists.mozilla.org>
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 1/7/13 7:00 PM
Hi,

I made some modification and updated the API according to Salvador's
advices(https://etherpad.mozilla.org/gaia-mozkeyboard-v3).

A new MoPad was created to comment and discuss the updated API:
https://etherpad.mozilla.org/gaia-mozkeyboard-v4.
Please feel free to add your thoughts and comment.

= Major changes from previous version
1. Change the name of the mozKeyboard interface to InputMethodConnection
and separate the InputMethodConnection from the InputMethodManager
interface.
The InputMethodConnection interface should be only visible to the input
method app.

2. Remove the method(showSoftKeyboard) to explicitly show the keyboard,
while keep the method(removeFocus) to explicitly hide the keyboard.
User could explicitly show the keyboard by focusing a text field, so
there is no need to provide an extra method to show the keyboard.

3. remove finishComposingText
The composing text could be cleared by:
setComposingText('');

4. remove selectedText attribute

5. Add asynchronous method getText to get the text content of the text
field.
Remove the methods(getTextBeforeCursor, getTextAfterCursor) to get text
around the cursor.

6. Add methods to switch focus onto the previous/next input field.

= API Proposal:

partial interface Navigator {
     readonly attribute InputMethodManager mozInputMethodManager;
     readonly attribute InputMethodConnection mozInputMethodConnection;
};

== InputMethodConnection  ==

The interface defines the communication channel from the keyboard to the
app that  is receiving user input. The IME app  use it to read and edit
the value of the input field.

interface InputMethodConnection {
     // User begins or finishes editing a field.
     // If the user change input directly from one input to another,
only one event will be dispatched.
     attribute Function oninputmethodstatechange;

     // Whether the user begins editing.
     readonly attribute boolean inputStarted;

     // User moves the cursor, changes the selection, or alters the
composing text length
     attribute Function onselectionchange;

     // Send a keyevent to the input field that is currently attached.
It exists in the current interface
     void sendKey(in long keyCode, in long charCode, in long modifiers);

     // Insert the given text at the cursor position. Replace the
selected text if any and clear the composing text.
     void commitText(in DOMString text);

     // Set the text value of the current input field.
     void setText(in DOMString text);
     // Get the text value of the current input field asynchronously.
     void getText(in long start, in long end, in Function callback);

     // Length of the content
     readonly attribute long textLength;

     // Delete text around the cursor
     void deleteSurroundingText(in long beforeLength, in long afterLength);

     // Set the composing text before the current cursor position.
     // To clear the composing text, set the text paramter to the empty
string.
     void setComposingText(in DOMString text);

     // The start and stop position of the selection.
     readonly attribute long selectionStart;
     readonly attribute long selectionEnd;

     // Set the selection range of the the editable text.
     // Note that the start position should be less or equal to the end
position.
     // To move the cursor, set the start and end position to the same
value.
     void setSelectionRange(in long start, in long end);

     // Clear the focus of the current input field and hide the keyboard.
     void removeFocus();

     // Focus the next text field.
     void advanceFocus();

     // Focus the previous text field.
     void rewindFocus();

     // This read only attribute is true if user can advance the focus
to the next
     // text field.
     readonly attribute boolean canAdvanceFocus;

     // This readonly attribute is true if user can rewind the focus
back to the
     // previous text field.
     readonly attribute boolean canRewindFocus;

     // The input mode string.
     readonly attribute DOMString inputMode;

     // The type of the input field, including text, number, password,
url, tel and email.
     readonly attribute DOMString inputType;

     // Get the return key type, which determines the title of the key
displayed in the keyboard and action performed when it is pressed. The
value could be "done", "next", "go", "search" and "send".

     // If the value is "next", when pressing the enter key, the focus
should be move to the next input field.
     readonly attribute DOMString returnKeyType;

     // Get the customized label for the return key.
     readonly attribute DOMString returnKeyLabel;
};

sample code:

var conn = navigator.mozInputMethodConnection;

// Called when the user starts or finishes editing an input field
conn.oninputmethodstatechange= function(event) {
     if (conn.inputStarted) {
         // Text input should start
     } else {
         // Text input should end
     }
};

// Insert a string at the current cursor position
conn.commitText('Hello world');

// Get the selected text
conn.getText(conn.selectionStart, conn.selectionEnd, function
callback(text) {
     var selectedText = text;
});

// Move the cursor position
var position = 10;
conn.setSelectionRange(position, position);

// Hide the keyboard
conn.removeFocus();

// Switch the focus onto the next input field if possible
if (conn.canAdvanceFocus) {
conn.advanceFocus();
}



On 11/27/2012 02:36 AM, Ben Francis wrote:
> On Mon, Nov 19, 2012 at 9:20 AM, Yuan Xulei <xy...@mozilla.com
> <mailto:xy...@mozilla.com>> wrote:
>
>     Thanks for your advices, but could please help to provide more
>     details on why not to be like android.
>
>
> I have no idea, I've never built an IME. This is just what the
> engineer from SwiftKey said, and they do have this experience :) I'm
> hoping they can provide further feedback.
>
> Ben
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 1/31/13 9:38 AM
Hi all,

A wiki page of the extended MozKeyboard API was created:

https://wiki.mozilla.org/WebAPI/KeboardIME

The patch of the implementation could be found in bug 737110:

https://bugzilla.mozilla.org/show_bug.cgi?id=737110

Hope it will help and the API is in great need of comments/suggestions.
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 2/6/13 7:37 AM
Hello guys.

I saw this bug referring to keyboard:
https://bugzilla.mozilla.org/show_bug.cgi?id=838308

In short, any application can use mozKeyboard.onfocuschange and
*setValue()* method to inject text in the target input, in any other
application.

 From my point of view, there is no problem to keep mozKeyboard open
without permission but only if the sender of the *setValue()* message is
in the same domain than the receiver. We should control these kind of
situations when designing the new API. Of course, the sender could be
the keyboard application which receives special permission to inject
text to any other input, whatever its domain was.

Cheers!
________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 2/6/13 6:19 PM
The mozKeyboard API currently is used by the keyboard app to generate
keyboard events and interact with input target.

IMO, It should be restrict to the keyboard apps and normal app has no
permission to use it.
And the keyboard app could be declared by the manifest file, for example
giving a special permission in the manifest file.

Furthermore, if there are more than one keyboard apps enabled, only the
active one in the foreground is able to inject text to other apps.

Yuan
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 2:21 AM
Hello guys,

We're working on implementing the new mozKeyboard API:
   https://wiki.mozilla.org/WebAPI/KeboardIME

Some changes are made according to people's advice and our requirements,
including:
1. Combined commitText with deleteSurroundingText into
replaceSurroundingText.
2. Add ontextchange attribute to allow IME get notifications when the
text content has been changed.
3. Combined returnKeyType with returnKeyLabel into returnKey.

If you have any question about the API, please let us know before we
finish the implementation work. We're waiting for your feedback.

FYI, The previous version of the proposed API is
https://wiki.mozilla.org/WebAPI/KeboardIME?title=WebAPI/KeboardIME&oldid=510135.

Thanks,

Yuan Xulei
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Anne van Kesteren 4/15/13 2:34 AM
On Mon, Apr 15, 2013 at 10:21 AM, Yuan Xulei <xy...@mozilla.com> wrote:
> We're working on implementing the new mozKeyboard API:
>   https://wiki.mozilla.org/WebAPI/KeboardIME

FWIW, it seems Microsoft might have an interest similar to ours:
http://lists.w3.org/Archives/Public/public-webapps/2013JanMar/1059.html

We might also want to tell the WebApps list what we're working on to
see if long term we can work out something that works the same
everywhere.


--
http://annevankesteren.nl/
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/15/13 3:03 AM
Hello people.

In-line commentaries:

Some changes are made according to people's advice and our requirements, including:
1. Combined commitText with deleteSurroundingText into replaceSurroundingText.
INMHO, the name is very long but I understand is very explicative. I suggest some replacements but may be other people can suggest a better (shorter) wording:

  *   alterText() - my favourite
  *   changeText()
  *   replaceText()

When I was developing the keyboard I did not find any situation requiring changing the surrounding text but I can be different when developing suggestion or other IMEs. If so, the I have nothing to say but if not, I do not fully understand why to include these "surrounding" methods. May be we can use something similar to splice in Array methods. Some like:

alterText(newText, fromIndex, toIndex, cursorRelative) - replace text between fromIndex to toIndex (not included) with newText. If cursorRelative is set to true fromIndex and toIndex are treated like beforeLength and afterLength.

But do whatever you want with this proposal. ;)

2. Add ontextchange attribute to allow IME get notifications when the text content has been changed.
What kind of information is included in this event? When this event is triggered? If I commit text from the keyboard, is it triggered as well?
3. Combined returnKeyType with returnKeyLabel into returnKey.
I would keep type and label separated, may be in the same attribute but I think this should be an object (or interface) with different fields for type and label.

Remember, if you are here in Madrid and want to talk remember I'm in the 6th floor.

Cheers.
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Ehsan Akhgari 4/15/13 7:24 AM
Hi Yuan,

Here's my feedback based on <https://wiki.mozilla.org/WebAPI/KeboardIME>
which I believe is the latest version of the proposal (please correct me
if I'm wrong.)

1. InputMethodManager is not defined anywhere as far as I can see.

2. oninputmethodstatechange, onselectionchange and ontextchange should
be EventHandlers (and InputMethodConnection should be an EventTarget).

3. As Jonas mentioned in the bug, I think that we should make sure that
the API doesn't make assumptions about the text edit control.
Specifically, setText for example will not be useful for a
contentEditable element (and probably not even for textareas).  If you
need an API to clear the text, you should add that, but the interaction
with multi-line text editing in contentEditable/textarea needs to be
thought out.

4. returnKey's value can be a localized string, right?  In that case, I
don't think that special casing the behavior for "done" is the right
thing to do, we should just add another boolean attribute to expose what
happens when pressing the return key.  Also, why is returnKey readonly?

5. What is the use case for the beforeLength and afterLength arguments
to replaceSurroundingText?

6. Do we want to expose some information about text change events, such
as the before/after text, the offset of the text inside the editable
field, etc.?  Same question for onselectionchange.

7. selectionStart and selectionEnd need to be unsigned long.

8. Can you please rewrite the proposal in Web IDL, and drop the moz
prefixes?

Thanks!
Ehsan
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 8:50 AM
On 04/15/2013 06:03 PM, Salvador de la Puente González wrote:
> Hello people.
>
> In-line commentaries:
>
>> Some changes are made according to people's advice and our
>> requirements, including:
>> 1. Combined commitText with deleteSurroundingText into
>> replaceSurroundingText.
> INMHO, the name is very long but I understand is very explicative. I
> suggest some replacements but may be other people can suggest a better
> (shorter) wording:
>
>   * alterText() - /my favourite/
>   * changeText()
>   * replaceText()
>
       I prefer alterText and replaceText.
>
> When I was developing the keyboard I did not find any situation
> requiring changing the surrounding text but I can be different when
> developing suggestion or other IMEs. If so, the I have nothing to say
> but if not, I do not fully understand why to include these
> "surrounding" methods.
>
The most three common actions of an IME  is inserting word, making
suggestion and auto-correction.  These actions are usually taken at or
around the cursor. Take auto-correction for example, when user selects a
misspelled word and chooses to correct it, the IME needs to replace the
misspelled word around the cursor. Since most of the time the IME only
cares about the text around the cursor,  we include these "surrounding"
method.
>
> May be we can use something similar to splice in Array methods. Some like:
>
> /alterText(newText, fromIndex, toIndex, cursorRelative)/ - replace
> text between /fromIndex/ to /to//Index/ (not included) with /newText/.
> If /cursorRelative/ is set to true /fromIndex/ and /toIndex/ are
> treated like /beforeLength/ and /afterLength/.
>
> But do whatever you want with this proposal. ;)
>
>> 2. Add ontextchange attribute to allow IME get notifications when the
>> text content has been changed.
> What kind of information is included in this event?
     Nothing. It should include the changed text, but for
contentEditable element, the text may be huge. Thus it requires IME to
invokes `getText ` method to get the changed text.
> When this event is triggered?
     A new input field get focused. The text of the input field has been
changed by key events, commit text, delete text,  JS or other operations.
> If I commit text from the keyboard, is it triggered as well?
Yes, it is.
>> 3. Combined returnKeyType with returnKeyLabel into returnKey.
> I would keep type and label separated, may be in the same attribute
> but I think this should be an object (or interface) with different
> fields for type and label.
I like the idea of storing type and label as an object.
>
> Remember, if you are here in Madrid and want to talk remember I'm in
> the 6th floor.
I'm not in Madrid ;-(
>
> Cheers.
>
> ------------------------------------------------------------------------
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 9:27 AM
On 04/15/2013 10:24 PM, Ehsan Akhgari wrote:
> Hi Yuan,
>
> Here's my feedback based on
> <https://wiki.mozilla.org/WebAPI/KeboardIME> which I believe is the
> latest version of the proposal (please correct me if I'm wrong.)
>
> 1. InputMethodManager is not defined anywhere as far as I can see.
Still not defined yet.
>
> 2. oninputmethodstatechange, onselectionchange and ontextchange should
> be EventHandlers (and InputMethodConnection should be an EventTarget).
>
> 3. As Jonas mentioned in the bug, I think that we should make sure
> that the API doesn't make assumptions about the text edit control.
> Specifically, setText for example will not be useful for a
> contentEditable element (and probably not even for textareas).  If you
> need an API to clear the text, you should add that, but the
> interaction with multi-line text editing in contentEditable/textarea
> needs to be thought out.
I'm consider removing `setText` from API as it is only used for some
special cases now, such as setting the value of a date time field, but
I'm not sure if there exist any other cases we need it.
>
> 4. returnKey's value can be a localized string, right?
      Yes.
> In that case, I don't think that special casing the behavior for
> "done" is the right thing to do, we should just add another boolean
> attribute to expose what happens when pressing the return key.  Also,
> why is returnKey readonly?
     The behaviour and title of the return key are specified by the
input field and the IME can't change it. For example, a search box could
set its return key to 'search' with the code `<input type='text'
returnKey='search' />`.

>
>
> 5. What is the use case for the beforeLength and afterLength arguments
> to replaceSurroundingText?
To correct a word, user would replace the word before or around the cursor.
>
> 6. Do we want to expose some information about text change events,
> such as the before/after text, the offset of the text inside the
> editable field, etc.?
Currently no information will be included. The before/after text and
text offset will be useful for making suggestion and auto-correction,
may be we could expose some of them without degrading the system
performance.

> Same question for onselectionchange.
The selectionStart and selectionEnd information will be enough, I think.
>
> 7. selectionStart and selectionEnd need to be unsigned long.
>
> 8. Can you please rewrite the proposal in Web IDL, and drop the moz
> prefixes?
>
> Thanks!
> Ehsan
I'll rewrite the proposal in Web IDL and fix the issues you mentioned.

Thanks!
Yuan
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/15/13 10:02 AM
Hello Yuan

>>> 2. Add ontextchange attribute to allow IME get notifications when
>>> the text content has been changed.
>> What kind of information is included in this event?
>     Nothing. It should include the changed text, but for
> contentEditable element, the text may be huge. Thus it requires IME to
> invokes `getText ` method to get the changed text.
Here we can add just the information about if this event was triggered
by this session or not. In order to distinguish between IME operations
or external action. My suggest is to add 'external' flag set to true if
the event is not triggered by the edit session.

That's all.



________________________________
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 10:56 AM
Hi Salvador,

That will be better. Or maybe only send event triggered by external? And
let the IME to track the edit session.
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/15/13 11:10 AM
Even better. I always can trigger a custom event If I want to listen for
my own modifications in this fashion.
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Ehsan Akhgari 4/15/13 12:23 PM
On 2013-04-15 12:27 PM, Yuan Xulei wrote:
>> 4. returnKey's value can be a localized string, right?
>       Yes.
>> In that case, I don't think that special casing the behavior for
>> "done" is the right thing to do, we should just add another boolean
>> attribute to expose what happens when pressing the return key.  Also,
>> why is returnKey readonly?
>      The behaviour and title of the return key are specified by the
> input field and the IME can't change it. For example, a search box could
> set its return key to 'search' with the code `<input type='text'
> returnKey='search' />`.

OK, but your proposal says:

If the value is "next", when pressing the enter key, the focus should be
move to the next input field.

I don't think this is a good idea since the web app might set returnKey
to the translation of "next" in the target locale, and that would break
the above behavior.

Also, is the returnKey attribute spec'ed somewhere?  How is one supposed
to specify it for contentEditable/designMode?

>> 5. What is the use case for the beforeLength and afterLength arguments
>> to replaceSurroundingText?
> To correct a word, user would replace the word before or around the cursor.

Wouldn't it be better to have a replaceWord(index) API instead, which
will replace the word that |index| would fall into?

I can see some complications with using this API for large text boxes
that are getting modified by both the keyboard and content script.
Tracking the correct offsets doesn't seem very easy in that case.

>> 6. Do we want to expose some information about text change events,
>> such as the before/after text, the offset of the text inside the
>> editable field, etc.?
> Currently no information will be included. The before/after text and
> text offset will be useful for making suggestion and auto-correction,
> may be we could expose some of them without degrading the system
> performance.

OK.


Cheers,
Ehsan
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/16/13 2:03 AM
On 04/16/2013 03:23 AM, Ehsan Akhgari wrote:
> On 2013-04-15 12:27 PM, Yuan Xulei wrote:
>>> 4. returnKey's value can be a localized string, right?
>>       Yes.
>>> In that case, I don't think that special casing the behavior for
>>> "done" is the right thing to do, we should just add another boolean
>>> attribute to expose what happens when pressing the return key.  Also,
>>> why is returnKey readonly?
>>      The behaviour and title of the return key are specified by the
>> input field and the IME can't change it. For example, a search box could
>> set its return key to 'search' with the code `<input type='text'
>> returnKey='search' />`.
>
> OK, but your proposal says:
>
> If the value is "next", when pressing the enter key, the focus should
> be move to the next input field.
>
> I don't think this is a good idea since the web app might set
> returnKey to the translation of "next" in the target locale, and that
> would break the above behavior.
Just as Salvador suggested, we could add a separate label property to
specify the return key label.
>
> Also, is the returnKey attribute spec'ed somewhere?
No, it's a new attribute and we should define it in this API.
> How is one supposed to specify it for contentEditable/designMode?
Add a returnKey attribute to the contentEditable element or the document
body for the design mode.
>>> 5. What is the use case for the beforeLength and afterLength arguments
>>> to replaceSurroundingText?
>> To correct a word, user would replace the word before or around the
>> cursor.
>
> Wouldn't it be better to have a replaceWord(index) API instead, which
> will replace the word that |index| would fall into?
For CJK IME, words are not separated by space. For example,  the chinese
sentence--我吃饭了(I had a meal). So I incline to provide a flexible API
to let IME implement the function of replaceWord.
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/17/13 3:12 AM
To remove any confusion, the Keyboard IME API and the IME API from
WebApps are totally different. They do not support the same use cases.
The former is to create virtual keyboards/IME using the Web Platform and
the later, as far as I understand it, is to show the system's IME in web
applications using canvas. For example, a textfield in a game.

--
Mounir
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/17/13 9:52 AM
Hi,

First of all, thank you for working on this Yuan. The API looks
generally good, but I have a few comments.

I think it would be good to clearly mention as a UC (Use Case) that this
API is there for dealing with text but not to deal with other form
controls like <select>.

I am not sure what InputMethodManager is there for given that it is not
defined. Is that intentional?

Also, we would like to no longer prefix stuff as much as we can. I would
be in favour to not push this API to Firefox OS until we have a rough
consensus on it and not prefix it with 'moz'. We should not expose it to
other platforms anyway, right?

I'm not a big fan of the 'inputmethodstatechange' event and the
'inputStarted' attribute. First of all, the event name is way too long
and developers don't like long name. 'statechange' would be enough. I
also have a hard time understanding what this 'inputStarted' is there
for. It seems there only to know if the 'statechange' event was fired
when we entered or left a text field. Is that correct? Should we simply
have some kind of 'start'/'change'/'finish' events? I actually wonder
how the Keyboard application is expected to be created. Shouldn't we
make the 'start' event a system message and maybe the 'finish' event
should not exist because we might expect the application to be killed at
that point? Though, I guess it might be platform dependant.

It seems that replaceSurroundingText could be replaced by replaceWord
unless there are some UCs I'm missing. Inserting some text maybe? But I
don't see in which case we want to insert more that one character at a
time without replacing the current word?

I feel that setComposingText UCs can be handled by replaceWord. If
replaceWord takes by the default the word at the cursor position and we
consider that in the following case "foo|" the cursur is position on the
word "foo", wouldn't setComposingText be exactly like replaceWord?

Why do we have getText() instead of always exposing the text inside the
field? Is that to improve performance?

What is the UC for removeFocus() all the keyboards I have seen are
dismissed by the system, not by themselves. Is there a particular UI in
mind?

advanceFocus() and rewindFocus() are interesting because they seem to
apply on the entire document but all the keyboards I have used would
make that feature work in a form, not in the document. Is that
applicable in the document on purpose? Also, this would skip some form
elements, which I believe a user wouldn't want.
If we include that, I think we should make those methods going to the
next/previous form control in the current form unless we have a strong
use case to go to the next/previous text field in the entire document.

I would like to keep inputMode outside of this API for the moment
because we do not have a clear and specified story for that.
Also, inputType could, maybe, be named 'type'.

Finally, returnKey will not be added as an attribute on form controls. I
do not think this is needed and people will reject that proposal if the
use case is a Keyboard API. Actually, what can't we do with the
information we already have? If type='search', you can use 'search', if
there is no form control after, you can use 'done' or 'go' (you might
need to know if the form is submittable for that though). I do not see
what 'send' is there for.
Anyway, the idea is that we should give enough context to the keyboard
so it can show its own UI based on those information.

Thanks,
--
Mounir

On 15/04/13 11:21, Yuan Xulei wrote:
> Hello guys,
>
> We're working on implementing the new mozKeyboard API:
>   https://wiki.mozilla.org/WebAPI/KeboardIME
>
> Some changes are made according to people's advice and our requirements,
> including:
> 1. Combined commitText with deleteSurroundingText into
> replaceSurroundingText.
> 2. Add ontextchange attribute to allow IME get notifications when the
> text content has been changed.
> 3. Combined returnKeyType with returnKeyLabel into returnKey.
>
> If you have any question about the API, please let us know before we
> finish the implementation work. We're waiting for your feedback.
>
> FYI, The previous version of the proposed API is
> https://wiki.mozilla.org/WebAPI/KeboardIME?title=WebAPI/KeboardIME&oldid=510135.
>
>
> Thanks,
>
> Yuan Xulei
> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi

Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/18/13 3:15 AM
One other thing: it seems that we do not give the IME access to the lang
of the element being accessed. I think it would be a valuable
information given that an IME application supporting multiple languages
could switch from a language to another.

--
Mounir
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/18/13 3:35 PM
On 04/18/2013 12:52 AM, Mounir Lamouri wrote:
> Hi,
>
> First of all, thank you for working on this Yuan. The API looks
> generally good, but I have a few comments.
>
> I think it would be good to clearly mention as a UC (Use Case) that this
> API is there for dealing with text but not to deal with other form
> controls like <select>.
>
> I am not sure what InputMethodManager is there for given that it is not
> defined. Is that intentional?
Yes, it will defined later.
>
> Also, we would like to no longer prefix stuff as much as we can. I would
> be in favour to not push this API to Firefox OS until we have a rough
> consensus on it and not prefix it with 'moz'.
I agree with you.
> We should not expose it to
> other platforms anyway, right?
Why not? The desktop and mobile browser may benefit from the API. We can
provide virtual keyboard to other platforms as well.
>
> I'm not a big fan of the 'inputmethodstatechange' event and the
> 'inputStarted' attribute. First of all, the event name is way too long
> and developers don't like long name. 'statechange' would be enough.
Shorter and simple event name will be better, but the name should tell
the developer that it is an IME event.
> I also have a hard time understanding what this 'inputStarted' is there
> for. It seems there only to know if the 'statechange' event was fired
> when we entered or left a text field. Is that correct?
Yes.
> Should we simply
> have some kind of 'start'/'change'/'finish' events? I actually wonder
> how the Keyboard application is expected to be created. Shouldn't we
> make the 'start' event a system message and maybe the 'finish' event
> should not exist because we might expect the application to be killed at
> that point? Though, I guess it might be platform dependant.
The life-cycle of the Keyboard application is:

1. create
|
2. start <-----
| |
3. change focus an input field
| |
4. finish -----
|
5. destroy

The Keyboard manager will create the keyboard app, which will run as a
normal app and receive the load event on start-up. When user focus an
input field, the keyboard app starts text input. During the input, the
content or attributes of input field may be changed by external
actions(for example, JS code to clear the text), the keyboard app should
get notification about the changes. After user finishes input of current
input field, the keyboard app will wait for the user to focus another
input field.

So there needs a least two types of events: one fired when user switches
between input fields, and the other fired when the current input field
has changed.

Besides the load and unload events, we may need additional 'start',
'change' and 'finish' events.
>
> It seems that replaceSurroundingText could be replaced by replaceWord
> unless there are some UCs I'm missing. Inserting some text maybe? But I
> don't see in which case we want to insert more that one character at a
> time without replacing the current word?
For CJK, the words are not separated by spaces. Each sentence could be
viewed as a word. replaceWord doesn't work for CJK IME.
And for Latin, when doing suggestions, we often need to insert some text
to complete a word, for example, input 'En' and get the suggestion of
'English'.
>
> I feel that setComposingText UCs can be handled by replaceWord. If
> replaceWord takes by the default the word at the cursor position and we
> consider that in the following case "foo|" the cursur is position on the
> word "foo", wouldn't setComposingText be exactly like replaceWord?
Yes, replaceWord or replaceSurroundingText can be used to add a string
to the input field as well. But the setComposingText is not used to add
string to the input field. It is used to handle composition events for
CJK IMEs. For example, to input the Chinese word '我', we will input the
composing text 'wo' first, then convert it to corresponding Chinese
character. Also sometimes we use composing text to help us input special
characters, such as mathematical symbols. For example, the composing
text of Σ is '\sum'. You can find more about composing text in the
section "2.1 Compose" of
https://dvcs.w3.org/hg/ime-api/raw-file/default/Overview.html#composer-section.
>
> Why do we have getText() instead of always exposing the text inside the
> field? Is that to improve performance?
Yes, to avoid transfer large text between apps, sine for the content
editable element, the text content may be several KB. But I think we
could change it to a sync method without callback.
>
> What is the UC for removeFocus() all the keyboards I have seen are
> dismissed by the system, not by themselves. Is there a particular UI in
> mind?

removeFocus could support a hide button for the keyboard. If the keyboard support full screen editing, it will need a hide button to cancel editing.

>
> advanceFocus() and rewindFocus() are interesting because they seem to
> apply on the entire document but all the keyboards I have used would
> make that feature work in a form, not in the document. Is that
> applicable in the document on purpose?
Yes, applicable in the document, but the app itself could the focus be
restricted in the forms.
> Also, this would skip some form
> elements, which I believe a user wouldn't want.
Yes, non-text field will be skipped. Those methods just follow the
behaviour of that on android and iOS.
> If we include that, I think we should make those methods going to the
> next/previous form control in the current form unless we have a strong
> use case to go to the next/previous text field in the entire document.
>
> I would like to keep inputMode outside of this API for the moment
> because we do not have a clear and specified story for that.
> Also, inputType could, maybe, be named 'type'.
>
> Finally, returnKey will not be added as an attribute on form controls. I
> do not think this is needed and people will reject that proposal if the
> use case is a Keyboard API. Actually, what can't we do with the
> information we already have? If type='search', you can use 'search', if
> there is no form control after, you can use 'done' or 'go' (you might
> need to know if the form is submittable for that though). I do not see
> what 'send' is there for.
> Anyway, the idea is that we should give enough context to the keyboard
> so it can show its own UI based on those information.
It will complicate for the keyboard to infer the return key meaning by
the context. Take the awesome bar of firefox browser for example, if
user inputs a keyword, the return key will be 'search'; if user inputs a
URL, the return key will be 'go'. Keybard can hardly determine the
return keyboard type by the content of the awesome bar.

'send' is used for sending a mail, SMS, instance message and so on.
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/18/13 3:42 PM
Should the IME get current language from system settings? Or each input
element could specify its language regardless of the system settings?
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/19/13 2:00 AM
On 19/04/13 00:42, Yuan Xulei wrote:
> Should the IME get current language from system settings? Or each input
> element could specify its language regardless of the system settings?

HTMLElement.lang can be used to specify the language of a web site or a
specific part of a website.

[1]
http://www.whatwg.org/specs/web-apps/current-work/multipage/elements.html#htmlelement

--
Mounir
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/19/13 2:50 AM
On 19/04/13 00:35, Yuan Xulei wrote:
> On 04/18/2013 12:52 AM, Mounir Lamouri wrote:
>> Hi,
>>
>> First of all, thank you for working on this Yuan. The API looks
>> generally good, but I have a few comments.
>>
>> I think it would be good to clearly mention as a UC (Use Case) that this
>> API is there for dealing with text but not to deal with other form
>> controls like <select>.
>>
>> I am not sure what InputMethodManager is there for given that it is not
>> defined. Is that intentional?
>
> Yes, it will defined later.

Please, define at least what would be the purpose.

>> We should not expose it to
>> other platforms anyway, right?
>
> Why not? The desktop and mobile browser may benefit from the API. We can
> provide virtual keyboard to other platforms as well.

I doubt any other platforms will use that API any time soon. Pushing it
to other platforms will increase the exposure of this API. We should
keep it inside Firefox OS to not harm the Web.

>> I'm not a big fan of the 'inputmethodstatechange' event and the
>> 'inputStarted' attribute. First of all, the event name is way too long
>> and developers don't like long name. 'statechange' would be enough.
>
> Shorter and simple event name will be better, but the name should tell
> the developer that it is an IME event.

This event is sent on an object named inputMethodConnection implementing
the InputMethodConnection interface. I believe it should be clear the
event is related to 'inputmethod'.

>> Should we simply
>> have some kind of 'start'/'change'/'finish' events? I actually wonder
>> how the Keyboard application is expected to be created. Shouldn't we
>> make the 'start' event a system message and maybe the 'finish' event
>> should not exist because we might expect the application to be killed at
>> that point? Though, I guess it might be platform dependant.
> The life-cycle of the Keyboard application is:
>
> 1. create
> |
> 2. start <-----
> | |
> 3. change focus an input field
> | |
> 4. finish -----
> |
> 5. destroy
>
> The Keyboard manager will create the keyboard app, which will run as a
> normal app and receive the load event on start-up. When user focus an
> input field, the keyboard app starts text input. During the input, the
> content or attributes of input field may be changed by external
> actions(for example, JS code to clear the text), the keyboard app should
> get notification about the changes. After user finishes input of current
> input field, the keyboard app will wait for the user to focus another
> input field.
>
> So there needs a least two types of events: one fired when user switches
> between input fields, and the other fired when the current input field
> has changed.

I'm not sure if you are agreeing or disagreeing with my proposal. It
seems that 'start', 'modify' and 'finish' would work correctly here,
wouldn't they? Also, 'inputStarted' could be replaced by 'enabled'
maybe? Basically, the keyboard application would know if it is currently
being in use.

> Besides the load and unload events, we may need additional 'start',
> 'change' and 'finish' events.

To be honest, I wonder if the start event is really needed. When the
keyboard application is started, it will be obviously used for a field
so a 'start' event is useless here. If the keyboard got dismissed and is
re-open again without being actually killed, we might as well just send
a 'modify' event. Another option could be to use a system message.
I'm not sure those are great ideas but probably worth thinking about.

>> It seems that replaceSurroundingText could be replaced by replaceWord
>> unless there are some UCs I'm missing. Inserting some text maybe? But I
>> don't see in which case we want to insert more that one character at a
>> time without replacing the current word?
>
> For CJK, the words are not separated by spaces. Each sentence could be
> viewed as a word. replaceWord doesn't work for CJK IME.
> And for Latin, when doing suggestions, we often need to insert some text
> to complete a word, for example, input 'En' and get the suggestion of
> 'English'.

For the latin case, you could simply do replaceWord('English') and that
would replace 'En' with 'English'.
For the CJK case, I do not know anything about that but I wonder how the
keyboard application would be able to know that something is a word if
the UA can't event know that. Are CJK users editing their previously
typed stuff and except to have auto-correction? or that is something
only latin users expect?

>> I feel that setComposingText UCs can be handled by replaceWord. If
>> replaceWord takes by the default the word at the cursor position and we
>> consider that in the following case "foo|" the cursur is position on the
>> word "foo", wouldn't setComposingText be exactly like replaceWord?
>
> Yes, replaceWord or replaceSurroundingText can be used to add a string
> to the input field as well. But the setComposingText is not used to add
> string to the input field. It is used to handle composition events for
> CJK IMEs. For example, to input the Chinese word '我', we will input the
> composing text 'wo' first, then convert it to corresponding Chinese
> character. Also sometimes we use composing text to help us input special
> characters, such as mathematical symbols. For example, the composing
> text of Σ is '\sum'. You can find more about composing text in the
> section "2.1 Compose" of
> https://dvcs.w3.org/hg/ime-api/raw-file/default/Overview.html#composer-section.

First use case:
- the user types 'wo';
- the keyboard application calls replaceWord('我') which is going to
replace the current word ('wo') with '我'.

Second use case:
- the user types '\sum';
- the keyboard application calls replaceWord('Σ') which is going to
replace the current word ('\sum') with 'Σ'.

Why wouldn't that work?

>> Why do we have getText() instead of always exposing the text inside the
>> field? Is that to improve performance?
>
> Yes, to avoid transfer large text between apps, sine for the content
> editable element, the text content may be several KB. But I think we
> could change it to a sync method without callback.

I think we should keep that async. If the concern is to save
performance, we shouldn't make that synchronous. Also, we should
probably use:
Future<DOMString> getText(start, end);

>> What is the UC for removeFocus() all the keyboards I have seen are
>> dismissed by the system, not by themselves. Is there a particular UI in
>> mind?
>
> removeFocus could support a hide button for the keyboard. If the
> keyboard support full screen editing, it will need a hide button to
> cancel editing.

I've never seen such UI. On Android, I'm able to dismiss a virtual
keyboard from the system UI but I never saw a virtual keyboard offering
a button to dismiss itself even when they are fullscreen.
Do you have specific UIs in mind? Are we planning to implement that in
our UI?

>> advanceFocus() and rewindFocus() are interesting because they seem to
>> apply on the entire document but all the keyboards I have used would
>> make that feature work in a form, not in the document. Is that
>> applicable in the document on purpose?
>
> Yes, applicable in the document, but the app itself could the focus be
> restricted in the forms.

I didn't catch that.

>> Also, this would skip some form
>> elements, which I believe a user wouldn't want.
>
> Yes, non-text field will be skipped. Those methods just follow the
> behaviour of that on android and iOS.

I haven't seen that behaviour on Android. I doubt the iOS behaviour is
that great. I think the form behaviour is way better but we could also
design the API such as a keyboard can implement both.

>> Finally, returnKey will not be added as an attribute on form controls. I
>> do not think this is needed and people will reject that proposal if the
>> use case is a Keyboard API. Actually, what can't we do with the
>> information we already have? If type='search', you can use 'search', if
>> there is no form control after, you can use 'done' or 'go' (you might
>> need to know if the form is submittable for that though). I do not see
>> what 'send' is there for.
>> Anyway, the idea is that we should give enough context to the keyboard
>> so it can show its own UI based on those information.
>
> It will complicate for the keyboard to infer the return key meaning by
> the context. Take the awesome bar of firefox browser for example, if
> user inputs a keyword, the return key will be 'search'; if user inputs a
> URL, the return key will be 'go'. Keybard can hardly determine the
> return keyboard type by the content of the awesome bar.
>
> 'send' is used for sending a mail, SMS, instance message and so on.

The problem is that first of all, you can't expect web pages to use that
magic returnKey attribute just because the Keyboard API uses it. If I
recall correctly, Firefox Android tried to use this and switched back to
guess the label based on the content.
Second of all, standardizing this API is going to be hard. If we have to
rely on a new magic attribute in HTML, this is going to be even harder.
The HTML specification will never accept that change. If we keep that
work self-contained, we increase the chances to have it standardized. If
that happen, we would have a stronger point if we go to the HTML WG and
ask for a new attribute for the virtual keyboards.

Cheers,
--
Mounir
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Yuan Xulei 4/28/13 11:46 AM
On 04/19/2013 05:50 PM, Mounir Lamouri wrote:
> On 19/04/13 00:35, Yuan Xulei wrote:
>> On 04/18/2013 12:52 AM, Mounir Lamouri wrote:
>>> Hi,
>>>
>>> First of all, thank you for working on this Yuan. The API looks
>>> generally good, but I have a few comments.
>>>
>>> I think it would be good to clearly mention as a UC (Use Case) that this
>>> API is there for dealing with text but not to deal with other form
>>> controls like <select>.
>>>
>>> I am not sure what InputMethodManager is there for given that it is not
>>> defined. Is that intentional?
>> Yes, it will defined later.
> Please, define at least what would be the purpose.
>
>>> We should not expose it to
>>> other platforms anyway, right?
>> Why not? The desktop and mobile browser may benefit from the API. We can
>> provide virtual keyboard to other platforms as well.
> I doubt any other platforms will use that API any time soon. Pushing it
> to other platforms will increase the exposure of this API. We should
> keep it inside Firefox OS to not harm the Web.
OK, I agree.
>>> I'm not a big fan of the 'inputmethodstatechange' event and the
>>> 'inputStarted' attribute. First of all, the event name is way too long
>>> and developers don't like long name. 'statechange' would be enough.
>> Shorter and simple event name will be better, but the name should tell
>> the developer that it is an IME event.
> This event is sent on an object named inputMethodConnection implementing
> the InputMethodConnection interface. I believe it should be clear the
> event is related to 'inputmethod'.
It is reasonable. So we can use a shorter name without confusion.
>>> Should we simply
>>> have some kind of 'start'/'change'/'finish' events? I actually wonder
>>> how the Keyboard application is expected to be created. Shouldn't we
>>> make the 'start' event a system message and maybe the 'finish' event
>>> should not exist because we might expect the application to be killed at
>>> that point? Though, I guess it might be platform dependant.
>> The life-cycle of the Keyboard application is:
>>
>> 1. create
>> |
>> 2. start <-----
>> | |
>> 3. change focus an input field
>> | |
>> 4. finish -----
>> |
>> 5. destroy
>>
>> The Keyboard manager will create the keyboard app, which will run as a
>> normal app and receive the load event on start-up. When user focus an
>> input field, the keyboard app starts text input. During the input, the
>> content or attributes of input field may be changed by external
>> actions(for example, JS code to clear the text), the keyboard app should
>> get notification about the changes. After user finishes input of current
>> input field, the keyboard app will wait for the user to focus another
>> input field.
>>
>> So there needs a least two types of events: one fired when user switches
>> between input fields, and the other fired when the current input field
>> has changed.
> I'm not sure if you are agreeing or disagreeing with my proposal. It
> seems that 'start', 'modify' and 'finish' would work correctly here,
> wouldn't they?
Yes, I agree.
> Also, 'inputStarted' could be replaced by 'enabled'
> maybe? Basically, the keyboard application would know if it is currently
> being in use.
>
I think "started" will be more precise than "enabled". An IME could be
enabled and disabled in the settings page. If an IME is enabled, user
could switch to that IME and use it. So "enabled" means the IME is
available to use. User will "start" an enabled IME after focusing an
input field.
>> Besides the load and unload events, we may need additional 'start',
>> 'change' and 'finish' events.
> To be honest, I wonder if the start event is really needed. When the
> keyboard application is started, it will be obviously used for a field
> so a 'start' event is useless here. If the keyboard got dismissed and is
> re-open again without being actually killed, we might as well just send
> a 'modify' event. Another option could be to use a system message.
> I'm not sure those are great ideas but probably worth thinking about.
I think both "start" and "modify" are necessary.
"start" is much like an "onfocus" event. It will be fired when user
focuses a new input field and the keyboard should reset its state to
meet the need of the input field.
For example, determines the keyboard layout by the input type.  If the
keyboard got dismissed and is re-open again without being actually
killed, we need send a 'start' event as well.

'modify' event is sent when the input field is changed by external
actions and the keyboard should not reset its state. For example, there
is a text field with a clear button inside. The default mode of the
keyboard is lower case mode. I change the keyboard to upper case mode
and start inputting. During inputting, I find a mistake and use the
clear button to clear the text field. The keyboard will get notified
through the "modify" and keep the lower case mode as I don't switching
input field. If we use "start" event instead, the keyboard will be reset
to lower case mode.
>>> It seems that replaceSurroundingText could be replaced by replaceWord
>>> unless there are some UCs I'm missing. Inserting some text maybe? But I
>>> don't see in which case we want to insert more that one character at a
>>> time without replacing the current word?
>> For CJK, the words are not separated by spaces. Each sentence could be
>> viewed as a word. replaceWord doesn't work for CJK IME.
>> And for Latin, when doing suggestions, we often need to insert some text
>> to complete a word, for example, input 'En' and get the suggestion of
>> 'English'.
> For the latin case, you could simply do replaceWord('English') and that
> would replace 'En' with 'English'.
> For the CJK case, I do not know anything about that but I wonder how the
> keyboard application would be able to know that something is a word if
> the UA can't event know that. Are CJK users editing their previously
> typed stuff and except to have auto-correction? or that is something
> only latin users expect?
The keyboard application uses complicated algorithm and large dictionary
to split a sentence into words. The CJK users seldom use
auto-correction, but they often use suggestion to complete a word. They
need a method to append string to current cursor.
>>> I feel that setComposingText UCs can be handled by replaceWord. If
>>> replaceWord takes by the default the word at the cursor position and we
>>> consider that in the following case "foo|" the cursur is position on the
>>> word "foo", wouldn't setComposingText be exactly like replaceWord?
>> Yes, replaceWord or replaceSurroundingText can be used to add a string
>> to the input field as well. But the setComposingText is not used to add
>> string to the input field. It is used to handle composition events for
>> CJK IMEs. For example, to input the Chinese word '我', we will input the
>> composing text 'wo' first, then convert it to corresponding Chinese
>> character. Also sometimes we use composing text to help us input special
>> characters, such as mathematical symbols. For example, the composing
>> text of Σ is '\sum'. You can find more about composing text in the
>> section "2.1 Compose" of
>> https://dvcs.w3.org/hg/ime-api/raw-file/default/Overview.html#composer-section.
> First use case:
> - the user types 'wo';
> - the keyboard application calls replaceWord('我') which is going to
> replace the current word ('wo') with '我'.
The style of the composing text(usually underlined) is different from
the committed text.  And the composing text is temporary and can be
cancelled by dismissing the keyboard or moving the cursor. It is not the
real text that user inputs. So we need a method to handle this special
kind of text.
First use case:
- the user types 'wo' and keyboard calls setComposingText to make it as
underlined composint text -- '_wo_'.
- the keyboard application calls replaceWord('我') which is going to
replace the composing text ('_wo_') with '我'.
>
> Second use case:
> - the user types '\sum';
> - the keyboard application calls replaceWord('Σ') which is going to
> replace the current word ('\sum') with 'Σ'.
>
> Why wouldn't that work?
>
>>> Why do we have getText() instead of always exposing the text inside the
>>> field? Is that to improve performance?
>> Yes, to avoid transfer large text between apps, sine for the content
>> editable element, the text content may be several KB. But I think we
>> could change it to a sync method without callback.
> I think we should keep that async. If the concern is to save
> performance, we shouldn't make that synchronous. Also, we should
> probably use:
> Future<DOMString> getText(start, end);
I agree.
>>> What is the UC for removeFocus() all the keyboards I have seen are
>>> dismissed by the system, not by themselves. Is there a particular UI in
>>> mind?
>> removeFocus could support a hide button for the keyboard. If the
>> keyboard support full screen editing, it will need a hide button to
>> cancel editing.
> I've never seen such UI. On Android, I'm able to dismiss a virtual
> keyboard from the system UI but I never saw a virtual keyboard offering
> a button to dismiss itself even when they are fullscreen.
> Do you have specific UIs in mind? Are we planning to implement that in
> our UI?
On Android, the Sogou IME(the most popular Chinese Pinyin IME) has a
small hide button at the end the candidates view.  And I see the default
keyboard of  some tablets of Android 4.0  has a hide button. On iPhone,
although there is no hide button, user can dismiss the keyboard by
swiping the screen. So I think we may provide such API to 3rd party IME,
but we don't have a plan to implement in our UI right now.
>>> advanceFocus() and rewindFocus() are interesting because they seem to
>>> apply on the entire document but all the keyboards I have used would
>>> make that feature work in a form, not in the document. Is that
>>> applicable in the document on purpose?
>> Yes, applicable in the document, but the app itself could the focus be
>> restricted in the forms.
> I didn't catch that.
>
>>> Also, this would skip some form
>>> elements, which I believe a user wouldn't want.
>> Yes, non-text field will be skipped. Those methods just follow the
>> behaviour of that on android and iOS.
> I haven't seen that behaviour on Android. I doubt the iOS behaviour is
> that great. I think the form behaviour is way better but we could also
> design the API such as a keyboard can implement both.
If we could implement the form behaviour, that will be great. But we'd
better provide a simple and straightforward API.
>>> Finally, returnKey will not be added as an attribute on form controls. I
>>> do not think this is needed and people will reject that proposal if the
>>> use case is a Keyboard API. Actually, what can't we do with the
>>> information we already have? If type='search', you can use 'search', if
>>> there is no form control after, you can use 'done' or 'go' (you might
>>> need to know if the form is submittable for that though). I do not see
>>> what 'send' is there for.
>>> Anyway, the idea is that we should give enough context to the keyboard
>>> so it can show its own UI based on those information.
>> It will complicate for the keyboard to infer the return key meaning by
>> the context. Take the awesome bar of firefox browser for example, if
>> user inputs a keyword, the return key will be 'search'; if user inputs a
>> URL, the return key will be 'go'. Keybard can hardly determine the
>> return keyboard type by the content of the awesome bar.
>>
>> 'send' is used for sending a mail, SMS, instance message and so on.
> The problem is that first of all, you can't expect web pages to use that
> magic returnKey attribute just because the Keyboard API uses it. If I
> recall correctly, Firefox Android tried to use this and switched back to
> guess the label based on the content.
> Second of all, standardizing this API is going to be hard. If we have to
> rely on a new magic attribute in HTML, this is going to be even harder.
> The HTML specification will never accept that change. If we keep that
> work self-contained, we increase the chances to have it standardized. If
> that happen, we would have a stronger point if we go to the HTML WG and
> ask for a new attribute for the virtual keyboards.
>
> Cheers,
> --
> Mounir
> _______________________________________________
>
Regards,
Yuan
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/29/13 12:34 AM
On 28/04/13 20:46, Yuan Xulei wrote:
On 04/19/2013 05:50 PM, Mounir Lamouri wrote:
On 19/04/13 00:35, Yuan Xulei wrote:
On 04/18/2013 12:52 AM, Mounir Lamouri wrote:
Hi,

First of all, thank you for working on this Yuan. The API looks
generally good, but I have a few comments.

I think it would be good to clearly mention as a UC (Use Case) that this
API is there for dealing with text but not to deal with other form
controls like <select>.

I am not sure what InputMethodManager is there for given that it is not
defined. Is that intentional?
Yes, it will defined later.
Please, define at least what would be the purpose.

We should not expose it to
other platforms anyway, right?
Why not? The desktop and mobile browser may benefit from the API. We can
provide virtual keyboard to other platforms as well.
I doubt any other platforms will use that API any time soon. Pushing it
to other platforms will increase the exposure of this API. We should
keep it inside Firefox OS to not harm the Web.
OK, I agree.
But exposing to the web is a proper way to test it and see if it's useful for the community and to get feedback.
I'm not a big fan of the 'inputmethodstatechange' event and the
'inputStarted' attribute. First of all, the event name is way too long
and developers don't like long name. 'statechange' would be enough.
Shorter and simple event name will be better, but the name should tell
the developer that it is an IME event.
This event is sent on an object named inputMethodConnection implementing
the InputMethodConnection interface. I believe it should be clear the
event is related to 'inputmethod'.
It is reasonable. So we can use a shorter name without confusion.
I suggest `imestatechange`
When does a session finish? Does it finish when leaving the input? If a session starts when entering an input or end when leaving the input, the maybe we can only have change and finish since we need an input to start, starting events are like change events.
Also, 'inputStarted' could be replaced by 'enabled'
maybe? Basically, the keyboard application would know if it is currently
being in use.

I think "started" will be more precise than "enabled". An IME could be enabled and disabled in the settings page. If an IME is enabled, user could switch to that IME and use it. So "enabled" means the IME is available to use. User will "start" an enabled IME after focusing an input field.
I suggest `selected` or the even more explanatory `userSelected` or `currentlySelected` because it make obvious what Yuen is trying to explain.
Besides the load and unload events, we may need additional 'start',
'change' and 'finish' events.
To be honest, I wonder if the start event is really needed. When the
keyboard application is started, it will be obviously used for a field
so a 'start' event is useless here. If the keyboard got dismissed and is
re-open again without being actually killed, we might as well just send
a 'modify' event. Another option could be to use a system message.
I'm not sure those are great ideas but probably worth thinking about.
I think both "start" and "modify" are necessary.
"start" is much like an "onfocus" event. It will be fired when user focuses a new input field and the keyboard should reset its state to meet the need of the input field.
For example, determines the keyboard layout by the input type.  If the keyboard got dismissed and is re-open again without being actually killed, we need send a 'start' event as well.

'modify' event is sent when the input field is changed by external actions and the keyboard should not reset its state. For example, there is a text field with a clear button inside. The default mode of the keyboard is lower case mode. I change the keyboard to upper case mode and start inputting. During inputting, I find a mistake and use the clear button to clear the text field. The keyboard will get notified through the "modify" and keep the lower case mode as I don't switching input field. If we use "start" event instead, the keyboard will be reset to lower case mode.
I'm a little bit lost between events and reviewing the etherpad is not helping me. So we need to detect the following actions:

  1.  The IME is selected by the user
  2.  The IME takes control of an input
  3.  The IME leaves an input
  4.  The input being edit has changed externally
  5.  Another IME is selected

Is this Ok? If so, we could use the following events:

  1.  focus - because it resembles the API for common elements
  2.  sessionstart - an edit session is started
  3.  sessionend - an edit session is finished
  4.  contentchange - to make explicit the content has changed
  5.  blur - same as 1

We could add another one saying `sessionswitch` when the user changes from one editable to another and it means to close the first session and to open a new one.
This is a little bit more tricky than it sounds. We are coined by the usual way to address these languages but the point here is we could leave all this logic up to the application. I don't know why we need API support for this. For instance, the former user case. If I want wo to be highlighted then I can keep the bounds of the word, then use `replaceSurroundingText()`.

If `setComposingText()` exists (because you need it to highlight the word), it should accept cursor positions, in this way we are abstracting from "words". Note we already have a method to make a selection via `selectionStart` and `selectionEnd` attributes. Then `replaceWord()` is equivalent to call `replaceSurroundingText()` using current selection values.
We can provide a boolean as parameter or alternative names to enable both uses.
Regards!

________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx
Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/30/13 12:53 PM
On 2013/4/29 15:34, Salvador de la Puente González wrote:
 > On 28/04/13 20:46, Yuan Xulei wrote:
 > But exposing to the web is a proper way to test it and see if it's
useful for the community and to get feedback.
 > I suggest `imestatechange`
 > When does a session finish? Does it finish when leaving the input? If
a session starts when entering an input or end when leaving the input,
the maybe we can only have change and finish since we need an input to
start, starting events are like change events.
The session starts when entering an input and end when leaving the
input. "change"(or "modify") events are fired when the input has changed
externally during a input session. More specifically, a "change" event
could be content change event, seletion change event and so on.
 > I suggest `selected` or the even more explanatory `userSelected` or
`currentlySelected` because it make obvious what Yuen is trying to explain.
 > I'm a little bit lost between events and reviewing the etherpad is
not helping me. So we need to detect the following actions:
 >
 >     The IME is selected by the user
 >     The IME takes control of an input
 >     The IME leaves an input
 >     The input being edit has changed externally
 >     Another IME is selected
 >
The IME app doesn't need to know if it is selected or deselected by the
user, since it is the keyboard manager's responsibility for loading and
unloading IMEs. I think we only need to detect actions from 2 to 4.
 >
 > Is this Ok? If so, we could use the following events:
 >
 >     focus - because it resembles the API for common elements
 >     sessionstart - an edit session is started
 >     sessionend - an edit session is finished
 >     contentchange - to make explicit the content has changed
 >     blur - same as 1
 >
If we only need actions 2-4, focus and sessionstart events can be
combined to focus event, while sessionend and blur events can be
combined to blur event.
The we will get:
1. focus -  an edit session is starte
2. contentchange - to make explicit the content has changed
3. blur - an edit session is finished
 >
 > We could add another one saying `sessionswitch` when the user changes
from one editable to another and it means to close the first session and
to open a new one.
How about sending `sessionend` followed by `sessionstart` without
introducing `sessionswitch`?
 >
 >> First use case:
 >> - the user types 'wo' and keyboard calls setComposingText to make it
as underlined composint text -- '_wo_'.
 >> - the keyboard application calls replaceWord('我') which is going to
 >> replace the composing text ('_wo_') with '我'.
Sorry I made a mistake here, `replaceWord('我')` should be
`replaceSurroundingText(('我')`. I don't think we need `replaceWord`.
 > This is a little bit more tricky than it sounds. We are coined by the
usual way to address these languages but the point here is we could
leave all this logic up to the application. I don't know why we need API
support for this. For instance, the former user case. If I want wo to be
highlighted then I can keep the bounds of the word, then use
`replaceSurroundingText()`.
 >
 > If `setComposingText()` exists (because you need it to highlight the
word), it should accept cursor positions, in this way we are abstracting
from "words". Note we already have a method to make a selection via
`selectionStart` and `selectionEnd` attributes. Then `replaceWord()` is
equivalent to call `replaceSurroundingText()` using current selection
values.
 > We can provide a boolean as parameter or alternative names to enable
both uses.
 > Regards!
 >
 > Este mensaje se dirige exclusivamente a su destinatario. Puede
consultar nuestra política de envío y recepción de correo electrónico en
el enlace situado más abajo.
 > This message is intended exclusively for its addressee. We only send
and receive email on the basis of the terms set out at:
 > http://www.tid.es/ES/PAGINAS/disclaimer.aspx
Thanks

Yuan
More topics »