## Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app

Showing 1-28 of 28 messages
 Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Yuan Xulei 11/8/12 11:20 PM Hi all, In order to enhance the virtual keyboard for more commonly used feature on the Android and the iOS platforms, and also to support the 3rd-party keyboard app after v1, we(Rudy, Tim and I) are trying to design a new mozKeyboard API. I created a MoPad to list and discuss our proposal:   https://etherpad.mozilla.org/gaia-mozkeyboard-v3 The draft API of the our proposal are listed bellow, which is similar to that of the android. == KeyboardManager == When user focuses an editable text field which requires the soft keyboard, an "imestatechange" event will be fired. The keyboard app should listen to the  "inputmethodstatechane" events by   the KeyboardManager#oninputmethodstatechane attribute. Then it check the KeyboardManager#connection attribute. If the KeyboardManager#connection  is not null, it's time to start input or else finish input. The full interface of KeyboardManager: partial interface Navigator { readonly attribute KeyboardManager mozKeyboard; }; interface KeyboardManager {      // Show a (system) menu for all input method that allow user to select      void showInputMethodPicker();      // Switch directly to the next input method      void switchToNextInputMethod();      // Hide and show the soft keyboard      void hideSoftKeyboard();      void showSoftKeyboard();      // Not null when start input readonly attribute KeyboardConnection? connection;      // User begins or finishes editing a field. attribute Function oninputmethodstatechane;      // User moves the cursor, changes the selection, or alers the composing text length attribute Function onselectionchange; }; == KeyboardConnection  == The KeyboardManager#connection  is an interface of KeyboardConnection that  defines the communication channel from the keyboard to the app that  is receiving user input. We  use the KeyboardConnection  to read and edit the value of the input field. interface KeyboardConnection {      // Send a keyevent to the input field that is currently attached. It exists in the current interface      void sendKey(in long keyCode, in long charCode);      // Commit text to the input field to current cursor position and clear composing text.      void commitText(DOMString text);      // Set the value of the current input field.  It exists in the current interface and is used for "special situations where the value had to be chosen amongst a list  (type=month) or a widget (type=date, time, etc.)".      // Check https://groups.google.com/group/mozilla.dev.webapi/browse_thread/thread/56cdfe1c6bfd34dcfor details.      void setValue(DOMString text);      // Get "length" characters of text before the current cursor position. DOMString getTextBeforeCursor(in long length);      // Get "length" characters of text after the current cursor position. DOMString getTextAfterCursor(in long length);      // Delete text around the cursor      void deleteSurroundingText(in long beforeLength, in long afterLength);      //  Set the composing text around the current cursor position as the parameter "text"      void setComposingText(DOMString text);      // Clear composing text      void finishComposingText();      // The start and stop position of the selection. The start position is the current cursor position. To set the cursor position, set the start and end position tthe same value. attribute long selectionStart; attribute long selectionEnd; readonly attribute DOMString selectedText;      // The type of the input field, including text, number, password, url, tel and email. readonly attribute DOMString inputType;      // Get the return key type, which determines the title of the key displayed in the keyboard and action performed when it is pressed. The value could be "done", "next", "go", "search" and "send".      // If the value is "next", when pressing the enter key, the focus will be move to the next input field. readonly attribute DOMString returnKeyType;      // Used to specifiy a customized label for the return key. attribute DOMString returnKeyLabel;      // The capitalization mode, which could be one of these types: none, words, sentences  and characters. readonly attribute DOMString capitalizationMode; }; == KeyboardSelectionChangeEvent == To moniter the changes of the input field, the keyboard app could listen to the  "selectionchange" events by  the KeyboardManager#onselectionchange attribute. An object of KeyboardSelectionChangeEvent will be passed when a "selectionchange" event is fired. interface KeyboardSelectionChangeEvent: Event { readonly attribute long oldSelectionStart; readonly attribute long oldSelectionEnd; readonly attribute long newSelectionStart; readonly attribute long newSelectionEnd; readonly attribute long composingTextStart; readonly attribute long composingTextEnd; }; sample code: // Called when the user starts or finishes editing an input field navigator.mozKeyboard.oninputmethodstatechane= function(event) {      // Get the KeyboardConnection instance      var conn = navigator.mozKeyboard.connection;      if (!conn) {          // finishes editing return;      }      // Send a string to user's input field      if (conn.inputType == "text") { conn.commitText("Hello world");      } else if (conn.inputType == "url") { conn.commitText("http://www.mozilla.org");      } else { conn.commitText("123");      } }; Please let me know if you have any questions. Thanks. Regards, Yuan Xulei Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Ben Francis 11/16/12 9:20 AM Hi, I was at a Career Fair at Cambridge University yesterday and was talking to an engineer from SwiftKey (swiftkey.net), a London-based startup who make an alternative keyboard for Android. Their advice was not to make our keyboard API anything like Android's! I have pointed them towards this thread to give them an opportunity to provide further feedback. Ben On Fri, Nov 9, 2012 at 7:20 AM, Yuan Xulei wrote: > Hi all, > > In order to enhance the virtual keyboard for more commonly used feature on > the Android and the iOS platforms, and also to support the 3rd-party > keyboard app after v1, we(Rudy, Tim and I) are trying to design a new > mozKeyboard API. > > I created a MoPad to list and discuss our proposal: >  https://etherpad.mozilla.org/**gaia-mozkeyboard-v3 > > The draft API of the our proposal are listed bellow, which is similar to > that of the android. > > == KeyboardManager == > > When user focuses an editable text field which requires the soft keyboard, > an "imestatechange" event will be fired. > > The keyboard app should listen to the  "inputmethodstatechane" events by >  the KeyboardManager#**oninputmethodstatechane attribute. Then it check >     // Check https://groups.google.com/**group/mozilla.dev.webapi/** > browse_thread/thread/**56cdfe1c6bfd34dcfordetails. > to the  "selectionchange" events by  the KeyboardManager#**onselectionchange > attribute. > An object of KeyboardSelectionChangeEvent will be passed when a > "selectionchange" event is fired. > > interface KeyboardSelectionChangeEvent: Event { > readonly attribute long oldSelectionStart; > readonly attribute long oldSelectionEnd; > readonly attribute long newSelectionStart; > readonly attribute long newSelectionEnd; > readonly attribute long composingTextStart; > readonly attribute long composingTextEnd; > }; > > sample code: > > // Called when the user starts or finishes editing an input field > navigator.mozKeyboard.**oninputmethodstatechane= function(event) { >     // Get the KeyboardConnection instance >     var conn = navigator.mozKeyboard.**connection; >     if (!conn) { >         // finishes editing > return; >     } >     // Send a string to user's input field >     if (conn.inputType == "text") { > conn.commitText("Hello world"); >     } else if (conn.inputType == "url") { > conn.commitText("http://www.**mozilla.org "); >     } else { > conn.commitText("123"); >     } > }; > > > Please let me know if you have any questions. > Thanks. > > Regards, > > Yuan Xulei > ______________________________**_________________ > dev-webapi mailing list > dev-w...@lists.mozilla.org > https://lists.mozilla.org/**listinfo/dev-webapi > -- Ben Francis http://tola.me.uk Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 11/19/12 1:20 AM Hi Ben, Thanks for your advices, but could please help to provide more details on why not to be like android. We borrowed some API from the android because it fits our needs and has been tested by 3-rd IMEs on android.  Also it will be easier for the developer to learn the IME framework and port the existing android IMEs to Firefox OS. If there exists limitation and flaws with the draft API, please feel free to  point it out and make it better. Regards, Yuan Xulei On 11/17/2012 01:20 AM, Ben Francis wrote: > Hi, > > I was at a Career Fair at Cambridge University yesterday and was > talking to an engineer from SwiftKey (swiftkey.net > ), a London-based startup who make an alternative > keyboard for Android. > > Their advice was not to make our keyboard API anything like Android's! > > I have pointed them towards this thread to give them an opportunity to > provide further feedback. > > Ben > > On Fri, Nov 9, 2012 at 7:20 AM, Yuan Xulei > wrote: > >     Hi all, > >     In order to enhance the virtual keyboard for more commonly used >     feature on the Android and the iOS platforms, and also to support >     the 3rd-party keyboard app after v1, we(Rudy, Tim and I) are >     trying to design a new mozKeyboard API. > >     I created a MoPad to list and discuss our proposal: >     https://etherpad.mozilla.org/gaia-mozkeyboard-v3 > >     The draft API of the our proposal are listed bellow, which is >     similar to that of the android. > >     == KeyboardManager == > >     When user focuses an editable text field which requires the soft >     keyboard, an "imestatechange" event will be fired. > >     The keyboard app should listen to the  "inputmethodstatechane" >     events by  the KeyboardManager#oninputmethodstatechane attribute. >     KeyboardManager#onselectionchange attribute. >     An object of KeyboardSelectionChangeEvent will be passed when a >     "selectionchange" event is fired. > >     interface KeyboardSelectionChangeEvent: Event { >     readonly attribute long oldSelectionStart; >     readonly attribute long oldSelectionEnd; >     readonly attribute long newSelectionStart; >     readonly attribute long newSelectionEnd; >     readonly attribute long composingTextStart; >     readonly attribute long composingTextEnd; >     }; > >     sample code: > >     // Called when the user starts or finishes editing an input field >     navigator.mozKeyboard.oninputmethodstatechane= function(event) { >         // Get the KeyboardConnection instance >         var conn = navigator.mozKeyboard.connection; >         if (!conn) { >             // finishes editing >     return; >         } >         // Send a string to user's input field >         if (conn.inputType == "text") { >     conn.commitText("Hello world"); >         } else if (conn.inputType == "url") { >     conn.commitText("http://www.mozilla.org"); >         } else { >     conn.commitText("123"); >         } >     }; > > >     Please let me know if you have any questions. >     Thanks. > >     Regards, > >     Yuan Xulei >     _______________________________________________ >     dev-webapi mailing list >     dev-w...@lists.mozilla.org Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 1/7/13 7:00 PM Hi, I made some modification and updated the API according to Salvador's advices(https://etherpad.mozilla.org/gaia-mozkeyboard-v3). A new MoPad was created to comment and discuss the updated API: https://etherpad.mozilla.org/gaia-mozkeyboard-v4. Please feel free to add your thoughts and comment. = Major changes from previous version 1. Change the name of the mozKeyboard interface to InputMethodConnection and separate the InputMethodConnection from the InputMethodManager interface. The InputMethodConnection interface should be only visible to the input method app. 2. Remove the method(showSoftKeyboard) to explicitly show the keyboard, while keep the method(removeFocus) to explicitly hide the keyboard. User could explicitly show the keyboard by focusing a text field, so there is no need to provide an extra method to show the keyboard. 3. remove finishComposingText The composing text could be cleared by: setComposingText(''); 4. remove selectedText attribute 5. Add asynchronous method getText to get the text content of the text field. Remove the methods(getTextBeforeCursor, getTextAfterCursor) to get text around the cursor. 6. Add methods to switch focus onto the previous/next input field. = API Proposal: partial interface Navigator {      readonly attribute InputMethodManager mozInputMethodManager;      readonly attribute InputMethodConnection mozInputMethodConnection; }; == InputMethodConnection  == The interface defines the communication channel from the keyboard to the app that  is receiving user input. The IME app  use it to read and edit the value of the input field. interface InputMethodConnection {      // User begins or finishes editing a field.      // If the user change input directly from one input to another, only one event will be dispatched.      attribute Function oninputmethodstatechange;      // Whether the user begins editing.      readonly attribute boolean inputStarted;      // User moves the cursor, changes the selection, or alters the composing text length      attribute Function onselectionchange;      // Send a keyevent to the input field that is currently attached. It exists in the current interface      void sendKey(in long keyCode, in long charCode, in long modifiers);      // Insert the given text at the cursor position. Replace the selected text if any and clear the composing text.      void commitText(in DOMString text);      // Set the text value of the current input field.      void setText(in DOMString text);      // Get the text value of the current input field asynchronously.      void getText(in long start, in long end, in Function callback);      // Length of the content      readonly attribute long textLength;      // Delete text around the cursor      void deleteSurroundingText(in long beforeLength, in long afterLength);      // Set the composing text before the current cursor position.      // To clear the composing text, set the text paramter to the empty string.      void setComposingText(in DOMString text);      // The start and stop position of the selection.      readonly attribute long selectionStart;      readonly attribute long selectionEnd;      // Set the selection range of the the editable text.      // Note that the start position should be less or equal to the end position.      // To move the cursor, set the start and end position to the same value.      void setSelectionRange(in long start, in long end);      // Clear the focus of the current input field and hide the keyboard.      void removeFocus();      // Focus the next text field.      void advanceFocus();      // Focus the previous text field.      void rewindFocus();      // This read only attribute is true if user can advance the focus to the next      // text field.      readonly attribute boolean canAdvanceFocus;      // This readonly attribute is true if user can rewind the focus back to the      // previous text field.      readonly attribute boolean canRewindFocus;      // The input mode string.      readonly attribute DOMString inputMode;      // The type of the input field, including text, number, password, url, tel and email.      readonly attribute DOMString inputType;      // Get the return key type, which determines the title of the key displayed in the keyboard and action performed when it is pressed. The value could be "done", "next", "go", "search" and "send".      // If the value is "next", when pressing the enter key, the focus should be move to the next input field.      readonly attribute DOMString returnKeyType;      // Get the customized label for the return key.      readonly attribute DOMString returnKeyLabel; }; sample code: var conn = navigator.mozInputMethodConnection; // Called when the user starts or finishes editing an input field conn.oninputmethodstatechange= function(event) {      if (conn.inputStarted) {          // Text input should start      } else {          // Text input should end      } }; // Insert a string at the current cursor position conn.commitText('Hello world'); // Get the selected text conn.getText(conn.selectionStart, conn.selectionEnd, function callback(text) {      var selectedText = text; }); // Move the cursor position var position = 10; conn.setSelectionRange(position, position); // Hide the keyboard conn.removeFocus(); // Switch the focus onto the next input field if possible if (conn.canAdvanceFocus) { conn.advanceFocus(); } On 11/27/2012 02:36 AM, Ben Francis wrote: > On Mon, Nov 19, 2012 at 9:20 AM, Yuan Xulei > wrote: > >     Thanks for your advices, but could please help to provide more >     details on why not to be like android. > > > I have no idea, I've never built an IME. This is just what the > engineer from SwiftKey said, and they do have this experience :) I'm > hoping they can provide further feedback. > > Ben Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 1/31/13 9:38 AM Hi all, A wiki page of the extended MozKeyboard API was created: https://wiki.mozilla.org/WebAPI/KeboardIME The patch of the implementation could be found in bug 737110: https://bugzilla.mozilla.org/show_bug.cgi?id=737110 Hope it will help and the API is in great need of comments/suggestions. Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 2/6/13 7:37 AM Hello guys. I saw this bug referring to keyboard: https://bugzilla.mozilla.org/show_bug.cgi?id=838308 In short, any application can use mozKeyboard.onfocuschange and *setValue()* method to inject text in the target input, in any other application.  From my point of view, there is no problem to keep mozKeyboard open without permission but only if the sender of the *setValue()* message is in the same domain than the receiver. We should control these kind of situations when designing the new API. Of course, the sender could be the keyboard application which receives special permission to inject text to any other input, whatever its domain was. Cheers! ________________________________ Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo. This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at: http://www.tid.es/ES/PAGINAS/disclaimer.aspx Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 2/6/13 6:19 PM The mozKeyboard API currently is used by the keyboard app to generate keyboard events and interact with input target. IMO, It should be restrict to the keyboard apps and normal app has no permission to use it. And the keyboard app could be declared by the manifest file, for example giving a special permission in the manifest file. Furthermore, if there are more than one keyboard apps enabled, only the active one in the foreground is able to inject text to other apps. Yuan Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 2:21 AM Hello guys, We're working on implementing the new mozKeyboard API:    https://wiki.mozilla.org/WebAPI/KeboardIME Some changes are made according to people's advice and our requirements, including: 1. Combined commitText with deleteSurroundingText into replaceSurroundingText. 2. Add ontextchange attribute to allow IME get notifications when the text content has been changed. 3. Combined returnKeyType with returnKeyLabel into returnKey. If you have any question about the API, please let us know before we finish the implementation work. We're waiting for your feedback. FYI, The previous version of the proposed API is https://wiki.mozilla.org/WebAPI/KeboardIME?title=WebAPI/KeboardIME&oldid=510135. Thanks, Yuan Xulei Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Anne van Kesteren 4/15/13 2:34 AM On Mon, Apr 15, 2013 at 10:21 AM, Yuan Xulei wrote: > We're working on implementing the new mozKeyboard API: >   https://wiki.mozilla.org/WebAPI/KeboardIME FWIW, it seems Microsoft might have an interest similar to ours: http://lists.w3.org/Archives/Public/public-webapps/2013JanMar/1059.html We might also want to tell the WebApps list what we're working on to see if long term we can work out something that works the same everywhere. -- http://annevankesteren.nl/ Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/15/13 3:03 AM Hello people. In-line commentaries: Some changes are made according to people's advice and our requirements, including: 1. Combined commitText with deleteSurroundingText into replaceSurroundingText. INMHO, the name is very long but I understand is very explicative. I suggest some replacements but may be other people can suggest a better (shorter) wording:   *   alterText() - my favourite   *   changeText()   *   replaceText() When I was developing the keyboard I did not find any situation requiring changing the surrounding text but I can be different when developing suggestion or other IMEs. If so, the I have nothing to say but if not, I do not fully understand why to include these "surrounding" methods. May be we can use something similar to splice in Array methods. Some like: alterText(newText, fromIndex, toIndex, cursorRelative) - replace text between fromIndex to toIndex (not included) with newText. If cursorRelative is set to true fromIndex and toIndex are treated like beforeLength and afterLength. But do whatever you want with this proposal. ;) 2. Add ontextchange attribute to allow IME get notifications when the text content has been changed. What kind of information is included in this event? When this event is triggered? If I commit text from the keyboard, is it triggered as well? 3. Combined returnKeyType with returnKeyLabel into returnKey. I would keep type and label separated, may be in the same attribute but I think this should be an object (or interface) with different fields for type and label. Remember, if you are here in Madrid and want to talk remember I'm in the 6th floor. Cheers. Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Ehsan Akhgari 4/15/13 7:24 AM Hi Yuan, Here's my feedback based on which I believe is the latest version of the proposal (please correct me if I'm wrong.) 1. InputMethodManager is not defined anywhere as far as I can see. 2. oninputmethodstatechange, onselectionchange and ontextchange should be EventHandlers (and InputMethodConnection should be an EventTarget). 3. As Jonas mentioned in the bug, I think that we should make sure that the API doesn't make assumptions about the text edit control. Specifically, setText for example will not be useful for a contentEditable element (and probably not even for textareas).  If you need an API to clear the text, you should add that, but the interaction with multi-line text editing in contentEditable/textarea needs to be thought out. 4. returnKey's value can be a localized string, right?  In that case, I don't think that special casing the behavior for "done" is the right thing to do, we should just add another boolean attribute to expose what happens when pressing the return key.  Also, why is returnKey readonly? 5. What is the use case for the beforeLength and afterLength arguments to replaceSurroundingText? 6. Do we want to expose some information about text change events, such as the before/after text, the offset of the text inside the editable field, etc.?  Same question for onselectionchange. 7. selectionStart and selectionEnd need to be unsigned long. 8. Can you please rewrite the proposal in Web IDL, and drop the moz prefixes? Thanks! Ehsan Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 8:50 AM On 04/15/2013 06:03 PM, Salvador de la Puente González wrote: > Hello people. > > In-line commentaries: > >> Some changes are made according to people's advice and our >> requirements, including: >> 1. Combined commitText with deleteSurroundingText into >> replaceSurroundingText. > INMHO, the name is very long but I understand is very explicative. I > suggest some replacements but may be other people can suggest a better > (shorter) wording: > >   * alterText() - /my favourite/ >   * changeText() >   * replaceText() >        I prefer alterText and replaceText. > > When I was developing the keyboard I did not find any situation > requiring changing the surrounding text but I can be different when > developing suggestion or other IMEs. If so, the I have nothing to say > but if not, I do not fully understand why to include these > "surrounding" methods. > The most three common actions of an IME  is inserting word, making suggestion and auto-correction.  These actions are usually taken at or around the cursor. Take auto-correction for example, when user selects a misspelled word and chooses to correct it, the IME needs to replace the misspelled word around the cursor. Since most of the time the IME only cares about the text around the cursor,  we include these "surrounding" method. > > May be we can use something similar to splice in Array methods. Some like: > > /alterText(newText, fromIndex, toIndex, cursorRelative)/ - replace > text between /fromIndex/ to /to//Index/ (not included) with /newText/. > If /cursorRelative/ is set to true /fromIndex/ and /toIndex/ are > treated like /beforeLength/ and /afterLength/. > > But do whatever you want with this proposal. ;) > >> 2. Add ontextchange attribute to allow IME get notifications when the >> text content has been changed. > What kind of information is included in this event?      Nothing. It should include the changed text, but for contentEditable element, the text may be huge. Thus it requires IME to invokes getText  method to get the changed text. > When this event is triggered?      A new input field get focused. The text of the input field has been changed by key events, commit text, delete text,  JS or other operations. > If I commit text from the keyboard, is it triggered as well? Yes, it is. >> 3. Combined returnKeyType with returnKeyLabel into returnKey. > I would keep type and label separated, may be in the same attribute > but I think this should be an object (or interface) with different > fields for type and label. I like the idea of storing type and label as an object. > > Remember, if you are here in Madrid and want to talk remember I'm in > the 6th floor. I'm not in Madrid ;-( > > Cheers. > > ------------------------------------------------------------------------ Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 9:27 AM On 04/15/2013 10:24 PM, Ehsan Akhgari wrote: > Hi Yuan, > > Here's my feedback based on > which I believe is the > latest version of the proposal (please correct me if I'm wrong.) > > 1. InputMethodManager is not defined anywhere as far as I can see. Still not defined yet. > > 2. oninputmethodstatechange, onselectionchange and ontextchange should > be EventHandlers (and InputMethodConnection should be an EventTarget). > > 3. As Jonas mentioned in the bug, I think that we should make sure > that the API doesn't make assumptions about the text edit control. > Specifically, setText for example will not be useful for a > contentEditable element (and probably not even for textareas).  If you > need an API to clear the text, you should add that, but the > interaction with multi-line text editing in contentEditable/textarea > needs to be thought out. I'm consider removing setText from API as it is only used for some special cases now, such as setting the value of a date time field, but I'm not sure if there exist any other cases we need it. > > 4. returnKey's value can be a localized string, right?       Yes. > In that case, I don't think that special casing the behavior for > "done" is the right thing to do, we should just add another boolean > attribute to expose what happens when pressing the return key.  Also, > why is returnKey readonly?      The behaviour and title of the return key are specified by the input field and the IME can't change it. For example, a search box could set its return key to 'search' with the code . > > > 5. What is the use case for the beforeLength and afterLength arguments > to replaceSurroundingText? To correct a word, user would replace the word before or around the cursor. > > 6. Do we want to expose some information about text change events, > such as the before/after text, the offset of the text inside the > editable field, etc.? Currently no information will be included. The before/after text and text offset will be useful for making suggestion and auto-correction, may be we could expose some of them without degrading the system performance. > Same question for onselectionchange. The selectionStart and selectionEnd information will be enough, I think. > > 7. selectionStart and selectionEnd need to be unsigned long. > > 8. Can you please rewrite the proposal in Web IDL, and drop the moz > prefixes? > > Thanks! > Ehsan I'll rewrite the proposal in Web IDL and fix the issues you mentioned. Thanks! Yuan Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/15/13 10:02 AM Hello Yuan >>> 2. Add ontextchange attribute to allow IME get notifications when >>> the text content has been changed. >> What kind of information is included in this event? >     Nothing. It should include the changed text, but for > contentEditable element, the text may be huge. Thus it requires IME to > invokes getText  method to get the changed text. Here we can add just the information about if this event was triggered by this session or not. In order to distinguish between IME operations or external action. My suggest is to add 'external' flag set to true if the event is not triggered by the edit session. That's all. ________________________________ Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/15/13 10:56 AM Hi Salvador, That will be better. Or maybe only send event triggered by external? And let the IME to track the edit session. Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/15/13 11:10 AM Even better. I always can trigger a custom event If I want to listen for my own modifications in this fashion. Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Ehsan Akhgari 4/15/13 12:23 PM On 2013-04-15 12:27 PM, Yuan Xulei wrote: >> 4. returnKey's value can be a localized string, right? >       Yes. >> In that case, I don't think that special casing the behavior for >> "done" is the right thing to do, we should just add another boolean >> attribute to expose what happens when pressing the return key.  Also, >> why is returnKey readonly? >      The behaviour and title of the return key are specified by the > input field and the IME can't change it. For example, a search box could > set its return key to 'search' with the code  returnKey='search' />. OK, but your proposal says: If the value is "next", when pressing the enter key, the focus should be move to the next input field. I don't think this is a good idea since the web app might set returnKey to the translation of "next" in the target locale, and that would break the above behavior. Also, is the returnKey attribute spec'ed somewhere?  How is one supposed to specify it for contentEditable/designMode? >> 5. What is the use case for the beforeLength and afterLength arguments >> to replaceSurroundingText? > To correct a word, user would replace the word before or around the cursor. Wouldn't it be better to have a replaceWord(index) API instead, which will replace the word that |index| would fall into? I can see some complications with using this API for large text boxes that are getting modified by both the keyboard and content script. Tracking the correct offsets doesn't seem very easy in that case. >> 6. Do we want to expose some information about text change events, >> such as the before/after text, the offset of the text inside the >> editable field, etc.? > Currently no information will be included. The before/after text and > text offset will be useful for making suggestion and auto-correction, > may be we could expose some of them without degrading the system > performance. OK. Cheers, Ehsan Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/16/13 2:03 AM On 04/16/2013 03:23 AM, Ehsan Akhgari wrote: > On 2013-04-15 12:27 PM, Yuan Xulei wrote: >>> 4. returnKey's value can be a localized string, right? >>       Yes. >>> In that case, I don't think that special casing the behavior for >>> "done" is the right thing to do, we should just add another boolean >>> attribute to expose what happens when pressing the return key.  Also, >>> why is returnKey readonly? >>      The behaviour and title of the return key are specified by the >> input field and the IME can't change it. For example, a search box could >> set its return key to 'search' with the code > returnKey='search' />. > > OK, but your proposal says: > > If the value is "next", when pressing the enter key, the focus should > be move to the next input field. > > I don't think this is a good idea since the web app might set > returnKey to the translation of "next" in the target locale, and that > would break the above behavior. Just as Salvador suggested, we could add a separate label property to specify the return key label. > > Also, is the returnKey attribute spec'ed somewhere? No, it's a new attribute and we should define it in this API. > How is one supposed to specify it for contentEditable/designMode? Add a returnKey attribute to the contentEditable element or the document body for the design mode. >>> 5. What is the use case for the beforeLength and afterLength arguments >>> to replaceSurroundingText? >> To correct a word, user would replace the word before or around the >> cursor. > > Wouldn't it be better to have a replaceWord(index) API instead, which > will replace the word that |index| would fall into? For CJK IME, words are not separated by space. For example,  the chinese sentence--我吃饭了(I had a meal). So I incline to provide a flexible API to let IME implement the function of replaceWord. Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/17/13 3:12 AM To remove any confusion, the Keyboard IME API and the IME API from WebApps are totally different. They do not support the same use cases. The former is to create virtual keyboards/IME using the Web Platform and the later, as far as I understand it, is to show the system's IME in web applications using canvas. For example, a textfield in a game. -- Mounir Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/17/13 9:52 AM Hi, First of all, thank you for working on this Yuan. The API looks generally good, but I have a few comments. I think it would be good to clearly mention as a UC (Use Case) that this API is there for dealing with text but not to deal with other form controls like . > > I am not sure what InputMethodManager is there for given that it is not > defined. Is that intentional? Yes, it will defined later. > > Also, we would like to no longer prefix stuff as much as we can. I would > be in favour to not push this API to Firefox OS until we have a rough > consensus on it and not prefix it with 'moz'. I agree with you. > We should not expose it to > other platforms anyway, right? Why not? The desktop and mobile browser may benefit from the API. We can provide virtual keyboard to other platforms as well. > > I'm not a big fan of the 'inputmethodstatechange' event and the > 'inputStarted' attribute. First of all, the event name is way too long > and developers don't like long name. 'statechange' would be enough. Shorter and simple event name will be better, but the name should tell the developer that it is an IME event. > I also have a hard time understanding what this 'inputStarted' is there > for. It seems there only to know if the 'statechange' event was fired > when we entered or left a text field. Is that correct? Yes. > Should we simply > have some kind of 'start'/'change'/'finish' events? I actually wonder > how the Keyboard application is expected to be created. Shouldn't we > make the 'start' event a system message and maybe the 'finish' event > should not exist because we might expect the application to be killed at > that point? Though, I guess it might be platform dependant. The life-cycle of the Keyboard application is: 1. create | 2. start <----- | | 3. change focus an input field | | 4. finish ----- | 5. destroy The Keyboard manager will create the keyboard app, which will run as a normal app and receive the load event on start-up. When user focus an input field, the keyboard app starts text input. During the input, the content or attributes of input field may be changed by external actions(for example, JS code to clear the text), the keyboard app should get notification about the changes. After user finishes input of current input field, the keyboard app will wait for the user to focus another input field. So there needs a least two types of events: one fired when user switches between input fields, and the other fired when the current input field has changed. Besides the load and unload events, we may need additional 'start', 'change' and 'finish' events. > > It seems that replaceSurroundingText could be replaced by replaceWord > unless there are some UCs I'm missing. Inserting some text maybe? But I > don't see in which case we want to insert more that one character at a > time without replacing the current word? For CJK, the words are not separated by spaces. Each sentence could be viewed as a word. replaceWord doesn't work for CJK IME. And for Latin, when doing suggestions, we often need to insert some text to complete a word, for example, input 'En' and get the suggestion of 'English'. > > I feel that setComposingText UCs can be handled by replaceWord. If > replaceWord takes by the default the word at the cursor position and we > consider that in the following case "foo|" the cursur is position on the > word "foo", wouldn't setComposingText be exactly like replaceWord? Yes, replaceWord or replaceSurroundingText can be used to add a string to the input field as well. But the setComposingText is not used to add string to the input field. It is used to handle composition events for CJK IMEs. For example, to input the Chinese word '我', we will input the composing text 'wo' first, then convert it to corresponding Chinese character. Also sometimes we use composing text to help us input special characters, such as mathematical symbols. For example, the composing text of Σ is '\sum'. You can find more about composing text in the section "2.1 Compose" of https://dvcs.w3.org/hg/ime-api/raw-file/default/Overview.html#composer-section. > > Why do we have getText() instead of always exposing the text inside the > field? Is that to improve performance? Yes, to avoid transfer large text between apps, sine for the content editable element, the text content may be several KB. But I think we could change it to a sync method without callback. > > What is the UC for removeFocus() all the keyboards I have seen are > dismissed by the system, not by themselves. Is there a particular UI in > mind? removeFocus could support a hide button for the keyboard. If the keyboard support full screen editing, it will need a hide button to cancel editing. > > advanceFocus() and rewindFocus() are interesting because they seem to > apply on the entire document but all the keyboards I have used would > make that feature work in a form, not in the document. Is that > applicable in the document on purpose? Yes, applicable in the document, but the app itself could the focus be restricted in the forms. > Also, this would skip some form > elements, which I believe a user wouldn't want. Yes, non-text field will be skipped. Those methods just follow the behaviour of that on android and iOS. > If we include that, I think we should make those methods going to the > next/previous form control in the current form unless we have a strong > use case to go to the next/previous text field in the entire document. > > I would like to keep inputMode outside of this API for the moment > because we do not have a clear and specified story for that. > Also, inputType could, maybe, be named 'type'. > > Finally, returnKey will not be added as an attribute on form controls. I > do not think this is needed and people will reject that proposal if the > use case is a Keyboard API. Actually, what can't we do with the > information we already have? If type='search', you can use 'search', if > there is no form control after, you can use 'done' or 'go' (you might > need to know if the form is submittable for that though). I do not see > what 'send' is there for. > Anyway, the idea is that we should give enough context to the keyboard > so it can show its own UI based on those information. It will complicate for the keyboard to infer the return key meaning by the context. Take the awesome bar of firefox browser for example, if user inputs a keyword, the return key will be 'search'; if user inputs a URL, the return key will be 'go'. Keybard can hardly determine the return keyboard type by the content of the awesome bar. 'send' is used for sending a mail, SMS, instance message and so on. Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app "Yuan Xulei(袁徐磊)" 4/18/13 3:42 PM Should the IME get current language from system settings? Or each input element could specify its language regardless of the system settings? Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/19/13 2:00 AM On 19/04/13 00:42, Yuan Xulei wrote: > Should the IME get current language from system settings? Or each input > element could specify its language regardless of the system settings? HTMLElement.lang can be used to specify the language of a web site or a specific part of a website. [1] http://www.whatwg.org/specs/web-apps/current-work/multipage/elements.html#htmlelement -- Mounir Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Mounir Lamouri 4/19/13 2:50 AM On 19/04/13 00:35, Yuan Xulei wrote: > On 04/18/2013 12:52 AM, Mounir Lamouri wrote: >> Hi, >> >> First of all, thank you for working on this Yuan. The API looks >> generally good, but I have a few comments. >> >> I think it would be good to clearly mention as a UC (Use Case) that this >> API is there for dealing with text but not to deal with other form >> controls like . >>> >>> I am not sure what InputMethodManager is there for given that it is not >>> defined. Is that intentional? >> Yes, it will defined later. > Please, define at least what would be the purpose. > >>> We should not expose it to >>> other platforms anyway, right? >> Why not? The desktop and mobile browser may benefit from the API. We can >> provide virtual keyboard to other platforms as well. > I doubt any other platforms will use that API any time soon. Pushing it > to other platforms will increase the exposure of this API. We should > keep it inside Firefox OS to not harm the Web. OK, I agree. >>> I'm not a big fan of the 'inputmethodstatechange' event and the >>> 'inputStarted' attribute. First of all, the event name is way too long >>> and developers don't like long name. 'statechange' would be enough. >> Shorter and simple event name will be better, but the name should tell >> the developer that it is an IME event. > This event is sent on an object named inputMethodConnection implementing > the InputMethodConnection interface. I believe it should be clear the > event is related to 'inputmethod'. It is reasonable. So we can use a shorter name without confusion. >>> Should we simply >>> have some kind of 'start'/'change'/'finish' events? I actually wonder >>> how the Keyboard application is expected to be created. Shouldn't we >>> make the 'start' event a system message and maybe the 'finish' event >>> should not exist because we might expect the application to be killed at >>> that point? Though, I guess it might be platform dependant. >> The life-cycle of the Keyboard application is: >> >> 1. create >> | >> 2. start <----- >> | | >> 3. change focus an input field >> | | >> 4. finish ----- >> | >> 5. destroy >> >> The Keyboard manager will create the keyboard app, which will run as a >> normal app and receive the load event on start-up. When user focus an >> input field, the keyboard app starts text input. During the input, the >> content or attributes of input field may be changed by external >> actions(for example, JS code to clear the text), the keyboard app should >> get notification about the changes. After user finishes input of current >> input field, the keyboard app will wait for the user to focus another >> input field. >> >> So there needs a least two types of events: one fired when user switches >> between input fields, and the other fired when the current input field >> has changed. > I'm not sure if you are agreeing or disagreeing with my proposal. It > seems that 'start', 'modify' and 'finish' would work correctly here, > wouldn't they? Yes, I agree. > Also, 'inputStarted' could be replaced by 'enabled' > maybe? Basically, the keyboard application would know if it is currently > being in use. > I think "started" will be more precise than "enabled". An IME could be enabled and disabled in the settings page. If an IME is enabled, user could switch to that IME and use it. So "enabled" means the IME is available to use. User will "start" an enabled IME after focusing an input field. >> Besides the load and unload events, we may need additional 'start', >> 'change' and 'finish' events. > To be honest, I wonder if the start event is really needed. When the > keyboard application is started, it will be obviously used for a field > so a 'start' event is useless here. If the keyboard got dismissed and is > re-open again without being actually killed, we might as well just send > a 'modify' event. Another option could be to use a system message. > I'm not sure those are great ideas but probably worth thinking about. I think both "start" and "modify" are necessary. "start" is much like an "onfocus" event. It will be fired when user focuses a new input field and the keyboard should reset its state to meet the need of the input field. For example, determines the keyboard layout by the input type.  If the keyboard got dismissed and is re-open again without being actually killed, we need send a 'start' event as well. 'modify' event is sent when the input field is changed by external actions and the keyboard should not reset its state. For example, there is a text field with a clear button inside. The default mode of the keyboard is lower case mode. I change the keyboard to upper case mode and start inputting. During inputting, I find a mistake and use the clear button to clear the text field. The keyboard will get notified through the "modify" and keep the lower case mode as I don't switching input field. If we use "start" event instead, the keyboard will be reset to lower case mode. >>> It seems that replaceSurroundingText could be replaced by replaceWord >>> unless there are some UCs I'm missing. Inserting some text maybe? But I >>> don't see in which case we want to insert more that one character at a >>> time without replacing the current word? >> For CJK, the words are not separated by spaces. Each sentence could be >> viewed as a word. replaceWord doesn't work for CJK IME. >> And for Latin, when doing suggestions, we often need to insert some text >> to complete a word, for example, input 'En' and get the suggestion of >> 'English'. > For the latin case, you could simply do replaceWord('English') and that > would replace 'En' with 'English'. > For the CJK case, I do not know anything about that but I wonder how the > keyboard application would be able to know that something is a word if > the UA can't event know that. Are CJK users editing their previously > typed stuff and except to have auto-correction? or that is something > only latin users expect? The keyboard application uses complicated algorithm and large dictionary to split a sentence into words. The CJK users seldom use auto-correction, but they often use suggestion to complete a word. They need a method to append string to current cursor. >>> I feel that setComposingText UCs can be handled by replaceWord. If >>> replaceWord takes by the default the word at the cursor position and we >>> consider that in the following case "foo|" the cursur is position on the >>> word "foo", wouldn't setComposingText be exactly like replaceWord? >> Yes, replaceWord or replaceSurroundingText can be used to add a string >> to the input field as well. But the setComposingText is not used to add >> string to the input field. It is used to handle composition events for >> CJK IMEs. For example, to input the Chinese word '我', we will input the >> composing text 'wo' first, then convert it to corresponding Chinese >> character. Also sometimes we use composing text to help us input special >> characters, such as mathematical symbols. For example, the composing >> text of Σ is '\sum'. You can find more about composing text in the >> section "2.1 Compose" of >> https://dvcs.w3.org/hg/ime-api/raw-file/default/Overview.html#composer-section. > First use case: > - the user types 'wo'; > - the keyboard application calls replaceWord('我') which is going to > replace the current word ('wo') with '我'. The style of the composing text(usually underlined) is different from the committed text.  And the composing text is temporary and can be cancelled by dismissing the keyboard or moving the cursor. It is not the real text that user inputs. So we need a method to handle this special kind of text. First use case: - the user types 'wo' and keyboard calls setComposingText to make it as underlined composint text -- '_wo_'. - the keyboard application calls replaceWord('我') which is going to replace the composing text ('_wo_') with '我'. > > Second use case: > - the user types '\sum'; > - the keyboard application calls replaceWord('Σ') which is going to > replace the current word ('\sum') with 'Σ'. > > Why wouldn't that work? > >>> Why do we have getText() instead of always exposing the text inside the >>> field? Is that to improve performance? >> Yes, to avoid transfer large text between apps, sine for the content >> editable element, the text content may be several KB. But I think we >> could change it to a sync method without callback. > I think we should keep that async. If the concern is to save > performance, we shouldn't make that synchronous. Also, we should > probably use: > Future getText(start, end); I agree. >>> What is the UC for removeFocus() all the keyboards I have seen are >>> dismissed by the system, not by themselves. Is there a particular UI in >>> mind? >> removeFocus could support a hide button for the keyboard. If the >> keyboard support full screen editing, it will need a hide button to >> cancel editing. > I've never seen such UI. On Android, I'm able to dismiss a virtual > keyboard from the system UI but I never saw a virtual keyboard offering > a button to dismiss itself even when they are fullscreen. > Do you have specific UIs in mind? Are we planning to implement that in > our UI? On Android, the Sogou IME(the most popular Chinese Pinyin IME) has a small hide button at the end the candidates view.  And I see the default keyboard of  some tablets of Android 4.0  has a hide button. On iPhone, although there is no hide button, user can dismiss the keyboard by swiping the screen. So I think we may provide such API to 3rd party IME, but we don't have a plan to implement in our UI right now. >>> advanceFocus() and rewindFocus() are interesting because they seem to >>> apply on the entire document but all the keyboards I have used would >>> make that feature work in a form, not in the document. Is that >>> applicable in the document on purpose? >> Yes, applicable in the document, but the app itself could the focus be >> restricted in the forms. > I didn't catch that. > >>> Also, this would skip some form >>> elements, which I believe a user wouldn't want. >> Yes, non-text field will be skipped. Those methods just follow the >> behaviour of that on android and iOS. > I haven't seen that behaviour on Android. I doubt the iOS behaviour is > that great. I think the form behaviour is way better but we could also > design the API such as a keyboard can implement both. If we could implement the form behaviour, that will be great. But we'd better provide a simple and straightforward API. >>> Finally, returnKey will not be added as an attribute on form controls. I >>> do not think this is needed and people will reject that proposal if the >>> use case is a Keyboard API. Actually, what can't we do with the >>> information we already have? If type='search', you can use 'search', if >>> there is no form control after, you can use 'done' or 'go' (you might >>> need to know if the form is submittable for that though). I do not see >>> what 'send' is there for. >>> Anyway, the idea is that we should give enough context to the keyboard >>> so it can show its own UI based on those information. >> It will complicate for the keyboard to infer the return key meaning by >> the context. Take the awesome bar of firefox browser for example, if >> user inputs a keyword, the return key will be 'search'; if user inputs a >> URL, the return key will be 'go'. Keybard can hardly determine the >> return keyboard type by the content of the awesome bar. >> >> 'send' is used for sending a mail, SMS, instance message and so on. > The problem is that first of all, you can't expect web pages to use that > magic returnKey attribute just because the Keyboard API uses it. If I > recall correctly, Firefox Android tried to use this and switched back to > guess the label based on the content. > Second of all, standardizing this API is going to be hard. If we have to > rely on a new magic attribute in HTML, this is going to be even harder. > The HTML specification will never accept that change. If we keep that > work self-contained, we increase the chances to have it standardized. If > that happen, we would have a stronger point if we go to the HTML WG and > ask for a new attribute for the virtual keyboards. > > Cheers, > -- > Mounir > _______________________________________________ > Regards, Yuan Re: Proposal: mozKeyboard API for extending built-in keyboard and supporting 3rd-party Keyboard app Salvador de la Puente González 4/29/13 12:34 AM On 28/04/13 20:46, Yuan Xulei wrote: On 04/19/2013 05:50 PM, Mounir Lamouri wrote: On 19/04/13 00:35, Yuan Xulei wrote: On 04/18/2013 12:52 AM, Mounir Lamouri wrote: Hi, First of all, thank you for working on this Yuan. The API looks generally good, but I have a few comments. I think it would be good to clearly mention as a UC (Use Case) that this API is there for dealing with text but not to deal with other form controls like
More topics »