Swipeselection prototype

[Sergi Mansilla]

Hi,

I've implemented a basic version of "swipe selection" (original concept video: http://youtu.be/RGQTaHGQ04Q) that is already useful. You can see a rough video I've made showing it in the unagi device here:

http://youtu.be/uDzb7HZY2mE

I've done that by modifying keyboard.js. One of the first problems I encountered is that I can't directly access the input element that the keyboard is hooked to. It seems to be handled natively by mozKeyboard, since the keyboard just calls mozKeyboard.sendKey() eventually. I worked around that by sending left and right arrow signals to move the cursor around.

The main problems I have now are that it becomes trickier to implement selection from Gaia, since I can't access ranges in the input element. I also noticed that my hack doesn't work on input elements inside web pages (accessed from the browser).

Is there a way to work around these issues? Or even better, is there a way to implement it properly? Perhaps I am missing some essential link between the keyboard and the target input element.

You can find the code that implements swipe selection here: https://github.com/comoyo/gaia/compare/master...swipe_selection

Thanks for your help!

Sergi

[Etienne Segonzac]

On Thu, Feb 21, 2013 at 11:53 AM, Sergi Mansilla <se...@comoyo.com> wrote:
> I've implemented a basic version of "swipe selection" (original concept video: http://youtu.be/RGQTaHGQ04Q) that is already useful. You can see a rough video I've made showing it in the unagi device here:
>
> http://youtu.be/uDzb7HZY2mE

You *completely* made my day!
This is so good, and with a 40 lines patch: \o/
--
Etienne

[Salvador de la Puente González]

Hello Sergi

I'm very impressed with your path. Take a look at the new Keyboard API
draft [1] and you'll find most of your doubts solved (but remember, it
is just a draft).

I like it so much I'm going to send it to UX mailing list to take a look ;)

Thanks!

[1] https://wiki.mozilla.org/WebAPI/KeboardIME

On 21/02/13 13:14, Etienne Segonzac wrote:
> On Thu, Feb 21, 2013 at 11:53 AM, Sergi Mansilla <se...@comoyo.com> wrote:

>> I've implemented a basic version of "swipe selection" (original concept video: http://youtu.be/RGQTaHGQ04Q) that is already useful. You can see a rough video I've made showing it in the unagi device here:
>>
>> http://youtu.be/uDzb7HZY2mE

> You *completely* made my day!
> This is so good, and with a 40 lines patch: \o/
> --
> Etienne

> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia
>
>


________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

[Sergi Mansilla]

Hi Salvador,

I'm glad you like it. I will definitely be looking at the new API, it looks good on a first glance. Any estimates on when can we expect it in B2G? I can't wait to play with it, it looks like even implementing selection will be a breeze!

Thanks,

Sergi

[Sergi Mansilla]

Thanks! I'm glad you like it :)

[Rick Waldron]

This is great!


What kind of feedback are you looking for, or are you even ready for
feedback yet?

Rick

[Josh Carpenter]

Badass! Nice work, Sergi. This looks like it could be an excellent improvement to our fairly limited v1 input tools. I'd love to have the UX team try it out and evaluate. What's the best way to make that happen? Gordon / Casey, would it be possible to work with Sergi to get his patch loaded onto a device?

—
Josh Carpenter
UX Designer
Mozilla

[Chris Lee]

Agree, this is awesome. Thanks Sergi.

If our UX team can test it out and we can have someone review the code, this would be very slick to land in an upcoming release.

Chris

On Feb 21, 2013, at 2:47 PM, Josh Carpenter wrote:

> Badass! Nice work, Sergi. This looks like it could be an excellent improvement to our fairly limited v1 input tools. I'd love to have the UX team try it out and evaluate. What's the best way to make that happen? Gordon / Casey, would it be possible to work with Sergi to get his patch loaded onto a device?
>
> —
> Josh Carpenter
> UX Designer
> Mozilla
>
> On Feb 21, 2013, at 2:53 AM, Sergi Mansilla <se...@comoyo.com> wrote:
>

[Andreas Gal]

That is seriously bad ass. Our Taipei team did the WebAPI for the keyboard. Maybe that needs some extra features. Rudy, Tim?

Andreas

On Feb 21, 2013, at 2:47 PM, Josh Carpenter <jcarp...@mozilla.com> wrote:

> Badass! Nice work, Sergi. This looks like it could be an excellent improvement to our fairly limited v1 input tools. I'd love to have the UX team try it out and evaluate. What's the best way to make that happen? Gordon / Casey, would it be possible to work with Sergi to get his patch loaded onto a device?
>
> —
> Josh Carpenter
> UX Designer
> Mozilla
>
> On Feb 21, 2013, at 2:53 AM, Sergi Mansilla <se...@comoyo.com> wrote:
>

[Tim Chien]

Rudy,

Can you tell Sergi here that does our proposed new Keyboard API comes with
cursor position and selection range?


Tim

On Fri, Feb 22, 2013 at 8:08 AM, Andreas Gal <g...@mozilla.com> wrote:

>
> That is seriously bad ass. Our Taipei team did the WebAPI for the
> keyboard. Maybe that needs some extra features. Rudy, Tim?
>
> Andreas
>
> On Feb 21, 2013, at 2:47 PM, Josh Carpenter <jcarp...@mozilla.com>
> wrote:
>
> > Badass! Nice work, Sergi. This looks like it could be an excellent
> improvement to our fairly limited v1 input tools. I'd love to have the UX
> team try it out and evaluate. What's the best way to make that happen?
> Gordon / Casey, would it be possible to work with Sergi to get his patch
> loaded onto a device?
> >
> > —
> > Josh Carpenter
> > UX Designer
> > Mozilla
> >
> > On Feb 21, 2013, at 2:53 AM, Sergi Mansilla <se...@comoyo.com> wrote:
> >

> >> _______________________________________________
> >> dev-gaia mailing list
> >> dev-...@lists.mozilla.org
> >> https://lists.mozilla.org/listinfo/dev-gaia
> >
> > _______________________________________________
> > dev-gaia mailing list
> > dev-...@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-gaia
>
>

--
Tim Guan-tin Chien, Senior Front-end Dev., Mozilla Corp. (Taiwan)

[Rudy Lu]

Hi all,

As Salva pointed out, the current draft Keyboard WebAPI is documented in
[1].
And we have considered this as one of the new features.
* The VKB app should be able to move the cursor position or select a
specified range of text.
(See https://wiki.mozilla.org/WebAPI/KeboardIME#Features)

And the function is:
// Set the selection range of the the editable text.
// Note that the start position should be less or equal to the end
position.
// To move the cursor, set the start and end position to the same value.
void setSelectionRange(in long start, in long end);
Please refer to [1] for more details.

-----
Keyboard WebAPI is not ready to make it into Gecko/Gaia code base.
But Xulei has done the implementation and demo of the MozKeyboard API, so
we may leverage his work to do some experiments.
- gecko
checkout the ime branch of
https://github.com/yxl/mozilla-central.git
- gaia
checkout the keyboard-test branch of
https://github.com/yxl/gaia.git


[1] https://wiki.mozilla.org/WebAPI/KeboardIME

--
Sergi, thanks for the great demo.
Please let me know if you have any questions.
Thank you.

Best regards,
Rudy, Mozilla Taiwan

On Fri, Feb 22, 2013 at 10:20 AM, Tim Chien <timd...@mozilla.com> wrote:

> Rudy,
>
> Can you tell Sergi here that does our proposed new Keyboard API comes with
> cursor position and selection range?
>
>
> Tim
>
>
> On Fri, Feb 22, 2013 at 8:08 AM, Andreas Gal <g...@mozilla.com> wrote:
>
> >
> > That is seriously bad ass. Our Taipei team did the WebAPI for the
> > keyboard. Maybe that needs some extra features. Rudy, Tim?
> >
> > Andreas
> >
> > On Feb 21, 2013, at 2:47 PM, Josh Carpenter <jcarp...@mozilla.com>
> > wrote:
> >
> > > Badass! Nice work, Sergi. This looks like it could be an excellent
> > improvement to our fairly limited v1 input tools. I'd love to have the UX
> > team try it out and evaluate. What's the best way to make that happen?
> > Gordon / Casey, would it be possible to work with Sergi to get his patch
> > loaded onto a device?
> > >
> > > —
> > > Josh Carpenter
> > > UX Designer
> > > Mozilla
> > >
> > > On Feb 21, 2013, at 2:53 AM, Sergi Mansilla <se...@comoyo.com> wrote:
> > >

[Rudy Lu]

On 2013/2/22, at 上午10:20, Tim Chien <timd...@mozilla.com> wrote:

> Rudy,
>
> Can you tell Sergi here that does our proposed new Keyboard API comes with cursor position and selection range?
>
>
> Tim
>
>
> On Fri, Feb 22, 2013 at 8:08 AM, Andreas Gal <g...@mozilla.com> wrote:
>
> That is seriously bad ass. Our Taipei team did the WebAPI for the keyboard. Maybe that needs some extra features. Rudy, Tim?
>
> Andreas
>
> On Feb 21, 2013, at 2:47 PM, Josh Carpenter <jcarp...@mozilla.com> wrote:
>
> > Badass! Nice work, Sergi. This looks like it could be an excellent improvement to our fairly limited v1 input tools. I'd love to have the UX team try it out and evaluate. What's the best way to make that happen? Gordon / Casey, would it be possible to work with Sergi to get his patch loaded onto a device?
> >
> > —
> > Josh Carpenter
> > UX Designer
> > Mozilla
> >
> > On Feb 21, 2013, at 2:53 AM, Sergi Mansilla <se...@comoyo.com> wrote:
> >

[Jan Jongboom]

I understand that the new API would solve quite some of the problems, but can we get a quick fix by enabling cursor movement in normal <input/> fields and <textarea/>'s? This way we have at least proper movement there. Don't know the time impact on this of course :-)

[Jan Jongboom]

On Friday, February 22, 2013 4:23:48 AM UTC+1, Rudy Lu wrote:

[Sergi Mansilla]

Thanks!

In order to get it into a device it should be as easy as installing the swiping gaia branch (https://github.com/comoyo/gaia/tree/swipe_selection) into a device with `make-install`. I'd be happy to help to get it working if necessary.

Cheers,

Sergi

[Sergi Mansilla]

I'd love to have it in the main branch and be able to use it already. It would be great if UX could give me feedback on it.

As a side note, I tried to implement it without breaking any existing UX functionality, for example, you can still press a key and slide your finger to the nearest keys in case your first key was a mistake, just like before. Let me know if I should file it as a bug in bugzilla in case you want to go ahead with the review.

Sergi

[Sergi Mansilla]

I'd love to have it in the main branch and be able to use it already. It would be great if UX could give me feedback on it.

As a side note, I tried to implement it without breaking any existing UX functionality, for example, you can still press a key and slide your finger to the nearest keys in case your first key was a mistake, just like before. Let me know if I should file it as a bug in bugzilla in case you want to go ahead with the review.

Sergi

On Friday, February 22, 2013 12:33:28 AM UTC+1, Chris Lee wrote:

[Sergi Mansilla]

I believe the new API solves most of the issues. Actually, my problems were not that much with the API, but also with unexpected behavior. For example, input fields inside of HTML pages in the browser would not work, and I suspect that they are treated in a different way internally, but I don't know yet. I had a similar thing with textareas. If somebody could shed some light on these cases I'd be grateful.

Sergi

[Sergi Mansilla]

I believe the new API solves most of the issues. Actually, my problems were not that much with the API, but also with unexpected behavior. For example, input fields inside of HTML pages in the browser would not work, and I suspect that they are treated in a different way internally, but I don't know yet. I had a similar thing with textareas. If somebody could shed some light on these cases I'd be grateful.

Sergi

On Friday, February 22, 2013 1:08:40 AM UTC+1, Andreas Gal wrote:

[Sergi Mansilla]

Hi Rudy,

This is great. When I have some time I'll try to implement it on top of the branches with the new keyboard API. Do I compile b2g in mozilla-central just as I compile the one in mozilla-b2g?

Thanks,

Sergi

[Chris Lee]

On Feb 22, 2013, at 4:14 AM, Sergi Mansilla wrote:

> As a side note, I tried to implement it without breaking any existing UX functionality, for example, you can still press a key and slide your finger to the nearest keys in case your first key was a mistake, just like before. Let me know if I should file it as a bug in bugzilla in case you want to go ahead with the review.

This was the main concern I had, but glad you were able to address it.

Josh, let's chat next week to see if there are any other usability gaps we may want to fix before landing.

Chris

[Josh Carpenter]

Sounds good. We're working on getting this loaded onto a phone as soon as possible so we can evaluate while in Barcelona. In addition to you and I from Product and UX, who is the right engineering gatekeeper to evaluate this for inclusion in upcoming releases?

Sent from my iPad

On Feb 22, 2013, at 10:24 AM, Chris Lee <cl...@mozilla.com> wrote:

>
> On Feb 22, 2013, at 4:14 AM, Sergi Mansilla wrote:
>

>> As a side note, I tried to implement it without breaking any existing UX functionality, for example, you can still press a key and slide your finger to the nearest keys in case your first key was a mistake, just like before. Let me know if I should file it as a bug in bugzilla in case you want to go ahead with the review.
>

[Chris Lee]

I'll chat with Andreas today and follow up in BCN.

Chris

On Feb 22, 2013, at 2:55 PM, Josh Carpenter wrote:

> Sounds good. We're working on getting this loaded onto a phone as soon as possible so we can evaluate while in Barcelona. In addition to you and I from Product and UX, who is the right engineering gatekeeper to evaluate this for inclusion in upcoming releases?
>
> Sent from my iPad
>
> On Feb 22, 2013, at 10:24 AM, Chris Lee <cl...@mozilla.com> wrote:
>
>>
>> On Feb 22, 2013, at 4:14 AM, Sergi Mansilla wrote:
>>

>>> As a side note, I tried to implement it without breaking any existing UX functionality, for example, you can still press a key and slide your finger to the nearest keys in case your first key was a mistake, just like before. Let me know if I should file it as a bug in bugzilla in case you want to go ahead with the review.
>>

[Rudy Lu]

Hi Sergi,

Yes, you may put the mozilla-central to replace gecko/ folder when you
build Firefox OS for device.
Or you may want to customize the build process with userconfig [1].

[1]
https://developer.mozilla.org/en-US/docs/Mozilla/Boot_to_Gecko/Customization_with_the_.userconfig_file

I am kind of curious on how you want to achieve this:
>> without breaking any existing UX functionality, you can still press a

key and slide your finger to the nearest keys

Are we going to use 2-finger gesture for the swipe cursor moving or
something?

--

Please let me know if you have any questions.
Thank you.


Best regards,
Rudy, Mozilla Taiwan

[Yuan Xulei]

Hi all,

A bug was created to support for the keyboard apps to access and modify
the selection range of the input element.

The patch will be created later.

On 02/23/2013 07:04 PM, Rudy Lu wrote:
> Hi Sergi,
>
> Yes, you may put the mozilla-central to replace gecko/ folder when you
> build Firefox OS for device.
> Or you may want to customize the build process with userconfig [1].
>
> [1]
> https://developer.mozilla.org/en-US/docs/Mozilla/Boot_to_Gecko/Customization_with_the_.userconfig_file
>
> I am kind of curious on how you want to achieve this:
> >> without breaking any existing UX functionality, you can still
> press a key and slide your finger to the nearest keys
>
> Are we going to use 2-finger gesture for the swipe cursor moving or
> something?
>
> --

> Please let me know if you have any questions.
> Thank you.
>
>
> Best regards,
> Rudy, Mozilla Taiwan
>
>

> On Fri, Feb 22, 2013 at 8:37 PM, Sergi Mansilla <se...@comoyo.com

> <jcarp...@mozilla.com <mailto:jcarp...@mozilla.com>>

> >
> > > > wrote:
> >
> > > >
> >
> > > > > Badass! Nice work, Sergi. This looks like it could be an
> excellent
> >
> > > > improvement to our fairly limited v1 input tools. I'd love
> to have the UX
> >
> > > > team try it out and evaluate. What's the best way to make
> that happen?
> >
> > > > Gordon / Casey, would it be possible to work with Sergi to
> get his patch
> >
> > > > loaded onto a device?
> >
> > > > >
> >
> > > > > —
> >
> > > > > Josh Carpenter
> >
> > > > > UX Designer
> >
> > > > > Mozilla
> >
> > > > >
> >
> > > > > On Feb 21, 2013, at 2:53 AM, Sergi Mansilla

> <mailto:dev-...@lists.mozilla.org>

> >
> > > > >> https://lists.mozilla.org/listinfo/dev-gaia
> >
> > > > >
> >
> > > > > _______________________________________________
> >
> > > > > dev-gaia mailing list
> >

> > > > > dev-...@lists.mozilla.org <mailto:dev-...@lists.mozilla.org>

> >
> > > > > https://lists.mozilla.org/listinfo/dev-gaia
> >
> > > >
> >
> > > >
> >
> > >
> >
> > >
> >
> > > --
> >
> > > Tim Guan-tin Chien, Senior Front-end Dev., Mozilla Corp. (Taiwan)
> >
> > > _______________________________________________
> >
> > > dev-gaia mailing list
> >

> > > dev-...@lists.mozilla.org <mailto:dev-...@lists.mozilla.org>

> >
> > > https://lists.mozilla.org/listinfo/dev-gaia
> >
> > >
>
> _______________________________________________
> dev-gaia mailing list

> dev-...@lists.mozilla.org <mailto:dev-...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-gaia
>
>

[Yuan Xulei]

The bug number is 844716.

Or you can view by the link:

https://bugzilla.mozilla.org/show_bug.cgi?id=844716

On 02/25/2013 03:30 PM, Yuan Xulei wrote:
> Hi all,
>
> A bug was created to support for the keyboard apps to access and
> modify the selection range of the input element.
>
> The patch will be created later.
>
> On 02/23/2013 07:04 PM, Rudy Lu wrote:
>> Hi Sergi,
>>
>> Yes, you may put the mozilla-central to replace gecko/ folder when
>> you build Firefox OS for device.
>> Or you may want to customize the build process with userconfig [1].
>>
>> [1]
>> https://developer.mozilla.org/en-US/docs/Mozilla/Boot_to_Gecko/Customization_with_the_.userconfig_file
>>
>> I am kind of curious on how you want to achieve this:
>> >> without breaking any existing UX functionality, you can still
>> press a key and slide your finger to the nearest keys
>>
>> Are we going to use 2-finger gesture for the swipe cursor moving or
>> something?
>>
>> --

>> Please let me know if you have any questions.
>> Thank you.
>>
>>
>> Best regards,
>> Rudy, Mozilla Taiwan
>>
>>

>> On Fri, Feb 22, 2013 at 8:37 PM, Sergi Mansilla <se...@comoyo.com
>> <mailto:se...@comoyo.com>> wrote:
>>

>> <jcarp...@mozilla.com <mailto:jcarp...@mozilla.com>>

>> >
>> > > > wrote:
>> >
>> > > >
>> >
>> > > > > Badass! Nice work, Sergi. This looks like it could be an
>> excellent
>> >
>> > > > improvement to our fairly limited v1 input tools. I'd love
>> to have the UX
>> >
>> > > > team try it out and evaluate. What's the best way to make
>> that happen?
>> >
>> > > > Gordon / Casey, would it be possible to work with Sergi to
>> get his patch
>> >
>> > > > loaded onto a device?
>> >
>> > > > >
>> >
>> > > > > —
>> >
>> > > > > Josh Carpenter
>> >
>> > > > > UX Designer
>> >
>> > > > > Mozilla
>> >
>> > > > >
>> >
>> > > > > On Feb 21, 2013, at 2:53 AM, Sergi Mansilla

>> <mailto:dev-...@lists.mozilla.org>

>> >
>> > > > >> https://lists.mozilla.org/listinfo/dev-gaia
>> >
>> > > > >
>> >
>> > > > > _______________________________________________
>> >
>> > > > > dev-gaia mailing list
>> >
>> > > > > dev-...@lists.mozilla.org

>> <mailto:dev-...@lists.mozilla.org>

>> >
>> > > > > https://lists.mozilla.org/listinfo/dev-gaia
>> >
>> > > >
>> >
>> > > >
>> >
>> > >
>> >
>> > >
>> >
>> > > --
>> >
>> > > Tim Guan-tin Chien, Senior Front-end Dev., Mozilla Corp. (Taiwan)
>> >
>> > > _______________________________________________
>> >
>> > > dev-gaia mailing list
>> >

>> > > dev-...@lists.mozilla.org <mailto:dev-...@lists.mozilla.org>

>> >
>> > > https://lists.mozilla.org/listinfo/dev-gaia
>> >
>> > >
>>
>> _______________________________________________
>> dev-gaia mailing list

>> dev-...@lists.mozilla.org <mailto:dev-...@lists.mozilla.org>
>> https://lists.mozilla.org/listinfo/dev-gaia
>>
>>

[Jan Jongboom]

Great, Sergi will be on holiday the coming two weeks but I am available for input or changes if necessary.

[Sergi Mansilla]

I'm back from holidays and available to help with this.

Also, it doesn't seem that anybody is currently reviewing https://bugzilla.mozilla.org/show_bug.cgi?id=844716. Should it be assigned to another reviewer?

Sergi

[Dietrich Ayala]

Yes! I'll cc some people.

> https://lists.mozilla.org/listinfo/dev-gaia

[Sergi Mansilla]

I've created an issue in bugzilla to implement it with the new API, assigned to me: https://bugzilla.mozilla.org/show_bug.cgi?id=851072

Sergi

[Sergi Mansilla]

I've created an issue in bugzilla to implement it with the new API, assigned to me: https://bugzilla.mozilla.org/show_bug.cgi?id=851072

Sergi

On Tuesday, March 12, 2013 6:02:59 PM UTC+1, Dietrich Ayala wrote: