Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Dynamic Touch Target for Gaia keyboard
73 views
Skip to first unread message
John Hu
Sep 25, 2013, 5:24:27 AM9/25/13
to dev-gaia, Evan Tseng, Greg Weng, Rudy Lu, Tim Chien
Hi all,
With the direction of Rudy, Greg and John had finished a fast prototype of Dynamic Touch Target[1]. The live video can be found at http://www.youtube.com/watch?v=eMFfmG8jhRg. We have to admit that it is not a real keyboard but just a demo app. We had demoed it to few people in Taipei, including engineers and UXs. The feedbacks from them are positive about this.
TL;DR
------------
The core algorithm of this dynamic area keyboard is based on Voronoi Diagram. The idea is from a paper called "Touchscreen Software Keyboard for Finger Typing"[2]. In this paper, they used Voronoi diagram to draw the dynamic keyboard, called CATKey, based on touching position. The adaptation of CATKey is made by the user behaviors, see section 3.3. When touching points of a key are moving to left, the keyboard moves its touching area to left. This behavior may be non-intentional. But our version is to predict the next possible keys and enlarge the size of them. We know there are lots ways to improve the accuracy of user input. Our method is to use Voronoi Diagram and word statistics to enlarge the next prossible keys.
The procedures of the dynamic touching target are 1. key prediction, 2. possibility to x-y location for all keys, 3. keys’ position to keys’ polygon, 4. drawing keys’ polygon, 5. touch detecting, 6. learning, and 7. presenting the candidate keys.
1. key dictionary
We didn’t include accurate dictionary in this fast prototype. The dictionary in this version is the statistic of characters based on 4000 frequently used words. In this dictionary, we use a look-up table for each key strike, take "thesis" as an example, we may have "t", "th", "the", "thes", and "thesi" as our key. That is contexual based. Each of them follows by a list of characters with possibility, from 1 to 3. And larger is the higher weight. We did not think this part thoroughly. It’s a fast prototype for proof of concept.
However, this mechanism gives us more opportunities to improve the user experiences under various contexts. It’s even possible to switch the forecasting dictionary when user types in different input element, like the E-mail or URL. Of course, how to build and use the dictionary would always be a serious issue, especially when we must consider it with the performance and accuracy.
2. possibility to x-y location for all keys
We try to use an algorithm as simple as possible to have better performance. In this fast prototype, we just divide the horizontal space for all keys. When a key with higher weight, we add 1 to 3 extra space to them. For example, if user typed th, the ’e’ and ’i’ has the higher possibility to show. We weight ’e’ for 1 and weight ’i’ for 2. And we suppose the horizontal space for the whole keyboard is 320. The original space for each keys at first row, qwertyuiop, is 32 pixel for each. After the weighting, it is divided by 13, including the weight for ’e’ and ’i’, that is 24 pixel for each. But it is 48px for ’e’, and 72px for ’i’. Once we have corresponding size for each key, we need to convert them to the center point. That’s for Voronoi diagram.
3. keys’ position to keys’ polygon
In this prototype, we use d3[4] to calculate the Voronoi diagram for us. It returns a list of polygon points for each key. You may wonder that why not to use the size calculated at step 2 and why we need to run voronoi diagram. The main reason is to have vertical resizing. It looks like a hexagon for each key, originally. Once we change the weight, the vertical size of the hexagon are also affected slightly. If we use rectangle, the outcome may be a rearrangement of horizontal space and no changes on vertical space. As we known, the Voronoi diagram is trying to find the middle perpendicular for two points. With this, the horizontal neighbors of weighted key is also enlarged about 1/4. If we use the example above for ’e’ key, the size of ’w’ and ’r’ is enlarged as 30px (24px + 6px) and the size of ’e’ is about 36px (24px + 12px). With this symptom, user may feel the ’e’ pops out. The same feelings also happen to the key ’i’. If we dealing with more complex cases, including vertical weighting, the Voronoi diagram helps more. Another advantage is to fill the dividing remainders for all keys.
4. drawing key polygon (optional)
In this prototype, we use SVG path to draw the polygon, without any performance tuning. This step may/should be optional. Users may be confused when the keyboard is changing frequently. Drawing it in this prototype may help us to know if the current behavior is correct.
It may be nice to make the dynamic layout as a invisible touching area above the static keyboard layout, so that user can type without any possibly inconveniences from the variable layout but still benefits from the adjustable touching area. We need more information from UX team and others’ feedback to decide which method is better.
5. touch detecting
Once having SVG path, we use click event to detect user touch. We know we should use touch end event to dispatch the key. It may not be supported in SVG. But for a fast prototype, we choose the easy way to implement it. There is another alternative for it, map element, of course.
6. learning
We haven’t thought thoroughly about it and not included in this prototype. But it is possible to have a dynamic changing on prediction table. For example, we could have a component to dynamically train our dictionaries to make the layout more suitable for every single user, and mixing it with our online updating dictionaries as well.
7. presenting the candidate keys
In this prototype, we color candidate keys with red, and users feel helpful to have the clues for the next key they should press. Of course, the visual effect we used here is coarse and could be refined, like to use a light border instead of filling the key with single color, but this additional trick may help our keyboard even greater. It can be a potential way to improve the user experience beside enlarge the keys.
The repository may be found at [1]. We also made a video for it, which may be found at [3].
Finally, thanks Rudy and Evan for more information about this fast prototype.
Thanks
John Hu
[1] https://github.com/snowmantw/gaia/tree/vk/test_apps/voronoi-keyboard
[2] http://scholar.google.com/scholar?cluster=11097390483940419161&as_sdt=0,5
[3] http://www.youtube.com/watch?v=cRP4NQTN1qo
[4] http://d3js.org
With the direction of Rudy, Greg and John had finished a fast prototype of Dynamic Touch Target[1]. The live video can be found at http://www.youtube.com/watch?v=eMFfmG8jhRg. We have to admit that it is not a real keyboard but just a demo app. We had demoed it to few people in Taipei, including engineers and UXs. The feedbacks from them are positive about this.
TL;DR
------------
The core algorithm of this dynamic area keyboard is based on Voronoi Diagram. The idea is from a paper called "Touchscreen Software Keyboard for Finger Typing"[2]. In this paper, they used Voronoi diagram to draw the dynamic keyboard, called CATKey, based on touching position. The adaptation of CATKey is made by the user behaviors, see section 3.3. When touching points of a key are moving to left, the keyboard moves its touching area to left. This behavior may be non-intentional. But our version is to predict the next possible keys and enlarge the size of them. We know there are lots ways to improve the accuracy of user input. Our method is to use Voronoi Diagram and word statistics to enlarge the next prossible keys.
The procedures of the dynamic touching target are 1. key prediction, 2. possibility to x-y location for all keys, 3. keys’ position to keys’ polygon, 4. drawing keys’ polygon, 5. touch detecting, 6. learning, and 7. presenting the candidate keys.
1. key dictionary
We didn’t include accurate dictionary in this fast prototype. The dictionary in this version is the statistic of characters based on 4000 frequently used words. In this dictionary, we use a look-up table for each key strike, take "thesis" as an example, we may have "t", "th", "the", "thes", and "thesi" as our key. That is contexual based. Each of them follows by a list of characters with possibility, from 1 to 3. And larger is the higher weight. We did not think this part thoroughly. It’s a fast prototype for proof of concept.
However, this mechanism gives us more opportunities to improve the user experiences under various contexts. It’s even possible to switch the forecasting dictionary when user types in different input element, like the E-mail or URL. Of course, how to build and use the dictionary would always be a serious issue, especially when we must consider it with the performance and accuracy.
2. possibility to x-y location for all keys
We try to use an algorithm as simple as possible to have better performance. In this fast prototype, we just divide the horizontal space for all keys. When a key with higher weight, we add 1 to 3 extra space to them. For example, if user typed th, the ’e’ and ’i’ has the higher possibility to show. We weight ’e’ for 1 and weight ’i’ for 2. And we suppose the horizontal space for the whole keyboard is 320. The original space for each keys at first row, qwertyuiop, is 32 pixel for each. After the weighting, it is divided by 13, including the weight for ’e’ and ’i’, that is 24 pixel for each. But it is 48px for ’e’, and 72px for ’i’. Once we have corresponding size for each key, we need to convert them to the center point. That’s for Voronoi diagram.
3. keys’ position to keys’ polygon
In this prototype, we use d3[4] to calculate the Voronoi diagram for us. It returns a list of polygon points for each key. You may wonder that why not to use the size calculated at step 2 and why we need to run voronoi diagram. The main reason is to have vertical resizing. It looks like a hexagon for each key, originally. Once we change the weight, the vertical size of the hexagon are also affected slightly. If we use rectangle, the outcome may be a rearrangement of horizontal space and no changes on vertical space. As we known, the Voronoi diagram is trying to find the middle perpendicular for two points. With this, the horizontal neighbors of weighted key is also enlarged about 1/4. If we use the example above for ’e’ key, the size of ’w’ and ’r’ is enlarged as 30px (24px + 6px) and the size of ’e’ is about 36px (24px + 12px). With this symptom, user may feel the ’e’ pops out. The same feelings also happen to the key ’i’. If we dealing with more complex cases, including vertical weighting, the Voronoi diagram helps more. Another advantage is to fill the dividing remainders for all keys.
4. drawing key polygon (optional)
In this prototype, we use SVG path to draw the polygon, without any performance tuning. This step may/should be optional. Users may be confused when the keyboard is changing frequently. Drawing it in this prototype may help us to know if the current behavior is correct.
It may be nice to make the dynamic layout as a invisible touching area above the static keyboard layout, so that user can type without any possibly inconveniences from the variable layout but still benefits from the adjustable touching area. We need more information from UX team and others’ feedback to decide which method is better.
5. touch detecting
Once having SVG path, we use click event to detect user touch. We know we should use touch end event to dispatch the key. It may not be supported in SVG. But for a fast prototype, we choose the easy way to implement it. There is another alternative for it, map element, of course.
6. learning
We haven’t thought thoroughly about it and not included in this prototype. But it is possible to have a dynamic changing on prediction table. For example, we could have a component to dynamically train our dictionaries to make the layout more suitable for every single user, and mixing it with our online updating dictionaries as well.
7. presenting the candidate keys
In this prototype, we color candidate keys with red, and users feel helpful to have the clues for the next key they should press. Of course, the visual effect we used here is coarse and could be refined, like to use a light border instead of filling the key with single color, but this additional trick may help our keyboard even greater. It can be a potential way to improve the user experience beside enlarge the keys.
The repository may be found at [1]. We also made a video for it, which may be found at [3].
Finally, thanks Rudy and Evan for more information about this fast prototype.
Thanks
John Hu
[1] https://github.com/snowmantw/gaia/tree/vk/test_apps/voronoi-keyboard
[2] http://scholar.google.com/scholar?cluster=11097390483940419161&as_sdt=0,5
[3] http://www.youtube.com/watch?v=cRP4NQTN1qo
[4] http://d3js.org
John Hu
Sep 25, 2013, 5:25:34 AM9/25/13
to dev-gaia, Evan Tseng, Greg Weng, Rudy Lu, Tim Chien
Sorry for wrong video url. This one is the correct: http://www.youtube.com/watch?v=cRP4NQTN1qo.
John Hu.
----- Original Message -----
From: "John Hu" <jo...@mozilla.com>
To: "dev-gaia" <dev-...@lists.mozilla.org>
Cc: "Rudy Lu" <r...@mozilla.com>, "Greg Weng" <gw...@mozilla.com>, "Evan Tseng" <ets...@mozilla.com>, "Tim Chien" <timd...@mozilla.com>
Sent: Wednesday, September 25, 2013 5:24:27 PM
Subject: Dynamic Touch Target for Gaia keyboard
Hi all,
With the direction of Rudy, Greg and John had finished a fast prototype of Dynamic Touch Target[1]. The live video can be found at http://www.youtube.com/watch?v=cRP4NQTN1qo. We have to admit that it is not a real keyboard but just a demo app. We had demoed it to few people in Taipei, including engineers and UXs. The feedbacks from them are positive about this.
John Hu.
----- Original Message -----
From: "John Hu" <jo...@mozilla.com>
To: "dev-gaia" <dev-...@lists.mozilla.org>
Cc: "Rudy Lu" <r...@mozilla.com>, "Greg Weng" <gw...@mozilla.com>, "Evan Tseng" <ets...@mozilla.com>, "Tim Chien" <timd...@mozilla.com>
Sent: Wednesday, September 25, 2013 5:24:27 PM
Subject: Dynamic Touch Target for Gaia keyboard
Hi all,
Josh Carpenter
Sep 25, 2013, 6:47:39 AM9/25/13
to John Hu, Evan Tseng, Greg Weng, dev-gaia, Rudy Lu, Tim Chien
Wow, this looks like a great start!
> It may be nice to make the dynamic layout as a invisible touching area above the static keyboard layout, so that user can type without any possibly inconveniences from the variable layout but still benefits from the adjustable touching area. We need more information from UX team and others’ feedback to decide which method is better.
Invisible dynamic touch targets layered over static visuals is probably the safe approach for this initial release, based on prior art. ThickButtons for Android is the only keyboard I know of that displays it's dynamism to the user. Video here: http://www.youtube.com/watch?v=itIPS3U2bf8
We should be open minded to all approaches, and I have not tried ThickButtons personally, but at first glance displaying the moving targets strikes me as creating visual noise that is as likely to distract the user as it is to help them. I wonder too if it would undermine confidence and muscle memory to see the targets constantly shifting. More research and time-with-prototypes needed...
Great work! Looking forward to seeing it evolve :)
—
Josh Carpenter
UX Lead, Firefox OS
Mozilla
> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia
> It may be nice to make the dynamic layout as a invisible touching area above the static keyboard layout, so that user can type without any possibly inconveniences from the variable layout but still benefits from the adjustable touching area. We need more information from UX team and others’ feedback to decide which method is better.
We should be open minded to all approaches, and I have not tried ThickButtons personally, but at first glance displaying the moving targets strikes me as creating visual noise that is as likely to distract the user as it is to help them. I wonder too if it would undermine confidence and muscle memory to see the targets constantly shifting. More research and time-with-prototypes needed...
Great work! Looking forward to seeing it evolve :)
—
Josh Carpenter
UX Lead, Firefox OS
Mozilla
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia
Stephany Wilkes
Sep 25, 2013, 2:06:17 PM9/25/13
to Josh Carpenter, Rudy Lu, Evan Tseng, John Hu, Greg Weng, Tim Chien, dev-gaia
Really, really nice work on this guys. I also appreciate the email with all of the thinking behind the prototype - very helpful to be able to read through it, and helpful also for understanding the demo video.
How might we obtain a testable prototype for quick rounds of usability testing, and for internal testing over time?
Many thanks,
Stephany
How might we obtain a testable prototype for quick rounds of usability testing, and for internal testing over time?
Many thanks,
Stephany
Carrie Wang
Sep 27, 2013, 2:48:05 AM9/27/13
to John Hu, Evan Tseng, Greg Weng, dev-gaia, Rudy Lu, Tim Chien
Hi guys,
We've tried this prototype here in Taipei and felt the accuracy has improved a lot (but still needs more tests to verify it though).
Currently, the prototype applies only 4000 frequently used words as the database for dynamic hit target.
Hence, if users want to type the word apart from these 4000 words, I'm afraid that the dynamic layout might distract them a lot.
Some research has addressed this issue as well, but we can still have some tests on it. :)
Here are some rough ideas on optimizing the current prototype.
1. The prefix mistype might counteract the dynamic hit target effect. If users mistype the first letter, then it will affect the consequence of regional error correction.
Thus, we'd suggest looking into users’ typing behavior and setting some invisible default offset on each key to prevent prefix errors.
2. The weight of possible keys could be more precise. We can try some different set of parameter values to carry out the best size changes.
We might also need to set an "anchored area" on keys to prevent some keys are impossible to be typed due to the extreme resize of their neighboring keys.
We are still working on it and will document more details later. All the comments and suggestions are appreciated!
Thanks!
Carrie
--
Carrie Wang
TAIWAN UX team, UX Designer
Mozilla Corporation
----- Original Message -----
From: "John Hu" <jo...@mozilla.com>
To: "dev-gaia" <dev-...@lists.mozilla.org>
Cc: "Evan Tseng" <ets...@mozilla.com>, "Greg Weng" <gw...@mozilla.com>, "Rudy Lu" <r...@mozilla.com>, "Tim Chien" <timd...@mozilla.com>
Sent: Wednesday, September 25, 2013 5:24:27 PM
Subject: Dynamic Touch Target for Gaia keyboard
Hi all,
With the direction of Rudy, Greg and John had finished a fast prototype of Dynamic Touch Target[1]. The live video can be found at http://www.youtube.com/watch?v=eMFfmG8jhRg. We have to admit that it is not a real keyboard but just a demo app. We had demoed it to few people in Taipei, including engineers and UXs. The feedbacks from them are positive about this.
We've tried this prototype here in Taipei and felt the accuracy has improved a lot (but still needs more tests to verify it though).
Currently, the prototype applies only 4000 frequently used words as the database for dynamic hit target.
Hence, if users want to type the word apart from these 4000 words, I'm afraid that the dynamic layout might distract them a lot.
Some research has addressed this issue as well, but we can still have some tests on it. :)
Here are some rough ideas on optimizing the current prototype.
1. The prefix mistype might counteract the dynamic hit target effect. If users mistype the first letter, then it will affect the consequence of regional error correction.
Thus, we'd suggest looking into users’ typing behavior and setting some invisible default offset on each key to prevent prefix errors.
2. The weight of possible keys could be more precise. We can try some different set of parameter values to carry out the best size changes.
We might also need to set an "anchored area" on keys to prevent some keys are impossible to be typed due to the extreme resize of their neighboring keys.
We are still working on it and will document more details later. All the comments and suggestions are appreciated!
Thanks!
Carrie
--
Carrie Wang
TAIWAN UX team, UX Designer
Mozilla Corporation
----- Original Message -----
From: "John Hu" <jo...@mozilla.com>
To: "dev-gaia" <dev-...@lists.mozilla.org>
Cc: "Evan Tseng" <ets...@mozilla.com>, "Greg Weng" <gw...@mozilla.com>, "Rudy Lu" <r...@mozilla.com>, "Tim Chien" <timd...@mozilla.com>
Sent: Wednesday, September 25, 2013 5:24:27 PM
Subject: Dynamic Touch Target for Gaia keyboard
Hi all,
Julien Wajsberg
Sep 27, 2013, 4:48:21 AM9/27/13
to Carrie Wang, John Hu, Evan Tseng, Greg Weng, dev-gaia, Rudy Lu, Tim Chien
Le 27/09/2013 08:48, Carrie Wang a écrit :
> 1. The prefix mistype might counteract the dynamic hit target effect. If users mistype the first letter, then it will affect the consequence of regional error correction.
> Thus, we'd suggest looking into users’ typing behavior and setting some invisible default offset on each key to prevent prefix errors.
>
I think this point is especially important because I've seen that
> 1. The prefix mistype might counteract the dynamic hit target effect. If users mistype the first letter, then it will affect the consequence of regional error correction.
> Thus, we'd suggest looking into users’ typing behavior and setting some invisible default offset on each key to prevent prefix errors.
>
depending on the device and the screen there can be different offsets
given by the hardware and drivers.
Salvador de la Puente González
Oct 6, 2013, 7:32:45 AM10/6/13
to Julien Wajsberg, Rudy Lu, Tim Chien, Evan Tseng, John Hu, Greg Weng, Carrie Wang, dev-gaia
I really like the idea. When I started to work in the keyboard one year ago I implemented a little library to abstract some events for the keyboard. This will include dynamic determination of the button-area taking into account this possibility. You can find the library here:
https://github.com/lodr/eal
Hope it thelps!
_______________________________________________
dev-gaia mailing list
dev-...@lists.mozilla.org<mailto:dev-...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-gaia
________________________________
Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx
https://github.com/lodr/eal
Hope it thelps!
dev-gaia mailing list
dev-...@lists.mozilla.org<mailto:dev-...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-gaia
________________________________
Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx
0 new messages