gesture recognition and OSD
Mathieu Blondel
I found two projects to do gesture recognition in Linux:
http://sourceforge.net/apps/trac/easystroke/
http://gestikk.reichbier.de/
These programs allow to perform gestures (if a certain predefined key
is hold at the same time) in order to trigger associated keys or
commands.
Since gesture recognition is very similar to online handwriting
recognition, I had a quick look at their source code but they seem to
use very rudimentary stroke recognition. (DTW or HMMs would perform
very well I think)
These projects can do On Screen Display (OSD) to display the gesture
that is currently performed (regardless of the currently selected
application).
This led me to think that OSD could be an input interface for tegaki
as well: you would press a pre-defined key (such as ctrl) to activate
recognition, write your character directly on the screen, release ctrl
when you're done and the result would appear directly where your
cursor is. With this approach it wouldn't be possible to select
alternative candidates when the recognizer makes a mistake though.
This could be used on a normal computer too but it would be especially
useful on tablet-PCs because the screen IS the input interface.
Mathieu
Roger Braun
> as well: you would press a pre-defined key (such as ctrl) to activate
> recognition, write your character directly on the screen, release ctrl
> when you're done and the result would appear directly where your
> cursor is.
ATOK for Mac OS has a mode like that. I don't really remember how it
works, but it would be great to have it in Tegaki. Maybe you should
take a look at how ATOK did it.
> With this approach it wouldn't be possible to select
> alternative candidates when the recognizer makes a mistake though.
Why not? Is this more than a transparent window that sits above all others?
> This could be used on a normal computer too but it would be especially
> useful on tablet-PCs because the screen IS the input interface.
You would also need something like this on mobile devices, I guess.
--
Roger Braun
http://yononaka.de
roger...@student.uni-tuebingen.de
Mathieu Blondel
> ATOK for Mac OS has a mode like that. I don't really remember how it
> works, but it would be great to have it in Tegaki. Maybe you should
> take a look at how ATOK did it.
Is there a video showing the functionality ?
BTW, with the new Google IME, the competition is gonna be hard for
ATOK... Especially since Google's IME is free.
>> With this approach it wouldn't be possible to select
>> alternative candidates when the recognizer makes a mistake though.
>
> Why not? Is this more than a transparent window that sits above all others?
If we want the user to be able to select alternative candidates, I
don't really see the need for the window to be transparent. A shortcut
to bring tegaki-recognize to the front should be enough.
My original idea was more like: you hold ctrl, draw you character
wherever you like on the screen and the recognized character appears
immediately where your cursor is when you release ctrl. The point here
is to be able to input the character as fast as possible.
You can see a video of on screen gesture here
http://gestikk.reichbier.de/downloads/demo-0.5.ogg
Another idea I had and I already mentioned is to use the trackpad of a
laptop like a drawing box. The only thing is that the trackpad can
only render relative moves, not the absolute position of your finger
on the trackpad. I think it can work with Wagomu because the feature
vectors that it uses encode the relative position of consecutive
points.
Anyway, it's nice to think about new ways to input characters.
Mathieu
PS: Until now, when you validated characters in tegaki-recognize, they
were copied to the clipboard (seems like quite a few people didn't
notice this functionality). Yesterday I've added support for fake key
events in tegaki-pygtk. tegaki-recognize can now send the characters
directly where your cursor is. No more need for scim-tegaki!
Mathieu Blondel
code from gestikk. I attached a proof-of-concept. If you want to try
it, you need an up-to-date copy of the git repo, and you can run it
like this
$ python osd.py zinnia Japanese
Then leave it running in the background. When you hold ctrl, you can
start drawing a character on the screen (by pressing the mouse
left-button). And when ctrl is released, the best match is copied to
the clipboard as well as sent to the window which has the focus.
Currently ctrl is used to determine the start and end of a character
while the mouse left button is used to determine the start and end of
strokes. However, the left button press event is also sent to the
actual application, which can cause undesirable behavior (like moving
text around in a text editor). I want to find a better way to express
the fact that a stroke is being started.
The code from gestikk is not ideal. It checks every 100ms for new
movements... I wonder if there's a better way to do it...
Mathieu
tw
> --
>
> You received this message because you are subscribed to the Google Groups "Tegaki Handwriting Recognition Project" group.
> To post to this group, send email to tegak...@googlegroups.com.
> To unsubscribe from this group, send email to tegaki-hwr+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/tegaki-hwr?hl=en.
>
>
> <osd.py>
Mathieu Blondel
> Can you run this on Mac OS X? Or is this Linux only?
Linux only.