jQuery Onscreen Keyboard

[Chris Hoffman]

I've been toying with the idea of a Web-based onscreen keyboard for a
while. I think it could be a great accessibility enhancement (especially
as an alternative to the awful Windows osk, or to proprietary
solutions), and that it could also be used in keyboardless (e.g., kiosk)
environments or where security is a concern.

I would like to know if anyone has tips or resources that might be
helpful. I've gotten some good ideas looking at things like the Gnome
Onscreen Keyboard, but am getting a little overwhelmed
architecture-wise.

For example, I think it would be good to have layouts stored as separate
files that can be ajaxed in, to provide support for multiple
languages/layouts. However, between umpteen modifier keys and locking
modifier keys and dead keys, any attempt I make at a format quickly
becomes unreadable or at least unwieldy. Here's what I would like/what
I'm trying to fit into this beast:

1) Formatted in JSON (as opposed to XML) for easy parsing.

2) Support for modifier keys and dead keys. Does each combination get
its own layout? Or is there a clean way to include them all without
creating a huge mess of a file?

3) Human-readable and modifiable, which leads to:

4) Identifiers for keys. I've looked at DOM3 identifiers, but there's
lots of "U+XXXX"--I like the Java or .Net "KEY_Q" better for
readability.

5) Potentially, keys for word-prediction.

6) Much later, support for alternate access methods. For now
direct-select is fine.

7) User-modifiable layouts, perhaps saved as cookies/to a dedicated
server/using something like Google Gears or HTML5 persistent data?

All of the infrastructure is easy enough to gather up from the jQuery UI
and a few nice extensions, but I'm having trouble getting my teeth
around the apple to bite into it. Anyone?

Chris

[David Bolter]

Hi Chris,

I have a few things to say.

1. Wow.

2. GOK. You probably noticed I'm the GNOME GOK dictator, and of course
I'm happy to help with ideas and recognizing pitfalls in OSK design. I
haven't taken my thoughts of a web based OSK as far as you have. On a
side note, I don't have much time for GOK anymore and am looking to
mentor my replacement.

3. Jambu. This project was (is?) almost trying to do the same thing
but from a strictly a11y angle. See recent thread on roadmap musings:
http://groups.google.com/group/jambu/browse_thread/thread/8f53d28588e989d2

4. File formats. We tried a few with GOK in the early days and I know
exactly what you mean regarding modifiers etc. Since GOK is housed in
Linux/GNOME we decided to play nice with system modifiers in the end.
For its regular keyboard synthesis (not the gui generated stuff), GOK
queries the system keyboard and recreates it on-screen. Whenever a
modifier is applied (via GOK or otherwise), GOK is notified and
changes the keyboard to show the modified key labels. We sort of had
to go this route so that GOK and the system didn't get their signals
crossed.

5. Access Methods. Even if coming in the distant future... keep it in
mind as you design :)

cheers,
David

[SteveLee]

On Apr 3, 7:59 pm, David Bolter <david.bol...@gmail.com> wrote:
> 3. Jambu. This project was (is?) almost trying to do the same thing

> but from a strictly a11y angle. See recent thread on roadmap musings:http://groups.google.com/group/jambu/browse_thread/thread/8f53d28588e...

Thanks David and hi again Chris, looks likes we've had similar ideas
again.

I'm working out what I want to do with Jambu next and a web based and
flexible way of describing ultra simple OSKs or interactive symbol
sets. I'm still wondering about a Firefox add-on or XULRunner but pure
web technology is very attractive for more general application. The
DOM and layout engine do most of the work and Accessibility provision
is there too. Web means Javascript for client side but also I think
WAI-ARIA is going to be important to interact with ATs and dojo has
that now. Though I believe it is WIP in JQuery? I'm not up to speed
with either dojo or JQuery but it is top of my list of stuff to play
with next. I'm also not opposed to JSON which especially makes sense
in Javascript but I'm committed to SVG and XML then blends well.

It would be good to talk, even if we decide to go our separate ways
again. I'd like to keep it public but perhaps this list is not that
place. Would you be happy to join up with the jambu list and discuss
it there?

One problem I see is we're stuck with the limited DOM events and that
may be were the other face of Jambu comes in, as a way of mapping and
controlling firefox and the web via switche guestures etc.

Finally we need to generate synthetic events to drive applications (or
use a11y API) and I'm not sure how to do that from a web app. Isn't it
sandboxed so you can't do such things? Here Firefox offers a solution
as XPCOM would allow such extensions, perhaps using python.

As for handling the different images for various states, the approach
I took in Jambu was to build a composite SVG image depending on state.
So some bits could be the same whatever the state and others change
(or you could swap the entire image). My implementation went down some
dead end paths but the basic idea works as I created a sample (slow)
OSK in a custom GTK widget.

Look forward to talking more.

Steve Lee
fullmeasure.co.uk

[Chris]

On Apr 3, 2:21 pm, I wrote:

> I've been toying with the idea of a Web-based onscreen keyboard for a
> while.

The project has found a home at jqosk.googlecode.com. I've only just
moved in, so please don't mind all the cardboard boxes and stacks of
plates lying around.

--Chris