Integrating Switch Accessibility into Mozilla
MichaelB
I support the idea of integrating accessibility into Mozilla. My goal
(as a final year project at university) is to write an interface for
switch access users (ie. users who have the ability to hit one or two
buttons - or produce a binary input signal by any other means available).
Input could be directly through the keyboard or through third-party
hardware. Third-party software would only have to send signals of the
form "left button" or "right button" to Mozilla (this will be done via a
socket connection). All further processing of the signal will happen in
my component. This brings some advantages:
1. third-party software needs no reprogramming when new features are
introduced into mozilla
2. The Mozilla community can add optimisations for switch access users
whenever they want (eg. when surfing, allow user to choose from most
recently visited links first, or: optimize filling out of forms, etc.)
Comments or ideas welcome!
Particularly, I would like to know from switch access users:
1. Which software are you currently using?
2. Can you fill out forms or send emails with this software?
3. Which improvements to the access mechanism would reduce complications
or help you to finish your task faster?
Thank you,
Michael Bensch
Bill Haneman
You might want to look at project GOK, the 'GNOME Onscreen Keyboard",
which currently integrated with Mozilla's GTK+-2 builds.
GOK is a general purpose dynamic onscreen keyboard utility which not
only emulates "qwerty" keyboards for users of various switch and pointer
devices, but also uses accessibility APIs integrated into the
GTK+-Mozilla versions which allow UI capture and direct interaction with
menus, toolbars, and other UI components via the "Action" interface.
I believe that the cross-platform Mozilla work now includes equivalent
'action' interfaces such as nsiAction, but I am not sure of the details
there.
GOK (and any other switch or pointer-driven assistive technology which
uses the 'AT-SPI' service provider interface) has the advantages you
list below, along with the additional advantage that onscreen-keyboard
and switch "interpretation" code doesn't have to live in Mozilla, but
can live in the assistive technology which inherently knows better how
to interpret switch events for a given user. GOK for instance can be
configured and customized to meet the needs of users who can control
between 1 and 5 switches, and/or a pointer of some kind, and can support
command prediction and word prediction. It also allows creation of
custom context-dependent keyboards so that users and/or clinical
therapists can customize access to specific applications like Mozilla.
The feedback you get from switch users to your questions below will be
of great value, and certainly I'd like for the Mozilla accessibility
teams and GOK team to read it carefully.
But I think that the right place for providing more sophisticated switch
'access methods' is in assisitive technology software itself, not in
Mozilla; Mozilla's responsibility is to expose its UI elements in a way
that allows them to be 'actioned' by external agents, regardless of how
those agents get their user input.
GOK doesn't work on Windows (yet), but since Mozilla now supports a
richer accessibility API internally which includes bidirectional
interaction, it or a similar technology could be written to use these
APIs on Windows as well, rather than introducing new switch-event-driven
behaviors into Mozilla.
best regards,
Bill
--
Bill Haneman <bill.h...@sun.com>
Bill Haneman
I forgot to include links to GOK:
and
http://developer.gnome.org/projects/gap/AT/GOK/
Source code for GOK can be obtained from GNOME CVS,
module gok, CVSROOT=
:pserver:anon...@anoncvs.gnome.org:/cvs/gnome
thanks,
Bill
Aaron Leventhal
What platform are you working on?
What advantage is there in working directly with a
specific application, rather than a general solution
simulating a keyboard?
- Aaron
Michael Bensch
your points are appreciated and certainly got me thinking. GOK is an
exciting project and maybe a good reason for switch users to switch to
Gnome :-)
A large part of the research here at the Institute of Medical Psychology
and Behavioral Neurobiology in Tübingen is devoted to brain-computer
interfaces (BCI) ie. users with severe motor disabilities, but able to
generate neural signals with the brain which can be measured and
interpreted as a binary choice.
This type of interaction differs from standard switch access in two points:
1. the probability for a correct signal can vary from 0.65 to 0.95
depending on the user - we thus need response verification in our software.
2. The maximum rate of communication is 10-20 bits per minute - we thus
have to reduce the selectable "accessibles" on any screen to a bare
minimum, and optimize the selection process by using a visual binary
recursive subgrouping algorithm. (A "direct" or "scanning" selection
technique, or even a keyboard (to answer Aaron's question) would be too
inefficient.)
We are using our own neural signal processing software, which is bound
to a win platform but does not currently offer any accessibility
features (so sadly we cannot use Gnome which seems to be a wonderfully
accessible OS).
Given these factors, I thought it best to create a XUL/CPP interface to
have direct access to mozilla. This allows us to:
1. present the visual binary subgrouping cue directly on the web page (I
am using the inIFlasher interface for this - thanks to Joe Hewitt and
any other contributors for this very useful piece of code!)
2. access history information for further optimizing link selection
(place links that are selected often, at the top of a huffman tree)
3. Keep the "switch interpretation code" platform-independant using
mozilla's wonderful XPCOM technology, rather than integrating it into
our win software, making it even harder to migrate to another platform
in the future.
(If there are other solutions to 1. or 2. I would be very interested.)
I agree with Bill that in the long run, more sophisticated switch
'access methods' should be in the assistive technology itself, and that
this is not really the responsibility of a web browser, but at the
moment, for our particular application, I am positive that the close
proximity of our module to the mozilla code base can only benefit its users!
Having said all that, I still believe that many standard switch access
users might see some use in this new interface. I must admit that I have
never used switch access software and the following would interest me:
1. What is the most efficient method to choose from say, 20 links on a
page, that current switch access software provides? (with efficient I
mean: with as little switch presses as possible. ie. a scanning
interface would be inefficient.) How would a task like this be
accomplished with the GOK?
2. Does current software allow for response verification, ie. offer
"back" options during selection of a link, in case of an error?
Regards,
Michael
Aaron Leventhal
It sounds like a really interesting project. Will there
be any way to access form controls in web content,
bookmarks or any of the menus or other features? Or
will it be for link activation and document reading only?
I now understand your justifcation better for building
it into Mozilla, using XPCOM. However, since you are
currently only on Windows, did you consider the need to
access other applications? Will the switch software
possibly conflict with other switch software the user
may be using to access those other applications? Will
you need to work with switch solution manufacturers?
As far as your other questions, I do think that forward
and back options would be useful. You could possibly
insert extra links or buttons for those things into the
start of each page as it is loaded.
I will be in Stuttgart for 6 months starting in May --
perhaps we can meet and work on some solutions
together. Have you tried to connect with any users or
other experts in switch access? The Mozilla
accessibility newsgroup is probably not the best place
to do that -- accessibility on this platform is still
being developed, so only cutting edge people are
attempting to use those features.
Good luck,
Aaron
Peter Korn
I'm cc-ing the GOK Development mailing list <gok...@gok.ca>. I have a feeling
that the GOK engineers aren't on the mozilla-accessibility mailing list.
Michael - at the risk of repeating arguments Bill has made to you, I strongly
suggest you take a close look at GOK. They've done a lot to provide direct
access to menus, toolbars, and with their "UI Grab" feature, the contents of a
dialog box. These they then format onto a switch keyboard for rapid direct
access. While GOK does not presently do this with links on a web page, it would
be a very straightforward application of their techniquies layered on top of the
GNOME Accessibility API (through which an AT can enumerate the links in an
AccessibleHypertext document such as the web page in Mozilla).
As GOK is open source, it is perhaps the best place for research - you can build
on the work of others to rapidly get the features you want, with no royalties or
licensing fees. And, thanks to the fact that it the code is LGPL, you have a
very strong incentive to contribute your work back to the commons - thereby
allowing the next person to build on your work.
By the way, I was recently at the Assistive Technology Industry Association
conference in Orlando, where I spoke with researchers in Japan working on a
brainwave<->switch interface, and other researchers in the U.S. looking at
switch interfaces based on recognition of a limited vocabulary from people with
highly dysarthic speech. Both were very interested in using GOK as a
development and execution platform for their switch interfaces.
Regards,
Peter Korn
Sun Accessibility architect
Aaron Leventhal
I don't think he can use GOK, does it run on the win
platform?
> We are using our own neural signal processing
> software, which is bound to a win platform but does
> not currently offer any accessibility features (so
> sadly we cannot use Gnome which seems to be a
> wonderfully accessible OS).
- Aaron
Bill Haneman
> Peter,
>
> I don't think he can use GOK, does it run on the win
> platform?
I don't see any big obstacles to getting GOK to run on win32. GTK+ has
been ported to win32, as has ATK. GOK has a couple of additional
dependencies which could be managed without too much trouble, but you'd
need to integrate GOK's input device support with win32 devices. Here
again, I don't see this as being a big job, and for Michael's purposes
since his input methods are unusual there'd be some work necessary there
regardless.
As for Mozilla, since the mozilla codebase exports ATK on *nix already,
GOK's more advanced features could readily be made available while
running Mozilla provided Mozilla could be build with the ATK
accessibility back-end on Win32. That seems quite doable in principle
if nontrivial (you'd need to have the GTK+ win32 developer libraries
installed).
Given the specialized nature of Michael's project there would need to be
significant coding in any case. I think that the work required to make
GOK run with an ATK-enabled Mozilla on win32 would be only a modest part
of the total project, and it would have the advantages of offering a
cross-platform solution which had more extensibility to applications
other than Mozilla.
regards,
Bill
--
Bill Haneman <bill.h...@sun.com>
MichaelB
yes, text form fields, bookmarks and a few other options (PageUp/Down,
Back, open URL, text size) will be available. The system will be
extensible so that more options may be added.
No, I have not considered the need to run other switch software
simultaneously. This would come into consideration for non-neural signal
users. My suggestion for these users would be:
1. Complete your normal tasks with your existing switch SW.
2. If you want highly efficient, no-frills surfing with mozilla, switch
to my extension.
(Existing SW would then have to stop reading and interpreting the
available accessibles and merely send a signal 0-4 (depending on the
number of switches) to my extension over a socket connection. This is
the way I have implemented it for our signal processing software.)
This implies that existing SW would have to be modified.
3. When you are finished, my extension will signalize to your switch SW
that it can continue its normal work.
I only expect users who have severe difficulties operating a switch, to
be interested in the efficient surfing, as the others won't mind hitting
the switch a few times more to obtain the same result with their current
SW. However, the few that can benefit, would have the additional
advantage that any XPCOM-based software exposing some kind of DOM could
theoretically be driven by my extension, with a few modifications.
If you're in Stuttgart we can definitely meet up.
We have connections to some neural signal users, but I must admit I have
no connections to switch access experts. I imagine the GOK mailing list
would be a good place to start.
Michael
MichaelB
sorry for the high latency - I am quite under time pressure at the moment.
Thanks for the info on GOK. If I would have known all this from the
start, I would have thought twice about using GOK.
Sadly I only have until beg. April to complete the project and I feel I
have done too much to start afresh. But I have passed this on to my
supervisors and I'm sure it will receive consideration for the next
project.
I think the open source holds for my project and mozilla as well.
Do you have contact details for the Japanese researchers? It would be
wise to join forces on the GOK issue, and they might be able to use my
extension in the meantime.
Bill, your suggestion on getting GOK to run on win32 sounds promising
although I don't have the time to test all that now... but if anyone
reading this has succeeded and is using GOK on win32 with Mozilla,
please report your findings to this newsgroup.
Michael
Aaron Leventhal
I don't want to linger on the point too much, but the
decision of whether or not to integrate with Mozilla is
still an interesting one, which can help us outsiders
understand your project better. Personally, I would at
least consider being an external app, for the purposes
of future extensibility and integration with other
switch software.
You can still have access to the DOM, focus events and
all sorts of other useful APIs as an external app. What
kinds of getters/setters will you need -- is there
anything you can get internally that you need, which
you would not be able to get through an accessibility API?
- Aaron