I feel left out

[Eric]

As a disabled person using speech recognition for the past 20 years, I
see absolutely nothing in your project addressing my needs and I'm
feeling left out. in fact, you are reinventing the same old
accessibility framework that was first published in roughly the early
1990s. gluing accessibility to a GUI which at best is a horrible mismatch.

Here's what I need as speech recognition user.

1) read the state of anything that can be displayed or changed via a
GUI. this is a getter function
2) change the state of anything that can be changed by a GUI. This is a
putter function.
3) do something. This is usually the action associated with a link or
button but can also drive mouse over or any other event causing an action.
4) tell me when something changes. These event notifications allow you
to use hand/mouse at the same time as speech and it lets the speech
system stay in sync with what's being displayed.

By these four simple interfaces, you eliminate the mismatch between a
GUI and a speech interface. No longer do you have to navigate through
menus and clicks and links to get to a data element which you can't even
speak because it has no name. Your grammar can short-circuit all that
and go change the data element in question.

Remember, a GUI navigation is tall and narrow, speech navigation is wide
and shallow. Don't try to force one to be the other.

If you look at the other accessibility requirements, you will see
similar mismatches and this API set or something very similar to it
would make it possible to address the needs of the new accessibility
interface and provide a simpler, more direct interface.

[Kaleb Hornsby]

I've been experimenting with aria-attributes in polymer element,s and they seem to do OK. Eric, what have your experiences been with polymer and WAI-ARIA?

[Alice Boxhall]

[Re-sending my message from April 6 which was emailed but is not showing up in the web interface: apologies to anybody who sees this twice!]

Hi Eric,

I'm an engineer on Chrome, and I've been working with the Polymer team on making sure that we address Custom Element accessibility.

I agree with you that it is critically important that we don't leave users with disabilities behind in the Web Components world. I am cautiously optimistic, however.

Firstly, web developers are already a long way down the path of creating complicated, mouse-driven UIs which lack semantic information. Web Components gives us an opportunity to formalize techniques for creating robust custom elements with good accessibility.

For example, the ARIA working group are also currently discussing proposals for allowing ARIA to be extended by developers (see the discussion beginning with this email: http://lists.w3.org/Archives/Public/public-pfwg/2014Apr/0034.html) to better support novel types of UI. One problem with ARIA is that it is not supported by the dominant voice control software, Dragon, which is still using very outdated technology - we do have work to do on the standards side, but we need those standards to be supported by the relevant tools as well.

There is also work underway to formalize the possible actions and events which may occur on an element: http://www.w3.org/TR/indie-ui-events/ - although this is currently in the early stages and in need of more support from the community.

Secondly, Web Components gives developers a chance to try out novel ways of supporting accessibility-related use cases, as argued by Steve Faulkner: http://blog.paciellogroup.com/2014/04/usability-accessibility-opportunities-web-compenent-world/ - which could, in the future, possibly become part of the standard HTML spec.

Our recent article, http://www.polymer-project.org/articles/accessible-web-components.html, outlines the strategies the Polymer team are using to address accessibility.

Here is how we are trying to address your four points using the technology available today:

> 1) read the state of anything that can be displayed or changed via a 

> GUI.  this is a getter function 

We annotate custom elements with ARIA role, name, state and value properties. This provides the state information which can be queried by speech recognition technology, via platform APIs like IAccessible2, UI Automation, NSAccessible etc., and allows you to query the interface by name.

 

> 2) change the state of anything that can be changed by a GUI. This is a 

> putter function. 

This is where there is currently a gap in specifications, and authors are forced to implement their own interactions. The IndieUI spec proposes one possible mechanism for addressing this: http://www.w3.org/TR/indie-ui-events/#intro-example-valuechangerequest . To fill this gap for now, we suggest using data binding to translate user gestures via keyboard or mouse events into changes to attributes on the custom elements, which are then reflected in the ARIA attributes accessible via platform APIs.

 

> 3) do something. This is usually the action associated with a link or 

> button but can also drive mouse over or any other event causing an action. 

Similarly, acting on an element is currently handled via keyboard and mouse events, and this could be supported at a higher level by something like IndieUI actions (http://www.w3.org/TR/indie-ui-events/#actions). Currently, Polymer elements listen for mouse and keyboard events which are used to drive actions on the element. As you say, these events can be simulated by assistive technology via the platform accessibility APIs. We do recommend having a robust keyboard story and ensuring that elements are focusable, to avoid having to perform fiddly mouse interactions.

 

> 4) tell me when something changes. These event notifications allow you

> to use hand/mouse at the same time as speech and it lets the speech 

> system stay in sync with what's being displayed. 

ARIA provides an aria-live attribute, which is also implicit in certain role values, to notify users when the content of certain regions changes.

I would greatly appreciate it if you could let us know if you see any areas where we could improve this strategy to address your needs. Also, it would help to hear more about your specific experience: what browser and assistive technology are you using? Are there any sites which work well for you now which could be used as a model for best practices?

Thanks,

Alice

[Eric]

On 4/10/2014 11:17 PM, Kaleb Hornsby wrote:
> I've been experimenting with aria-attributes in polymer element,s and
> they seem to do OK. Eric, what have your experiences been with polymer
> and WAI-ARIA?

I'm just getting started. It is difficult to find time because when you
are disabled, making a living is significantly harder than you think it
should be. I'm running at about 70% normal because I need to take time
to let my hands cool down and going "ow ow ow" tends to ruin your
concentration.

The only reason I'm describing my experience is because I have yet to
encounter developers have experienced this level of disability, or if
they have experienced it, developers who are not in denial about how bad
it is. Living this life is why I want to change it. I don't want anybody
else to suffer the same kind of experience.

If you don't mind, Kaleb, I work best through practical example. Let me
describe to you an editor and I would like to hear how can we make it
happen with polymer.

Imagine an editor which is just a text window. No menu items, no drop
downs nothing. It's just a window. You can dictate into the window/text
area and perform all of the usual nuance Select-and-Say type
operations.[1]. Data is saved automatically so that if the machine
crashes, the application dies, you don't lose your text. In fact, you
don't even need to exit the application when shutting down the machine.
The auto saving should be sufficient for preserving your text.

If associated with the file, every time the speech command that invokes
the editor with associate a file occurs, there is at most one copy of
the window with file. For example "add log entry" brings up the log file
for this month. I go do more work, and I say again "add log entry", the
existing window comes up to focus and I can dictate. by the way, a lot
of the calculations that go into specifying the log file name and things
of that nature are done within the speech user interfaced is implemented
by natlink.

The editor lets me query from natlink and retrieve text with and without
cursor information. I can also replace text and position the cursor
anywhere within the window.

So, how could one implement this with polymer?

--- eric

[1] we have some documentation from when Dragon Systems first released
natlink which helps us with this capability.

[Eric]

On 4/14/2014 10:39 AM, Alice Boxhall wrote:

[Re-sending my message from April 6 which was emailed but is not showing up in the web interface: apologies to anybody who sees this twice!]

My apologies for taking so long to respond.

Hi Eric,

I'm an engineer on Chrome, and I've been working with the Polymer team on making sure that we address Custom Element accessibility.

I agree with you that it is critically important that we don't leave users with disabilities behind in the Web Components world. I am cautiously optimistic, however.

Firstly, web developers are already a long way down the path of creating complicated, mouse-driven UIs which lack semantic information. Web Components gives us an opportunity to formalize techniques for creating robust custom elements with good accessibility.

I'll probable repeat myself on this point multiple times, and I apologize in advance but if you talking about elements on the screen, you're already too late. The information you need to do a good interface has degraded from the original enough that you can't do the right thing. We'll get to the definition of "right thing" later.

For example, the ARIA working group are also currently discussing proposals for allowing ARIA to be extended by developers (see the discussion beginning with this email: http://lists.w3.org/Archives/Public/public-pfwg/2014Apr/0034.html) to better support novel types of UI. One problem with ARIA is that it is not supported by the dominant voice control software, Dragon, which is still using very outdated technology - we do have work to do on the standards side, but we need those standards to be supported by the relevant tools as well.

The kindest thing I can say about Nuance is that they have their head up their ass. Those of us dependent on it, only use it as a speech recognition dictation and command and control engine. We use natlink+python (aka natpython) to build high-level toolkits which allow us to build good speech interfaces. I was hoping to have my programming by speech tool little more advanced but in a nutshell, natpython makes it possible for me to create a speech programming environment.

There is also work underway to formalize the possible actions and events which may occur on an element: http://www.w3.org/TR/indie-ui-events/ - although this is currently in the early stages and in need of more support from the community.

Secondly, Web Components gives developers a chance to try out novel ways of supporting accessibility-related use cases, as argued by Steve Faulkner: http://blog.paciellogroup.com/2014/04/usability-accessibility-opportunities-web-compenent-world/ - which could, in the future, possibly become part of the standard HTML spec.

Our recent article, http://www.polymer-project.org/articles/accessible-web-components.html, outlines the strategies the Polymer team are using to address accessibility.

Okay, here's my example of how focusing on the graphical elements fails. bring up LO writer. (Yes, I know this is not a web app but the concepts are the same)

Dictate some text such as "this is some simple text". The task is put that text in a box. I'm choosing the simple route of turning on the borders in a paragraph.

By hand	by speech by graphical element	by speech with deeper application knowledge
Move mouse to beginning of text, hold down mouse button, drag to end of text Right-click text move down menu to paragraph left click paragraph Click tab 'borders' click second from the left line arrangement click okay	move mouse the beginning of text "hold left mouse button" "drag right" ...wait for it... "stop" "right-click" ...counting menu items.. "move down seven" "mouseclick" move mouse over Borders tab "mouseclick" move mouse over the line arrangement box (second to the left) "mouseclick"	move mouse to start of text, "leave Mark", move mouse to end of text turn on all borders

This is a reasonable facsimile of what the user experience would be using mouse, speech as a mouse replacement, and a speech interface.some of the nuance tools such as mouse grid and mouse movement are broken, as I just discovered. Like I said, they have their head up their ass.

Now change the task to increasing the size of the box gets even more interesting because with a speech interface, you could say "expand box vertically by five" and the underlying code would read the current height, at five, and replace the value of changing the box height. By hand, it's a bit more complicated  requiring touching  multiple tabs and count user elements.

One important thing to remember, in this concept, I see the ability to create speech user interfaces or, for that matter, any accessibility user interface should reside in the hands of the user, not the application developer. I noticed a harsh comment but I can't think of any other way to express it quite yet. The GUI interfaces I've seen and suffered through do not give me any hope that accessibility interface will be any better. Leave in the hands of the user so they can create the interface they need.

Here is how we are trying to address your four points using the technology available today:

> 1) read the state of anything that can be displayed or changed via a 

> GUI.  this is a getter function 

We annotate custom elements with ARIA role, name, state and value properties. This provides the state information which can be queried by speech recognition technology, via platform APIs like IAccessible2, UI Automation, NSAccessible etc., and allows you to query the interface by name.

Yeah, solving the wrong problem. As I showed above, you don't want touch the buttons. You want to touch the data or code the buttons refer to. For example, you don't know want speak incrementing and decrementing of value for margin spacing. You want to say "margin spacing is 0.57 inches". You don't need to know where it lays out on the screen, you don't care if the user interface changes because you're touching underneath the user interface.

 

> 2) change the state of anything that can be changed by a GUI. This is a 

> putter function. 

This is where there is currently a gap in specifications, and authors are forced to implement their own interactions. The IndieUI spec proposes one possible mechanism for addressing this: http://www.w3.org/TR/indie-ui-events/#intro-example-valuechangerequest . To fill this gap for now, we suggest using data binding to translate user gestures via keyboard or mouse events into changes to attributes on the custom elements, which are then reflected in the ARIA attributes accessible via platform APIs.

That's another dead loss. For example, if I want to automate my bank login sequence, I have to be able to query what's being displayed and find out what security question is. That text is simply part of a paragraph. once I can read the text, then I need to generate a different response. Doing so by injecting keystrokes has serious reliability problems. I've learned with 20 years experience with speech recognition that keystroke injection or GUI elements automationnnever fails to disappoint.

 

> 3) do something. This is usually the action associated with a link or 

> button but can also drive mouse over or any other event causing an action. 

Similarly, acting on an element is currently handled via keyboard and mouse events, and this could be supported at a higher level by something like IndieUI actions (http://www.w3.org/TR/indie-ui-events/#actions). Currently, Polymer elements listen for mouse and keyboard events which are used to drive actions on the element. As you say, these events can be simulated by assistive technology via the platform accessibility APIs. We do recommend having a robust keyboard story and ensuring that elements are focusable, to avoid having to perform fiddly mouse interactions.

In the example I gave above, I eliminated many element activations thereby increasing reliability and decreasing time to execute. I don't have to go through a series of selections and mouse and keyboard events simulations to get to where I need to be.

 

> 4) tell me when something changes. These event notifications allow you

> to use hand/mouse at the same time as speech and it lets the speech 

> system stay in sync with what's being displayed. 

ARIA provides an aria-live attribute, which is also implicit in certain role values, to notify users when the content of certain regions changes.

Sounds useful. This is the kind of thing you would use when making a change in one window and transforming the results into another window. For example, there may things you want to do with speech but you can't speak because it's not the right form. So, for the moment, imagine a simple text window with an input filter and an output filter. The filters convert to and from a speakable form. When you're in the text window, you can use all the normal text editing commands NaturallySpeaking users have come to expect. When you're done, you push the text over to the application through the output filter and, in theory, everything will be nice inside the application environment.

This notification system could be useful to keep the application buffer and the speakable text window synchronized

I would greatly appreciate it if you could let us know if you see any areas where we could improve this strategy to address your needs. Also, it would help to hear more about your specific experience: what browser and assistive technology are you using? Are there any sites which work well for you now which could be used as a model for best practices?

For the most part, I have almost zero accessibility when it comes to websites. I just suffer and my hands burn because is no other way to use speech recognition with browsers without burning out my voice. The only accessibility I have is plaintext dictation if I dictate into a dictation box or a speech-enabled editing tool. That's it. Nuance has some extensions for chrome and Firefox that theoretically provide some of the capabilities you've described but, remember, head-up-ass, they release the extensions and didn't bother to notice that it causes system failures (browser lockup, core dumps etc.) anytime flash is enabled. Who would've thought that flash would be running on the same system as speech recognition???

Thanks,

Alice

Thanks for continuing the dialogue. I have 20 years of frustration about how badly accessibility interfaces are implemented and as someone who lives the life, I get even more frustrated when the simple  intent is  replaced with the complex incomplete solution.

I know some of my UI ideas are nontraditional. I started with the Wang freestyle project and I would have gone father in the UI world because I seem to have a talent for creating simple, intent-based UI designs. Sadly, my hands busted 1994 and I've spent the past 20 years trying to climb the hill of making it possible to get back to doing UI work using speech recognition.

[Ryan Gahl]

Google, sounds like you should hire Eric to help you get this right :)

[Eric]

On 4/17/2014 10:20 AM, Ryan Gahl wrote:
> Google, sounds like you should hire Eric to help you get this right :)

how kind of you to say that. I don't think google want me because 1) I
do not do well on trivial pursuit employment tests and 2) I'm an uppity
crip. ;-) but for others, I'm starting a consulting practice focusing
on intent/task driven UI with an accessibility awareness. in3x will be
live in the next month.

first project with spare money will be building a chrome plugin
implementing the 4 API model mapped onto the naked DOM to demo/learn if
I'm on the right track and if I need to change anything from what I
learned. I'm also involved in the local unemployed community, helping
where I can. to implement this project, I'd love to hire someone who
has been out of work for a couple of years and give them a boost back
into the world of work.