IAccessible / IUIAutomation

[elwinwi...@gmail.com]

Hi,

I am trying out under Windows 7 the accessibility of Chrome but in many ways having problems with accessibility under windows using uiautomationcore.dll the windows implementation.
It seems chrome has many accessibility features not fully implemented.

textbox within html seems not to be supporting the name attribute / not recognizable by name (open www.google.name and try to get the name of the search / q textbox)

When I have a pattern like LegacyIAccessible according to be available as ILegacyIAccessiblePatternAvailable=true i am unable to call IAccessible.setvalue and get error not_implemented
Any ideas if this is wrong in what I am doing (same stuff works properly on notepad for example) to retrieve and set textbox value.

Some Is...PatternAvailable gives true back but when calling the functions below the pattern they seem not to be implemented

regards,

Elwin

[Dominic Mazzoni]

Elwin,

Try opening chrome://accessibility and click the link to enable accessibility support. Does it work now?

Chrome attempt to detect screen readers and magnifiers and automatically enable accessibility support. Were you trying to use these APIs for automation? If so, you can try launching Chrome with --force-renderer-accessibility on the command line instead.

If you are still seeing issues even when you're sure Chrome accessibility is enabled, we can file bugs.

- Dominic

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-accessib...@chromium.org.

[elwinwi...@gmail.com]

Dominic,

 

I allways start chrome with the option for accessibility and check the tree in chrome with chrome://accessibility

 

Even when I try out of the box with Windows SDK tool inspect.exe I see those issues. So in general I have access / recognize the html elements but I can see I miss attribute values like name for a textbox and I cannot use the action patterns although the Is...PatternAvailable methods return initially true.

 

For example I filed this issue (but no clue if somebody will act on that quickly)

http://code.google.com/p/chromium/issues/detail?id=271248

 

So I make use of

http://msdn.microsoft.com/en-us/library/windows/desktop/ee671406(v=vs.85).aspx

using scripting language AutoIT

 

1. Getting the element goes fine (although textboxes cannot be found by name, so i just traverse the tree with a treewalker)

2. Getting the patterns goes fine (at least I get access to those objects, using IUIAutomationElement::GetCurrentPattern method)

for example patterns found

UIA_IsLegacyIAccessiblePatternAvailablePropertyId ( returns IUIAutomationLegacyIAccessiblePattern )

http://msdn.microsoft.com/en-us/library/windows/desktop/ee696074(v=vs.85).aspx

UIA_IsValuePatternAvailablePropertyId (returns IUIAutomationValuePattern)

3. But once I have the pattern and for example try to use on

a. www.google.com

b. the search textbox

c. use the setvalue method from the LegacyIAccessiblePattern

http://msdn.microsoft.com/en-us/library/windows/desktop/ee696098(v=vs.85).aspx

 

Easiest way to reproduce/see this

is by selecting the element in inspect.exe (out of windows SDK) and in the actions menu choose for one of the setvalue options in the menu on the searchtextbox of www.google.com

 

regards,

 

Elwin

[Dominic Mazzoni]

Try a different textbox. I think that one may not have a name.

I think that SetValue may not be implemented. I filed a bug for that here: http://crbug.com/280174

- Dominic

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-accessib...@chromium.org.

[elwinwi...@gmail.com]

Tried a different textbox. Still incorrect (but I think i know better understand the issue for me still a bug)

Good: As aria-label is filled

If I check in this forum the Search for topics textbox I get for the name: Search for topics

If I check with left mouse, inspect element I see HTML with attribute name ="q" 

<input id="gbqfq" class="gwt-SuggestBox GGYB0KICF1B" name="q" type="text" autocomplete="off" value="" placeholder="Search for topics" aria-label="Search for topics" dir="">

Incorrect:

When I go to www.google.com and inspect the searchtextbox (which is empty) i see the name attribute and no aria-label

I see: <input id="gbqfq" class="gbqfif" name="q" type="text" autocomplete="off" value="" style="border: none; padding: 0px; margin: 0px; height: auto; width: 100%; background-image: url(data:image/gif;base64,R0lGODlhAQABAID/AMDAwAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw%3D%3D); background-color: transparent; position: absolute; z-index: 6; left: 0px; outline: none; background-position: initial initial; background-repeat: initial initial;" dir="ltr" spellcheck="false">

Logic should be

• if aria-label is available use that for name
• if name is available use that for name
• otherwise use value of the textbox

I will post a different list of issues I see on the accessible and pattern implementation as not only setvalue is not working.

[Dominic Mazzoni]

On Tue, Aug 27, 2013 at 11:54 AM, <elwinwi...@gmail.com> wrote:

Logic should be

• if aria-label is available use that for name
• if name is available use that for name
• otherwise use value of the textbox

No, "name" is not supposed to be a user-visible name. It's only used for the query string or form submission.

The specification for accessible name calculation is here - Chrome tries to be as consistent with this spec as possible:

http://www.w3.org/TR/wai-aria/roles#namecalculation

I will post a different list of issues I see on the accessible and pattern implementation as not only setvalue is not working.

Great!

For faster debugging, try visiting chrome://accessibility - that will show you the full accessibility tree that Chrome is exposing to IAccessible.

For best results, create a relatively simple HTML page, and post both the html and the output of chrome://accessibility to the bug, along with what you think the results should be.

- Dominic

[elwinwi...@gmail.com]

thx.

Any clue why this is this way in the W3C specification and where I can drop argumentation to change?

• The way its specified it leads for a textbox allways to a situation that attribute value and attribute name are equal and even attribute name changes when you type data in the textbox.
  

From a screen reader perspective on a bankform with 6 textcontrols I would like to be able for example to identify  

accountFrom, accountTo, amountDollar, amountCents, Description1, Description2

but as aria labels are not there and all fields are initially empty I cannot find control by name

[elwinwi...@gmail.com]

only when aria label is not there i ment

[Dominic Mazzoni]

Chrome supports IAccessible2 and ISimpleDOM, both of which give you ways to access the associated label element and all of the attributes of each element.

http://accessibility.linuxfoundation.org/a11yspecs/ia2/docs/html/interface_i_accessible2.html

Are there APIs like that in UI Automation? If so we could support those, too. But as currently defined, MSAA / IAccessible does not provide any way to access the control "name".

- Dominic

On Tue, Aug 27, 2013 at 3:46 PM, <elwinwi...@gmail.com> wrote:

only when aria label is not there i ment

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-accessib...@chromium.org.

[elwinwi...@gmail.com]

As I am on the Windows 7 and 8 platform I have to use the Microsoft technology and they are going into the direction of IUIAutomation for accessbility. Its a huge API which would be very nice if chrome would support that.

 

Three links below describe this

What an application should do to support MS UIA framework

http://msdn.microsoft.com/en-us/library/windows/desktop/ee684084(v=vs.85).aspx

The whole description

http://msdn.microsoft.com/en-us/library/windows/desktop/ee671585(v=vs.85).aspx

getting a property like name should be done by supporting this method

http://msdn.microsoft.com/en-us/library/windows/desktop/ee696040(v=vs.85).aspx

IUIAutomationElement::GetCurrentPropertyValue method

 

I am automating it from autoitscript and put example(s) in this forum

http://www.autoitscript.com/forum/topic/153520-iuiautomation-interface-microsoft-framework/

 

As sometimes I try to find by name

http://msdn.microsoft.com/en-us/library/windows/desktop/ff625913(v=vs.85).aspx#FindByName

    // Get a top-level element by name, such as "Program Manager"
    IUIAutomationCondition* pCondition;
    hr = g_pAutomation->CreatePropertyCondition(UIA_NamePropertyId, varProp, &pCondition);
    if (FAILED(hr))
        goto cleanup;

    pRoot->FindFirst(TreeScope_Children, pCondition, &pFound);

• Then I will not find anything within chrome on textboxes if aria attribute is not there (as value in the textbox is unpredicatable or empty)

 I have workarounds by making my own Finding logic (by using a treewalker) but thats slower and more work and maintenance

[dan.gu...@gmail.com]

Hello,

Can you please show a sample of the treewalker?

I am using FindFirst and I am not happy with the results.

Thanks,

Dan.

[elwinwi...@gmail.com]

Seehttp://www.autoitscript.com/forum/topic/153520-iuiautomation-interface-microsoft-framework/

[thin...@gmail.com]

Hi, Elwin,

    This problem solved?

Thx

在 2013年8月27日星期二UTC+8上午5时04分48秒，elwinwi...@gmail.com写道：

[elwinwi...@gmail.com]

definition of problem solved is relative. Chrome is not having full support for IUIautomation and hopefully this will increase in future

In this thread you can follow what I do with AutoIT in combination with the IUIautomation stuff

http://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/

I worked around the limitations by using a combination of

IUIAutomation

IAccessible

IAccessible2

ISimpleDom

offcourse it would be better if chromium supports the whole IUIAutomaton direction of microsoft under windows as thats the direction MS is going

Looking at the chromium source it seems only this is supported regarding the patternproviders

1. STDMETHODIMP BrowserAccessibilityWin::GetPatternProvider(PATTERNID id,
   
2.                                                          IUnknown** provider) {
   
3.   DVLOG(1) << "In Function: "
   
4.            << __FUNCTION__
   
5.            << " for pattern id: "
   
6.            << id;
   
7.   if (id == UIA_ValuePatternId || id == UIA_TextPatternId) {
   
8.     if (IsEditableText()) {
   
9.       // The BrowserAccessibilityManager keeps track of instances when
   
10.       // we don't want to show the on-screen keyboard.
    
11.       if (!manager()->IsOSKAllowed(GetGlobalBoundsRect()))
    
12.         return E_NOTIMPL;
    
13. 
    
14.       DVLOG(1) << "Returning UIA text provider";
    
15.       base::win::UIATextProvider::CreateTextProvider(true, provider);
    
16.       return S_OK;
    
17.     }
    
18.   }
    
19.   return E_NOTIMPL;
    
20. }