Intent to Implement and Ship: UIAutomation Provider Mappings

[Rossen Atanassov]

Contact emails

Rossen.A...@microsoft.com, Melanie....@microsoft.com

Explainer
UI Automation Explainer

Design doc/Spec
UI Automation Providers Overview

A TAG review is not requested, as these interfaces are specific to the Windows platform and not intended to be introduced as web standards. It is not expected that web developers will need to make changes to their content as a result of this work, as UIAutomation support reflects lower-level implementation details, and will not be replacing or removing accessibility APIs already supported in Chromium.

Summary

Microsoft UI Automation (UIA) provides programmatic access to most user interface (UI) elements of desktop applications, as well as web content and web applications. This API enables assistive technology (AT) products, such as screen readers, to provide information about applications, their UI and contents to end users. With this information ATs can allow user to manipulate applications by means other than standard input.

 

At a high level, UIA achieves its functionality by exposing two sets of APIs, provider APIs, those implemented by a web browser for example, and client APIs, those implemented by an AT. This document’s focus is on implementing the provider APIs inside Chromium. These APIs are not exposed to web developers, and it is not expected that web developers should change the way they build sites and web apps—these APIs are meant to map web content into a format useful to C/C++ programmers.

 

Motivation

A UIA implementation benefits the browser/web platform, AT clients, and ultimately the end user. By supporting UIA, Chromium-based browsers can:

• Ensure that their platform is available to users who choose to browse with UIA-powered ATs, such as Narrator.
• Provide accessibility mappings in a secure fashion.
• Take advantage of the official Windows accessibility API.

With UIA, ATs can enhance web browsing for their users in Chromium-based browsers. ATs can:

• Innovate on top of rich text-level interaction and smooth reading experiences.
• Spend less time writing to different accessibility APIs, and more time on core capabilities.
• Take advantage of a continually-evolving platform, and the performance gains and extended capabilities that come along with it.

For extended background on motivations, please refer to the explainer.

Risks

Interoperability Risks
This work has no impact to the core web platform exposed to JavaScript developers. As such the risk of introducing interoperability gaps in the web layer is low to none.

A UIA implementation has no bearing on browser support for any other assistive technology platform APIs, such as MSAA and IA2, as these APIs are distinct from UIA. While interoperable support would be ideal for primarily-UIA-based AT experiences, other browsers are free to choose to support other accessibility APIs, in addition to or instead of UIA.

• Edge: Shipped
• Chrome: In progress
• Firefox: No signals
• Safari: N/A
• Web / Framework developers: N/A

Compatibility Risks
Same as the interoperability risks.

Ergonomics

UIA doesn’t pose any specific performance concerns that aren’t applicable to any other assistive API, such as IA2.

 

In order to extend UIA support from basic to full coverage, this work will continue to leverage Chromium’s platform-agnostic accessibility tree abstraction layer, and will map tree elements to the appropriate UIA control types. Support of UIA interfaces will be consistent with other UIA providers, as outlined in UI Automation Providers Overview. For an overview of how web content semantics will map to UIA, refer to the following mapping specifications: Core-AAM, HTML-AAM, Graphics-AAM, and SVG-AAM.

Activation

Since the feature isn’t exposed to the web API layer, it has no bearing to web developers nor is it expected to pose any challenges or additional requirements to them.

 

In Chromium today, accessibility API code is activated only when ATs are attached, or the user has specified via [browsername]://accessibility that the browser should be run in accessibility mode.

 

Debuggability

UIA is implemented and debugged at the browser layer, thus is not intended to be exposed to DevTools. At the same time, it is useful to allow web developers to observe the information provided to UIA–the control types, properties and supported patterns. Since DevTools support isn’t critical to this feature, it can be considered TBD. However, it is reasonable to expect that these DevTools will be similar to representations found cross-browser today:

• A representation of the current document’s accessibility tree
• A list of key-value pairs for the computed accessibility mappings of the currently-inspected node

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No. UIAutomation provider APIs are specific to Windows and will only be supported on Windows 7+ platforms.

Is this feature fully tested by web-platform-tests?

This feature is not a web standard, but mappings from ARIA and host language semantics into UIA can be tested. The relevant test suites include:

• CORE-AAM-1.1
• CORE-AAM-1.2
• GRAPHICS-AAM

Link to entry on the feature dashboard

Don’t have one yet.

Requesting approval to ship?

Yes.  The functionality will be guarded by experimental flag and won't be enabled by default.

 

Thanks,

Rossen

 

[Dominic Mazzoni]

Hi Rossen,

Thanks for sending this out! You probably discovered that we did begin implementing this in Chromium already, but we really only scratched the surface, and we'd love to work with you to complete it.

Technically I don't think we need formal approval for the Intent-to-implement because as you noted, it won't affect the web platform.

I'm looking forward to diving into your first patch.

- Dominic

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/DM5PR21MB0139BF54497D72E0A923AF5E9BAB0%40DM5PR21MB0139.namprd21.prod.outlook.com.

[Harald Alvestrand]

Asking because I don't know these technologies:

- What's the security model for these APIs? In particular, how do we ensure that a malicious program claiming to be an assistive technology tool isn't able to bypass Web security mechanisms? Or if that is impossible - how can a normal user discover that assistive technologies that he didn't intend to authorize are in use?

- Are these APIs, and the tooling that has to be in place in order to support them, relevant for the development of testing APIs such as WebDriver? If yes - are the efforts coordinated?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFz-FYy96NXu5zT%2BNz-1q3xGot0%2B_Hs7VUPqp3ATwJ4T%3DJ%3D99w%40mail.gmail.com.

[PhistucK]

Regarding the first point, it is not a threat model Chrome is trying to protect against, as far as I know. A malicious program might as well patch Chrome to have additional amazing abilities, or apply a system proxy that intercepts the entire browsing transport (except WebRTC?).

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVF0CLDS_DA9CMG2%3DPd59pUbL-pH%2B0R9c8RgH%3DN%3DEmp-fg%40mail.gmail.com.

[Daniel Bratell]

Intent to Implement is always good, as it connects interested parties and allows early feedback of the kind dmazzoni was able to provide. A shipping approval by API owners is not necessary though since this will not change the web platform. On the other hand it doesn't hurt to play it safe.

>  Link to entry on the feature dashboard

> Don’t have one yet.

This is good to have though since it sounds like a feature that might be worth tracking, and later document in a release. There should also be a crbug and there might already be one but I only found https://crbug.com/675899 which is about Windows Narrator not talking. You will be asked for a bug number when you file an entry on the feature dashboard.

/Daniel - with the api owner hat on

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_JpV3ws7S1x%3DYr%2BhyUTtmx5oyFZWDomECusQSi6%3D8y5ig%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Rossen Atanassov]

Thank you all for the quick engagement! Let me try and address some of the points being made.

 

> Thanks for sending this out! You probably discovered that we did begin implementing this in Chromium already, but we really only scratched the surface, and we'd love to work with you to complete it.

 

You’re most welcome Domenic! We’re looking forward to landing all of this work into Chromium and enabling more users on the web as well! It’s going to be a fun project for sure :)

 

> What's the security model for these APIs? In particular, how do we ensure that a malicious program claiming to be an assistive technology tool isn't able to bypass Web security mechanisms? Or if that is impossible - how can a normal user discover that assistive technologies that he didn't intend to authorize are in use?

 

This is a great question. In general there could be two possible vectors of attach I can think of here. One is through manipulating the provider-side UIA interfaces and the other is by registering as a UIA client and using their client-side API to manipulate the browser and extract data. Both scenarios run under the assumption that you have running user-mode code and are able to send us WM_ messages (to obtain the UIA provider-side object through WM_GetObject at least).

 

One assertion to be made is that in this case the user would be compromised a lot more than what UIA could offer. In essence, if user-mode code is already executing an attacker would already have access to local disk, screen and other ways to obtain user data. In fact, the data passed through UIA interfaces (in a compliant impl) shouldn't differ from that we expose to the visual layer. If a system proxy is able to intercept calls to DierectWrite for example and collect string user data being, intercepting UIA messages shouldn't be different.

 

All in all, the threat model of a UIA provider shouldn't be any different than that of IAccessible (MSAA) or IAccessible2, both of which are already implemented and present in the Chromium project. I'd be happy to dive deeper if you think is necessary.

 

> Are these APIs, and the tooling that has to be in place in order to support them, relevant for the development of testing APIs such as WebDriver? If yes - are the efforts coordinated?

 

Extending WebDriver itself to support the provider-side interfaces will be unnatural as that means we bypass UIAutomation logic itself. Extending WebDriver with an extension that acts as a UIAutomation client and collects/verifies test data is something we considered doing down the road. However, none of these should require changes to WebDriver’s API surface itself.

 

> This is good to have though since it sounds like a feature that might be worth tracking, and later document in a release.
> There should also be a crbug and there might already be one but I only found https://crbug.com/675899 which is about
> Windows Narrator not talking. You will be asked for a bug number when you file an entry on the feature dashboard.

 

Thank you Daniel. This is super helpful and quite frankly I didn’t know where to start but was hoping someone with more experience will point me at the right direction. I will make sure to follow up on it.

 

Thanks,

Rossen

[Harald Alvestrand]

Thanks for the thoughtful responses! Seems like this does not give an attacker handles he did not already have, so no additional worrying needed!

[Arvind Murching]

Does this need an entry in the feature dashboard?  Who is expected to add this?

Thanks

Arvind

[catt.comm...@gmail.com]

Wondering what happened to this... is it being worked on?

Cheers,

James

[ra...@microsoft.com]

Great and very timely question. All of the APIs are implemented at this point. We are working through addressing some edge cases to match different AT expectations and will be transitioning to i2e soon after. 

Thanks,

Rossen