accessibility inspect scripting support

[Alexander Surkov]

Hi.

I am looking to convert the existing NSProtocol API browser_tests [1] to the new ax_inspect scripting language [2]. In particular, the test [3] could be rewritten this way:

text_range: textarea.AXTextMarkerRangeForUIElement(textarea)

marker_0: text_range.start

marker_1: textarea.AXNextWordEndTextMarkerForTextMarker(marker_0)

marker_2: textarea.AXNextWordEndTextMarkerForTextMarker(marker_1)

marker_3: textarea.AXNextWordEndTextMarkerForTextMarker(marker_2)

marker_4: textarea.AXPreviousWordStartTextMarkerForTextMarker(marker_3)

target_selected_marker_range: textarea.AXTextMarkerRangeForUnorderedTextMarkers([marker_3, marker_4])

textarea.AXSelectedTextMarkerRange(target_selected_marker_range)

waitfor AXSelectedTextChanged

textarea.AXSelectedText

The only thing not yet implemented is `waitfor` instruction to stop test execution until an event is fired. So, before doing this I was wondering whether this approach looks good in general and whether it'd be better (if it'd be possible) to transform the test into a bunch of smaller ax_dump_tree/ax_dump_events instead.

Thanks!

Alexander.

[1] https://source.chromium.org/chromium/chromium/src/+/main:content/test/data/accessibility/scripts/

[2] https://source.chromium.org/chromium/chromium/src/+/main:content/test/data/accessibility/readme.md

[3] https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/accessibility_scripts_mac_browsertest.mm;l=271-304?q=set-selection-textarea

[Eugene Rysai]

Hi Alex!

Have you considered using JS for scripting instead of creating a custom language?

We could create bindings for accessibility APIs and write test scripts using a proper programming language.

[Alexander Surkov]

Hi, Eugene.

Yes, it's something I personally like, and I think it should be the right step in ax_inspect scripting evolution. I'm not quite certain about technology though. To keep it browser-independent I thought it should be possible to implement it via nodejs platform plus c++ nodejs extensions to support the accessible object and accessible events concepts. Afaik nodejs is available on all major platforms we take care of accessibility on, so nodejs should be the right platform if I don't miss anything. I'm not quite sure though if there's a better solution.

Having said that, the custom scripting looks like a low hanging fruit. Chromium already has simple parsers for this kind of thing and it requires some adjustments to support accessible events. That's basically why I stick with it. I suppose when we set up with the required features for the accessibility scripting, we could move towards porting it to the JS platform.

[Alexander Surkkov]

I prototyped the thing, so you can do basic NSAccessibility scripting now. Here is an example [1] of a test which generates output like this [2]. You can read more at [3]. Let me know what you think.

[1] https://source.chromium.org/chromium/chromium/src/+/main:content/test/data/accessibility/mac/selection/set-selection-contenteditable.html?q=content%2Ftest%2Fdata%2Faccessibility%2Fmac%2Fselection%2Fset-selection-contenteditable.html

[2] https://source.chromium.org/chromium/chromium/src/+/main:content/test/data/accessibility/mac/selection/set-selection-contenteditable-expected-mac.txt

[3] https://source.chromium.org/chromium/chromium/src/+/main:content/test/data/accessibility/readme.md?q=Readme.md%20path:accessibility#Scripting

[Eugene Rysai]

Hi Alex!

If I understand correctly, it is now possible to run AX scripts "out of process", by relying on system accessibility API (AXUIElementRef and related functions from HIServices).

Are there any plans to make the existing script / dump tree tests to operate in such mode?

There was a regression recently (https://bugs.chromium.org/p/chromium/issues/detail?id=1280252), which should have been caught by the script tests, but wasn't because

AppKit has more complex logic for dispatching accessibility calls (affected by NSAccessibility protocol) than currently implemented for NSAccessibilityElement subclasses.

[Alexander Surkov]

Hi, Eugene.

It is possible, we need though to make a bunch of adjustments in the code to make it working.

Personally I lean towards to think it's a good idea to switch to out-of-process testing since it makes us closer to what AT deal with and will help to avoid bugs like the one you referred to, but also somewhat worried that we may run into bugs in the platform layer (the thing that converts NSAccessibilityElement to AXUIElementRef) which could complicate debugging. I wonder, if we can/should support two modes when running the tests: external vs internal tree.

[Eugene Rysai]

Keeping the ability to run tests in both modes sounds like a good idea: we could keep existing dump/script tests and gradually increase the coverage by more trustworthy "external" tests, skipping problematic ones until resolved.

Please let me know if/when introducing this kind of external testing appears on the roadmap, I would be interested in helping out.

[Alexander Surkov]

Hi, Eugene.

I think the easiest and the most elegant way is to have two mac tree formatters: one for internal tree and the second one is for external tree. It allows you to indicate which tree will be used for ax_dump_tree tests. You can also run a test against both trees in case the trees differ. You cannot though indicate a tree per a test, but if we really want to have this option available then we can adjust ax_dump tree testing suite to support it.

Here's some implementation details. The mac tree formatter lives here [1]. The class can be easily templatized to handle both formatters because the code really depends on a number of functions working with a tree such as AttributeOf/ParameterizedAttributeOf, which can be given as template arguments. Then we could have separate classes one for each tree formatter so that the external tree formatter overrides BuildTree() method (which is used by ax_dump testsuite) to return AXUIElement instance and the internal tree returns NSAccessibiltyElement instance from BuildTreeForSelector (which is used by ax_dump tools). I think a method returning either AXUIElement or NSAccessibiltyElement could be also templatized to keep common implementation for BuildTree/BuildTreeForSelector methods.

If you want to take a shot, it'd be great, I'm happy to help.

[1] https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/accessibility_tree_formatter_mac.h;l=16?q=tree_formatter_mac