Reusing chromium accessibility tools for cross-platform, cross-browser Core-AAM test suite

[Valerie Young]

Hi everyone,

I’m looking into reusing some of chromium’s content_browsertest code to write a new test harness/test suite for the Core-AAM standard. The Core-AAM standardizes the mapping between ARIA roles, attributes, states and events and the platform accessibility API generated by a browser.

The current Core-AAM tests in WPT are difficult to write, difficult to read and difficult to run -- so Joanie had the idea to re-use the chromium `ax_dump_tree` tool, especially after the recent work Alex has done on it, to create a new cross-platform test suite for the standard. Some of the content_browsertest functionality has already surfaced in `ax_dump_tree` and `ax_dump_events`.

I've taken up the task, and here are some technical details if you are curious or want to weigh in! The general plan is:

1. Write a NodeJs test harness that loads a test file sequentially in all browsers supported on the given platform and runs ax_dump_tree against each browser. The result is compared to an expectations file for the accessibility AP being testedI. I have a prototype of this.

2. Design tests very similar to chrome's Dump Accessibility Test (content_browsertests) (which test “internal nodes”, where as Core-AAM needs to test “external nodes”). Already, you can write "tree tests" with the ax_dump_tree tool and do some filtering of attributes.

Currently limitations and proposed additions to ax_dump_tree:

ax_dump_tree dumps the entire accessibility tree, and it would be better for us to be able to test a specified node in the tree. I'm currently looking into ways to do this. One idea is to go in the "dump node" chrome_browsertest direction, and filter based on DOM id, and another is to implement something similar to the @MAC-SCRIPT directive.

If you are interested and would like to follow along or advise, I'd appreciate it! Or if you have concerns/ideas on directions to take this test suite in, please share :)

Valerie

[Aaron Leventhal]

Hi Valerie, thanks for working on this. Being able to dump part of the tree would be good. 

Which way are you leaning as far as the subtree filter?

We do have @NO_DUMP and @NO_CHILDREN_DUMP -- maybe something like that?

(Incidentally I noticed that those use underscores instead of hyphens like everything else — we should fix that).

Aaron

[Alexander Surkkov]

@NO_DUMP instructions are good but there's some inaccuracy around them, in particular they partially overlap with @DENY-NODE filter. I sent an email about the issue a while ago to the group [1], so perhaps now it is a good time to revisit it.

Otherwise personally I lean towards to reusing scripting directive since it allows keep tests straightforward. It'd be nice to see a bunch of examples for both approaches, it might be easier to make a decision.

Alexander.

[1] https://groups.google.com/a/chromium.org/g/chromium-accessibility/c/potKGeihmaU/m/Ove1ppOCAwAJ

[Valerie Young]

Hi Aaron,

There is Alex's concern, and I'm also don't think I can just use @N0_DUMP", "@NO_DUMP_CHILDREN" and "@...-DENY-NODE", because they all specify what not to dump, and I want to specify what TO dump.

If I don't go the `@...-SCRIPT` route (Alex's preference), I thought I would do something more like the dump node tests, and filter based on `id="test"`. I could maybe create "@...-ONLY-NODE" and specifying an ID, instead of a platform specific property/attribute pair.

Valerie

[Aaron Leventhal]

Hi Valerie, I like Alex's idea of comparing some examples. What do you think of writing up some options? You don't have to show every example, just the best 2-3.

[Valerie Young]

Whoops I missed that suggestion, but here is an example of the different tests we would have to write for CORE-AAM using the script directive and using the "node test" method:

https://docs.google.com/document/d/1cHGcuiv-cKMgx4wj0eX4_GXLyxo2kyxFDWslklKoLQ4/edit?usp=sharing

[Aaron Leventhal]

I see, so you are really testing specific properties or events on specific nodes. Thanks for the explanation.

Alex wrote:

> Otherwise personally I lean towards to reusing scripting directive since it allows keep tests straightforward. It'd be nice to see a bunch of examples for both approaches, it might be easier to make a decision.

Alex, what are you suggesting that would be simpler than what Valerie wrote? Or, what did you think of it?

- Aaron

[Alexander Surkov]

Ah, it wasn't a new suggestion, I just prefer one option over another one, both are from Valerie's email. Just to illustrate why I prefer one over the other, here's a scripting example from Valerie's doc [1]:

@MAC-SCRIPT
  ariaalert.AXRole
  ariaalert.AXSubrole
@WIN-SCRIPT
  ariaalert.role
@UIA-WIN-SCRIPT
  ariaalert.LocalizedControlType
  ariaalert.ControlType
  ariaalert.LiveSetting
@AURALINUX-SCRIPT
  ariaalert.role

Here's `ariaalert` is the DOM id of an element, @PLATFORM-SCRIPT directives indicate which properties/methods should be called on an accessible element with the given DOM id.

Here's an example of filter-based approach:

@MAC-ALLOW-EMPTY:AXRole
@MAC-ALLOW-EMPTY:AXSubrole
@WIN-ALLOW-EMPTY:role
@UIA-WIN-ALLOW-EMPTY:LocalizedControlType
@UIA-WIN-ALLOW-EMPTY:ControlType
@UIA-WIN-ALLOW-EMPTY:LiveSetting
@AURALINUX-ALLOW-EMPTY:role

It specifies which properties/methods to dump and HTML test portion indicates which accessible element to dump (a tested element has to have DOM id='test').

The scripting approach looks more readable and more explicit with me. I think it should be easy enough to prototype it on mac to see how it works. It will require some extra work to make it working on other platforms because ax_scripting is implmented on mac only.

[1] https://docs.google.com/document/d/1cHGcuiv-cKMgx4wj0eX4_GXLyxo2kyxFDWslklKoLQ4/edit?usp=sharing

--
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-accessib...@chromium.org.

[Aaron Leventhal]

Agree that the script approach is cleaner.