This is a part of the ticket #31617
--
Ticket URL: <https://code.djangoproject.com/ticket/33620>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
* status: assigned => closed
* resolution: => duplicate
Comment:
IMO a separate ticket is not needed, please use `Refs #31617 --` in your
patch.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:1>
Comment (by Carlton Gibson):
Ooops. I asked Sarah to open a separate ticket, because the multiple
conversations around the large scope accessibility tickets were getting
too hard to follow. Sorry 😬
Either way, thanks for picking this up Sarah. Yes, adding the CI tools is
a good step to help us move forward.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:2>
* status: closed => new
* resolution: duplicate =>
* stage: Unreviewed => Accepted
Comment:
Fine, a separate ticket works for me too :D
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:3>
Comment (by Thibaud Colas):
Just seeing this now – this does feel like something that warrants its own
ticket in my opinion. Evaluating the right tool is going to be a big task
in its own right, let alone setting it up locally, and in CI.
I can provide a few options I would recommend considering:
- [https://github.com/mozilla-services/axe-selenium-python axe-selenium-
python] seems worth considering, since Django already uses Selenium. I
haven’t worked with Selenium in years myself so don’t really have the
ability to judge how good of a fit this is. At a high level, all this does
is injecting the Axe script onto the page and running it, with a nice API
to configure Axe and get a useful report.
- Puppeteer + Axe is what I have the most experience with myself,
generally running within Jest to structure a test suite. This is very
similar to the above but with JS tooling rather than Python.
- Playwright + Axe is also popular these days. Playwright comes with its
own testing framework contrary to Puppeteer.
One important aspect is the need to test pages protected by login. To test
public-facing pages, we could also consider:
- [https://github.com/pa11y Pa11y], an opinionated "Puppeteer + Axe" CLI
tool.
- [https://darekkay.com/evaluatory/ Evaluatory], same but with Playwright
My
[https://github.com/thibaudcolas/django_admin_tests/blob/main/pa11y/test.js
demo test suite] uses Pa11y + Puppeteer + Lighthouse. Since both Pa11y and
Lighthouse are built upon Puppeteer, they can be used via their JS API so
we can first use a Puppeteer script to log in, then hand over to those
tools.
---
To go through this, I’d recommend the following initial steps:
1. Confirm what exactly is needed between a testing tool (write
assertions, break the build if fails) and a reporting tool (tests a given
page and reports results)
2. Check whether there are any restrictions on what can run in CI (I can’t
see whether Selenium tests run there)
3. Decide whether to use a Python Selenium integration or a JS-based tool
(looks like [https://github.com/django/django/blame/main/package.json#L13
Django already uses Puppeteer?]).
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:4>
Comment (by Sarah Abderemane):
Thank you Thibaud for your suggestions and recommendations !
> 1. Confirm what exactly is needed between a testing tool (write
assertions, break the build if fails) and a reporting tool (tests a given
page and reports results)
My guess was to do both, to be able to test fields separately and also the
hole admin to view improvements needed for templates of the admin to keep
the admin accessible. I may be wrong, but not sure how to test correctly
fields / inputs separately.
> 2. Check whether there are any restrictions on what can run in CI (I
can’t see whether Selenium tests run there)
Actually there are javascript tests run with QUnit and those tests are
already run in the pipeline, I just add another workflow dedicated for
accessibility. I started to implement
[https://github.com/django/django/compare/main...sabderemane:ticket_33620?expand=1
Axe with QUnit], may be not the best way. I can go for another solution as
you have suggested.
For restrictions it's not a problem, Tom suggest me to use his
[https://github.com/knyghty/django-admin-demo django admin demo], I can
run his demo in the pipeline and launch a reporting tool on it. I can show
result in console and even add a report in a file which will be an
artifact keep a defined period if we want. I just have to pay attention to
the duration of the job.
> 3. Decide whether to use a Python Selenium integration or a JS-based
tool (looks like Django already uses Puppeteer?).
I think QUnit is run via puppeteer with Grunt, that's why I started my
implementation based on this tool but we can decide to go for something
else if it's not relevant.
''Since both Pa11y and Lighthouse are built upon Puppeteer, they can be
used via their JS API so we can first use a Puppeteer script to log in,
then hand over to those tools.'' => Yes, I think with the use of the
reporting tool, I can go for it since Puppeteer is already in the project.
I don't have any particular opinion about this.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:5>
* cc: Tom Carrick (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:6>
* owner: Sarah Abderemane => Tushar
* status: new => assigned
Comment:
Hey, I was digging into this and was able to get aXe run with the current
testing suite using axe-selenium-python. I am not sure about the scope of
checking the accessibility of the admin panel, and what views and admin
page state are we looking to test.
It could be 0 violations on the first admin landing page and then we might
get violations when we visit a specific group or table or other section of
the admin dashboard.
What I have in my mind is after every selenium test that is run, run the
aXe accessibility test as well by using a decorator probably. Please share
your views on how you would want it.
cc @Carlton Gibson
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:7>
Comment (by Sarah Abderemane):
Hi Tushar,
Thank you for your interest on the subject :)
Just for people who read this ticket, a part of the discussion was out of
the ticket, we agreed with Thibaud and Tom, the admin should be tested as
long as there is a change on it. We should have a page or something to be
able to test each component like (input, search ...) but also test the
full admin to get the best accessibility and avoid regressions in the
future.
IMO, from the pipeline we should be able to generate a report which could
be downloaded with artifact on Github Actions.
I didn't go further on the subject I was trying to launch the admin from
the project and avoid to use another repo/dependencies for that. It's
still in progress (and I didn't have time to check more yet).
> What I have in my mind is after every selenium test that is run, run the
aXe accessibility test as well by using a decorator probably. Please share
your views on how you would want it.
I'm not very familiar with Selenium, but I think you can do assertions
directly in tests but I guess some tests may be not really useful for
accessibility testing but useful for the purpose of the initial test and
vice versa. It really depends on the selenium test to me.
Do you have the code somewhere to check ? I'll be happy to help you on
this subject.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:8>
Comment (by Tushar):
I have made a draft PR to get some comments and better clarity on what
could be done. https://github.com/django/django/pull/16372. Linking it
here manually as issue tracker is not picking it up (I did something wrong
probably)
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:9>
Comment (by Carlton Gibson):
Thanks Tushar, this looks interesting. Let's see what folks say.
I adjusted the PR title to start `Fixed #33620 -- ...` which allows Trac
to pickup that it links to this issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:10>
Comment (by Tushar):
pa11y seems to be much better than using axe. pa11y can be called directly
from CLI on URLs, we can use axe-core as a pa11y runner, it can output a
decent formatted result, and have options to customize them even further.
Apart from that, I see pa11y-dashboard as well to visualize results if
needed.
After every UI change, writing a helper function that calls pa11y on the
current selenium instance will not take much effort.
I highly favour using the existing admin tests for accessibility as
compared to writing and maintaining accessibility tests.
One issue that I see with using pa11y as described above is redundancy in
pa11y results because every time we would call pa11y, it would scan the
whole page instead of only the new UI change.
pa11y has an option rootElement to only test a subset of DOM but this
would require the use of "#" id tag to identify the elements that I doubt
might be present on every UI piece of admin.
Other way than writing a whole accessibility testing suite is to iterate
overall the .html files in the project and run pa11y over them to generate
results. This might give us a lot of false positives like missing head tag
or title tags due to .html file being in jinja template format.
Let me know your thoughts on the above points before I go ahead with
something.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:11>
Comment (by Sarah Abderemane):
Hi Tushar,
Sorry for the late reply.
Pa11y seems a good option to me, it's well maintained and Thibaud already
suggested it previously so I think you can go ahead.
> Other way than writing a whole accessibility testing suite is to iterate
overall the .html files in the project and run pa11y over them to generate
results. This might give us a lot of false positives like missing head tag
or title tags due to .html file being in jinja template format.
I think it's a very good idea to re-use the existing test suite rather
than maintaining whole new ones but pa11y should be run on rendered pages
and we may have specific tests case to test correctly accessibility for
the whole page. A small change can affect the rest of the page and we
should be aware of that according to me.
I suggest to use the existing tests if possible, for those when we need a
dedicated test a part, we will add it to an accessibility testing suite.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:12>
Comment (by Tushar):
Hi folks, I have made a draft PR to communicate my approach, would love to
have some feedback there. PR link -
https://github.com/django/django/pull/17074
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:13>
Comment (by Thibaud Colas):
Following up from
[https://github.com/django/django/pull/16372#issuecomment-1357132971
#16372#issuecomment-1357132971], particularly with another new approach
proposed by @Tushar in [https://github.com/django/django/pull/17074
#17074], here are the points of decision I would recommend making before
we proceed with coding, ordered based on how much I think they influence
one-another:
1. **Test outcome**: whether the accessibility checks will fail the build
if there are issues, or only provide a warning / reporting
2. **Test Output**: either pass/fail of specific checks, or JSON or HTML
report(s), or both, or something in-between
3. **Test suite**: reuse of the existing Selenium tests (like in
[https://github.com/django/django/pull/16372 #16372],
[https://github.com/django/django/pull/17074 #17074]), or something
separate altogether (like in [https://github.com/django/django/pull/16115
#16115]).
4. **Accessibility checker**: Axe, Pa11y (HTML Code Sniffer + Axe), IBM
Equal Access, Alfa, etc
5. **Configuration of the checks**: either "run all supported checks", or
opt-in / allow-list "run specific subset that pass", or opt-out /
disallow-list "run all minus the ones that fail".
---
The ideal scenario to me would be for previously-undetected accessibility
issues to fail the build, even if that means only running a subset of
possible accessibility checks in CI. With this in mind, I think the best
setup is:
1. Test outcome: fail the build
2. Test output: pass/fail of specific checks in the command line output
3. Test suite: reuse of the existing Selenium tests (for consistency, and
because we might as well)
4. Checker: Axe (most configurable, usable in lots of different scenarios)
5. Configuration: opt-out – run all checks where possible. Exclude
problematic elements or checks as needed, scenario-by-scenario
With this in mind, of the three attempts so far my preference is for
[https://github.com/django/django/pull/16372 #16372] with
[https://github.com/mozilla-services/axe-selenium-python axe-selenium-
python]. If we have concerns about its maintenance
([https://github.com/django/django/pull/16372#issuecomment-1343332628
#16372#issuecomment-1343332628], I’d say we should investigate the
specific concerns further and either re-implement the functionality (200
LOC) in django/django, or fork, or approach someone at Mozilla to
[https://github.com/mozilla-services/axe-selenium-python/issues/192
discuss maintenance]. cc @smithdc1 who raised this initially – and we also
discussed this together in Edinburgh.
I’m also interested in having more comprehensive automated reports of
Django’s accessibility, though
[https://github.com/thibaudcolas/django_admin_tests
thibaudcolas/django_admin_tests] has worked well enough so far that it
didn’t seem as pressing to me. If we wanted full reports as an "official"
implementation, I’d recommend proceeding with
[https://github.com/django/django/pull/16115 #16115]. Ideally towards the
goal of having a full-blown official demo site we could use to
troubleshoot those issues in a reliable way, rather than contributors
having to come up with their own test suite (cf.
[https://github.com/knyghty/django-admin-demo
knyghty/https://github.com/knyghty/django-admin-demo]).
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:14>
Comment (by Tushar):
To bring in other conversations here as well, @David also shared this blog
https://blog.pamelafox.org/2023/08/accessibility-snapshot-testing-for.html
which seems like a good approach for setting up a baseline and testing
against it. Also, in a single selenium test, there are multiple instances
within the test that should be tested for accessibility. For example when
a new UI element gets visible or gets deleted.
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:15>
Comment (by Thibaud Colas):
Sarah, Tom and I met a couple weeks ago to discuss this and are generally
in agreement that [https://github.com/django/django/pull/16372 #16372]
with [https://github.com/mozilla-services/axe-selenium-python axe-
selenium-python] is the best approach for the time being.
As far as snapshot tests, that sounds great to me if we’re ok with using
snapshots in Django :)
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:16>
We have an update! After contacting the DSF board back in November 2023,
we now have our answer that moving [https://github.com/mozilla-services
/axe-selenium-python axe-selenium-python] to the Django org in GitHub
wouldn’t be appropriate. As it’s not a Django-related package, we don’t
want to risk its maintenance falling to the Django ops team.
So -- the accessibility team will proceed another way. Currently we’re
planning to still try to take over maintenance of the package, but in a
separate GitHub organization so it’s clearer whose role it is to maintain
it.
@Tushar would you be up for helping with maintenance as well?
--
Ticket URL: <https://code.djangoproject.com/ticket/33620#comment:17>