[Django] #30974: Selenium asserts should be wait statements
Django
---------------------------------------------+--------------------------
Reporter: Johannes Hoppe | Owner: nobody
Type: Bug | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 1
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+--------------------------
I just took a deep dive into our selenium tests, because ... well... they
are "not deterministic sometimes" ;)
I realized that we use a lot of assertions in our tests, where I believe
we should be using wait statements. Mainly because selenium tests are
asynchronous and don't really when an assertion is true. It seems more
reasonable to be, to simply say, what we expect and give it a reasonable
amount of time before we fail.
So instead of:
{{{
self.assertEqual(
self.selenium.switch_to.active_element,
self.selenium.find_element_by_id('id_name')
)
}}}
We do:
{{{
self.wait_until(
lambda selenium:
selenium.switch_to.active_element ==
selenium.find_element_by_id('id_name')
)
}}}
I know this is not a foreign concept if you have been doing a lot of
Selenium testing. Still, there are many cases where we do it "wrong" and
the test suite fails "sometimes".
I believe the main difference is that in one case you raise an
`AssertionError` in the other a `TimeoutError`. However, we could cast
those to have the tests fail not error.
--
Ticket URL: <https://code.djangoproject.com/ticket/30974>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Johannes Hoppe | Owner: nobody
Type: Bug | Status: assigned
Component: Testing framework | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Johannes Hoppe):
Haha on my first patch push a test failed like this:
{{{
Traceback (most recent call last):
File "/home/jenkins/workspace/pull-requests-
selenium/database/postgres/label/bionic-
pr/python/python3.7/tests/admin_views/test_autocomplete_view.py", line
256, in test_select_multiple
self.assertEqual(len(select.all_selected_options), 2)
AssertionError: 1 != 2
}}}
I believe there might be a lot of places that would need to use wait over
assert ;)
--
Ticket URL: <https://code.djangoproject.com/ticket/30974#comment:1>
Django
Reporter: Johannes Hoppe | Owner: nobody
Type: Bug | Status: assigned
Component: Testing framework | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Johannes Hoppe):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/30974#comment:2>
Django
Reporter: Johannes Hoppe | Owner: nobody
Type: Bug | Status: assigned
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* stage: Unreviewed => Accepted
Comment:
Yep, thanks again. Very happy to review these :) (If we could get selenium
running reliably on more than just Chrome that would be a win...)
--
Ticket URL: <https://code.djangoproject.com/ticket/30974#comment:3>
Django
Reporter: Johannes Hoppe | Owner: Johannes
| Hoppe
Type: Bug | Status: assigned
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Johannes Hoppe):
* owner: nobody => Johannes Hoppe
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/30974#comment:4>
Django
Reporter: Johannes Hoppe | Owner: Johannes
| Hoppe
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: assigned => closed
* resolution: => duplicate
Comment:
So let's call this a duplicate of #29892, where this is being addressed.
--
Ticket URL: <https://code.djangoproject.com/ticket/30974#comment:5>