[Django] #25653: Provide a way to run only the selenium tests
Django
---------------------------------------------+------------------------
Reporter: timgraham | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
It would be nice if we had an option to run only the selenium tests in the
Django test suite. These tests are slow and don’t run by default (you must
specify `./runtests.py --selenium`). A common use case is to want to check
the selenium tests after a normal run of the test suite which skips them.
Currently you must rerun the entire test suite with selenium enabled
instead of running only the selenium tests that were skipped in the first
test run.
I think it would be okay if we changed `./runtests.py --selenium` to do
that. My argument is that there isn't a common need to run both the non-
selenium and selenium tests in one command which that option provides now.
Feel free to voice disagreement on this; for example, if you think we
should add a separate option for running only the selenium tests.
I'm not sure about the best implementation, so to whoever picks this up,
please share what you have in mind before coding.
--
Ticket URL: <https://code.djangoproject.com/ticket/25653>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: timgraham | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by auvipy):
+1
any guide for workaround?
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:1>
Django
Reporter: timgraham | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by akaariai):
I really, really think we want to add "test tags". Users could define tags
manually. Selenium tests could have tag "selenium" by default.
So, to run tests without selenium tests: `python manage.py test --exclude-
tags=selenium`. To run only selenium tests: `python manage.py test
--tags=selenium`.
Tags would have a lot of other uses, too. Say, you might want to tag a
subset of your tests as smoke tests, or you might want to tag those tests
that are slow (not only selenium tests are slow).
If you want to do an incremental test run, and you have tags fast, slow
and selenium.
{{{
python manage.py test --tags=fast # run fast tests first
python manage.py test --tags=slow --exclude-tags=selenium # run those
slow tests that aren't selenium
python manage.py test --tags=selenium # run selenium tests
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:2>
Django
Reporter: timgraham | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
I agree that tagging or marking tests is a really useful feature.
It seems worth noting that this would be another case where we would be
re-inventing an existing py.test (and probably nose?) feature in our test
runner.
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:3>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by mrbox):
* status: new => assigned
* owner: nobody => mrbox
* cc: jakub@… (added)
Comment:
I've prepared a proof of
concept(https://github.com/django/django/pull/5585) to this feature- what
do you think about it?
About re-inventing py.test or nose- well, in my opinion we're doing this
this partially this way or another and this is feature small enough to fit
into our codebase. It's also quite useful so shouldn't be that much of a
problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:4>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by carljm):
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
Comment:
Yes, I think you're right. The patch is small and useful, and I don't
think switching to nose or py.test is a good or feasible plan in the
medium-term anyway (posted more rationale for that just now in #25707).
PR just needs some cleanup, and there's a question about whether we want
an `--exclude-tags` flag as well. I think probably we'll end up wanting
it, but it could also always be added later if we do.
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:5>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
I've created a new ticket (#25735) specifically for the test-tagging
feature, since currently that's all the PR implements, and I think it's
best to separate the new feature from its use in Django's own test suite.
Even once we merge that feature, before this ticket can be closed we still
need to figure out how the use of the new tagging feature for Selenium
tests interacts with the existing `--selenium` option. Some options (all
of these options assume that we tag all Selenium tests with a "selenium"
tag):
1) Deprecate/remove `--selenium` and just use the tagging feature alone.
This means that you would need to specify `--exclude-tags selenium` to get
the equivalent of the current behavior. I don't like this option, because
I still don't think the Selenium tests should run by default in local
development: they are too slow and the constant windows-popping-up is too
invasive.
2) Leave `--selenium` as-is. This means that to run all tests you'd
provide `--selenium`, to run Selenium only you'd provide `--selenium
--tags selenium`. The latter seems a bit redundant, but perhaps OK?
3) Leave `--selenium` as-is, and make `--tags selenium` automatically
imply `--selenium`. So to run all tests you'd run `--selenium`, to run
only the Selenium tests you'd run `--tags selenium`.
There are probably other reasonable permutations I haven't considered.
Thoughts?
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:6>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Unless I missed something, the `--selenium` option is specific to
`runtests.py` so I don't think we need to worry about backwards
compatibility and can just remove it.
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:7>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
I'm not concerned about backwards-compatibility with the `--selenium`
option, I'm concerned about arriving at a good usable situation for
Django's internal tests, and (as I described above) I'm not sure that
"just remove `--selenium`" achieves that. I mentioned above why I don't
like the option of just removing `--selenium` -- because that would imply
that Selenium tests would be run by default, which I don't think is good.
Another option might be to get rid of `--selenium` and then have
`--exclude-tag=selenium` be the default, but it's not obvious how that
would work either. If you used `--exclude-tag=somethingelse` would the
Selenium tests suddenly start running as a side effect? That's not good.
So if you can't override the default that way, how can you ever run the
Selenium tests at all, since `--exclude-tag` overrides `--tag`?
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:8>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
I think I would remove `--selenium` and add some logic in `runtests.py`
that appends `selenium` to `--exclude-tag` except if `--tag=selenium` is
specified.
(Sorry, I didn't read your first comment closely.)
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:9>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
No problem. That's an option, but then how do you run "all the tests,
including Selenium" (which I presume is what CI wants to do)?
`--tag=selenium` means "run only the Selenium tests."
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:10>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
On CI we have a separate build for selenium and I'd like to run only those
tests there. If we ever moved to a testing service like Sauce Labs, I
think we'd want the same functionality.
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:11>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
Ah right! I forgot that CI does run Selenium separately. In that case I
think your proposal is fine. Personally for local dev I think I'd be quite
happy to either run the full test suite without Selenium, or just the
Selenium tests, but not have a simple way to run both together.
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:12>
Django
Reporter: timgraham | Owner: mrbox
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"6670da75ff8a59b2ec0b465846e3f76aab9155b2" 6670da75]:
{{{
#!CommitTicketReference repository=""
revision="6670da75ff8a59b2ec0b465846e3f76aab9155b2"
Fixed #25653 -- Made --selenium run only the selenium tests.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:13>
Django
Reporter: timgraham | Owner: mrbox
Type: New feature | Status: closed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"87994b40b3874e45666edfceb313e19611cd460e" 87994b40]:
{{{
#!CommitTicketReference repository=""
revision="87994b40b3874e45666edfceb313e19611cd460e"
Refs #25653 -- Corrected help text for runtests.py --selenium option.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25653#comment:14>