[Django] #24522: Add --random option to test command to run test cases in random order
Django
-------------------------------+--------------------
Reporter: AnSavvides | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
Similar in spirit to [https://docs.djangoproject.com/en/dev/ref/django-
admin/#django-admin-option---reverse --reverse] introduced in 1.8, the
idea is to add a --random option that will randomise the test run order.
Grouping by test class would be preserved when using this option.
I think the idea behind --reverse is great, so I see something like
--random a way of going a step further to help with debugging test cases
that are not properly isolated.
Going a step further - which might be unnecessary, so this is just a minor
aside - we could even add an option that generates and runs X randomised
test runs in all possible orders given a set of tests. That way, people
who are really paranoid can confirm that their tests really pass and order
genuinely does not matter in any case.
--
Ticket URL: <https://code.djangoproject.com/ticket/24522>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: AnSavvides | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I think we would need some way of writing the order to disk so a
particular random order could be reproduced. I'm not sure it's worth it.
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:1>
Django
Reporter: AnSavvides | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: master
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => wontfix
Comment:
If you feel strongly about this, please raise it on the
DevelopersMailingList so we can get some more opinions.
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:2>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: master
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Dan Palmer):
For what it's worth, I don't think this needs writing to disk.
I propose that with `--random` Django generates a random seed and prints
it out at the beginning of the test run. This will likely be stored in
shell scrollback, or on disk/in a database if running tests in a CI
environment. From there, tests could be re-run with a given seed with
`--random-seed=XXXXX`.
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:3>
Django
Reporter: Andreas Savvides | Owner: nobody
Component: Testing framework | Version: master
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Ahmad A. Hussein):
* cc: Ahmad A. Hussein (added)
* type: Uncategorized => New feature
* component: Uncategorized => Testing framework
Comment:
I agree with Dan, we could use a random seed to accomplish this. This is
worth looking into to expand our test runner's functionality.
It is something already supported in pytest though.
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:4>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: New feature | Status: closed
Component: Testing framework | Version: master
Severity: Normal | Resolution: wontfix
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Adam Wróbel):
* cc: Adam Wróbel (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:5>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: New feature | Status: closed
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Chris Jerdonek):
* cc: Chris Jerdonek (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:6>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: New feature | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: wontfix
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
I just posted a message to the developers' list asking if this ticket
could be reopened: https://groups.google.com/g/django-
developers/c/68LZwTr2vpI/m/m3wlER5rAAAJ
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:7>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: New feature | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: wontfix
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Adam Johnson):
* cc: Adam Johnson (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:8>
Django
Reporter: Andreas Savvides | Owner: nobody
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Chris Jerdonek):
* status: closed => new
* has_patch: 0 => 1
* resolution: wontfix =>
Comment:
I'm reopening the ticket based on the mailing list discussion (a number of
people replied in support).
I also posted a PR here: https://github.com/django/django/pull/14137
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:9>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:10>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
By the way, while working on the PR, I realized that we could name the
option `--shuffle` instead of `--random`. I felt `--shuffle` was more
specific and self-descriptive, whereas `--random` can be used in broader
contexts (e.g. altering random data in tests themselves). So if we have
the opportunity to be more specific in naming, I thought we should take
it.
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:11>
Django
Reporter: Andreas Savvides | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam Johnson):
`--shuffle` sounds good to me.
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:12>
Django
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* owner: nobody => Chris Jerdonek
* needs_better_patch: 0 => 1
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:13>
Django
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Chris Jerdonek):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Adam Johnson):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:16>
Django
-------------------------------------+-------------------------------------
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Chris Jerdonek):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:17>
Django
-------------------------------------+-------------------------------------
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:18>
Django
-------------------------------------+-------------------------------------
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Component: Testing framework | Version: dev
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"90ba716bf060ee7fef79dc230b0b20644839069f" 90ba716b]:
{{{
#!CommitTicketReference repository=""
revision="90ba716bf060ee7fef79dc230b0b20644839069f"
Fixed #24522 -- Added a --shuffle option to DiscoverRunner.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:19>
Django
-------------------------------------+-------------------------------------
Reporter: Andreas Savvides | Owner: Chris
| Jerdonek
Type: New feature | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by GitHub <noreply@…>):
In [changeset:"f5dccbafb957841d0867f0b153d7f7123f0ec83d" f5dccbaf]:
{{{
#!CommitTicketReference repository=""
revision="f5dccbafb957841d0867f0b153d7f7123f0ec83d"
Refs #24522 -- Fixed code comment about seeds in Shuffler.__init__().
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24522#comment:20>