request to reopen ticket #24522: --random test option

[Chris Jerdonek]

I'd like to ask folks if they would support reopening ticket #24522, which is to add a --random option to Django's test runner:
https://code.djangoproject.com/ticket/24522

It was opened 6 years ago and closed as won't fix.

I've been doing some simplifications of the test-suite related code in Django's DiscoverRunner, and I think a --random option could be added pretty simply. It seems like every project could benefit by running their tests in a random order. It would help to unearth hidden dependencies between tests.

If I were to implement this, the --random option would accept an optional integer "seed" argument. If --random is passed with no seed argument, the seed would be generated randomly by choosing a random integer between 0 and some large integer. Whether or not an explicit seed value is passed, Django can log the seed value being used before running tests (if --random was passed), and DiscoverRunner can store the seed as an attribute like self.random_order_seed. That would give callers the option of doing something with the seed programmatically, if logging to stderr isn't enough.

When --random is used, the order of classes can be randomized within their groups:
https://docs.djangoproject.com/en/3.1/topics/testing/overview/#order-in-which-tests-are-executed
and the order of methods within their class can also be randomized.

The seed can be used by instantiating a Python random.Random() object with it:
https://docs.python.org/3/library/random.html#random.Random
and the random orders could be generated by calling methods of that now-deterministic object.

--Chris

[Adam Johnson]

This would be something I'd really like to see. Also it would be nice to make it the default.

I suggest we add the flag as "--random" taking an integer that is the seed, or 0 to disable.

Do check out my project pytest-randomly for a battle-tested project randomly shuffling tests.

One thing pytest-randomly also does is it resets the random seeds in some well-known libraries, for example factory boy. I don't think that's something we'd want in Django since it's a framework to be built upon. Instead we could just make sure it would be possible to override a DiscoverRunner method in order to use the seed value to reset other libraries' randomization.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAOTb1wd7YnjmLwS_Gvvj3iGOx_Oodp7Xv%2BwZb_tCJ2Qmy%2BHO5g%40mail.gmail.com.

--

Adam

[Mariusz Felisiak]

Hi,

     I agree that it would be a nice new feature.

I suggest we add the flag as "--random" taking an integer that is the seed, or 0 to disable.

This kind of APIs are always confusing. Moreover I don't think that changing the current behavior is a good idea, it should be opt-in not opt-out.

Best,
Mariusz

[Fran Hrženjak]

On 10.03.2021., at 13:06, Mariusz Felisiak <felisiak...@gmail.com> wrote:

I suggest we add the flag as "--random" taking an integer that is the seed, or 0 to disable.

This kind of APIs are always confusing. Moreover I don't think that changing the current behavior is a good idea, it should be opt-in not opt-out.

I usually agree with new features being opt-in, but perhaps this case is different?

If I had tests that are breaking if executed randomly, I’d want to know about it yesterday. IOW, I’m having difficulty imagining a scenario where the user would be thankful for not activating this feature by default. So personally, I’d like to see an opt-out setting for this in settings.py.

/$0.02 

Fran

[Mariusz Felisiak]

I usually agree with new features being opt-in, but perhaps this case is different?

If I had tests that are breaking if executed randomly, I’d want to know about it yesterday. IOW, I’m having difficulty imagining a scenario where the user would be thankful for not activating this feature by default. So personally, I’d like to see an opt-out setting for this in settings.py.

/$0.02 

Fran

`--reverse`  will catch 95% of test isolation issues for you. It's highly more likely that running tests in reverse order will catch isolation issue for you than running them in a non-deterministic order.

[chris.j...@gmail.com]

For those that would rather it be opt-out, remember that we can always make it opt-out later, once we have more experience with the feature.

Regarding the suggestion to have `--random 0` mean not to use randomizing, I also think that could be confusing. 0 is also a valid integer seed, and it seems nice to have the option of passing 0 as a simple seed if one wants. We could use --no-random for disabling if we ever wanted to make the feature opt-out in the future.

--Chris

[Florian Apolloner]

Is the current order defined in any way -- if not one should consider it randoms anyways and as such actually randomizing should not make much of a difference. FWIW I am fine with making it the default and outputting the seed at the end of the run so it can reproduced. I am curious if it finds many remaining issues.

[Florian Apolloner]

Btw while I am fine with making it default (I really do not care much either way), we certainly need a way to disable it and keep the current algorithm (even though it might not be guaranteed stable). When I develop I often run one suite and fix the errors as they occur. Having a stable order there ensures that the tests I fixed before are still fixed when the next test fails (I quite often also run with failfast there). I think having a stable test output can be useful for development. Having randomization in CI might be interesting, but then we probably need a way to derive the salt from the PR number or so, because I think the number of tests inside a PR is small enough that you probably do not see isolation failures, but you really want to see progress in a PR without random failures due to isolation.

[Chris Jerdonek]

On Friday, March 12, 2021 at 10:47:56 AM UTC+1 Florian Apolloner wrote:

Is the current order defined in any way -- if not one should consider it randoms anyways and as such actually randomizing should not make much of a difference. FWIW I am fine with making it the default and outputting the seed at the end of the run so it can reproduced. I am curious if it finds many remaining issues.

I would characterize it as "mostly" sorted -- I think. I believe unittest's test discovery is used by default, in conjunction with a couple large test groups that Django configures (`DiscoverRunner.reorder_by = (TestCase, SimpleTestCase)`).

The unittest package's test discovery is documented as first sorting by file path:

Paths are sorted before being imported so that execution order is the same even if the underlying file system’s ordering is not dependent on file name.

https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover

And within test cases, test methods are sorted by name. For example, see--

https://docs.python.org/3/library/unittest.html#unittest.TestLoader.getTestCaseNames

and--

Note: The order in which the various tests will be run is determined by sorting the test method names with respect to the built-in ordering for strings.

https://docs.python.org/3/library/unittest.html#organizing-test-code

I don't know about classes within modules.

On Fri, Mar 12, 2021 at 1:53 AM Florian Apolloner <f.apo...@gmail.com> wrote:

... Having randomization in CI might be interesting, but then we probably need a way to derive the salt from the PR number or so, because I think the number of tests inside a PR is small enough that you probably do not see isolation failures, but you really want to see progress in a PR without random failures due to isolation.

The question of how to use seeds in conjunction with CI if randomization were the default is an interesting one (e.g. how to tell CI to repeat a particular seed). Certainly if randomization were the default, CI could still be configured to be deterministic by passing --no-random / --sorted or whatever the opt-out option would be called. But, regardless of the default, to get the full benefit I think you'd want at least one of the test runs in CI to be a random one.

--Chris

[Florian Apolloner]

Thank you for diving into how unittest discovery tests!

On Friday, March 12, 2021 at 12:14:05 PM UTC+1 chris.j...@gmail.com wrote:

But, regardless of the default, to get the full benefit I think you'd want at least one of the test runs in CI to be a random one.

This is were I somewhat disagree. We have to run it as often with as many different seeds as possible to have an effect. The space of tests is so big that I am not sure we'd find something otherwise (though I have to agree that is theoretical thinking I did not try in practice).

Cheers,

Florian

[Adam Johnson]

because I think the number of tests inside a PR is small enough that you probably do not see isolation failures, but you really want to see progress in a PR without random failures due to isolation.

I think it's the opposite - most isolation problems are best detected at the time they're introduced. So using a different seed for each run of a PR as it's updated will help flush out any newly authored isolation problems before merging.

It's highly more likely that running tests in reverse order will catch isolation issue for you than running them in a non-deterministic order.

There's actually a study on this, across Apache projects - see the link in https://github.com/adamchainz/pytest-reverse#history . They found that random is best, with reverse being almost as good.

Reverse order fails to find three-way-plus isolation failures, e.g. in tests A, B, C where B depends on a side-effect triggered in both A and C. Both the ABC and CBA orders allow the side effect to leak into test B so the failure is not detected.

We have to run it as often with as many different seeds as possible to have an effect. The space of tests is so big that I am not sure we'd find something otherwise (though I have to agree that is theoretical thinking I did not try in practice).

Only two randomly sorted test runs are needed to flush out most pairwise problems. For example, imagine test B depends on a side effect of test A. Each random test run has a 50% order of running A then B, or B then A. So the expected number of test runs to see the isolation failure is 2. Yes, one could see 32 test runs in a row without the failure, but the chance of that is 1 in 2^32.

For three-way isolation problems, you might expect more runs, depending on how many of the 6 possible configurations of 3 tests reveal the problem. For four-way it could be worse again. But at least such isolation problems will appear eventually, as opposed to never.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4034f3db-e6ee-4d78-b140-553c73c1a170n%40googlegroups.com.

--

Adam