Splitting tests instead of testcases by processors when parallelizing tests

[Alan Evangelista]

Django only offers the possibility of parallelizing tests by splitting TestCases among processes. As the number of tests per TestCase class can be very different, it seems that splitting individual tests among processes would make more sense. I have replaced partition_suite_by_case() in test/runner.py with a new method that partitions the tests in the test suite evenly among the processes and at first sight it works well. Is there any known downfall in doing so? 

Thanks in advance!

Regards,

Alan

[charettes]

The main drawback will be that won't be able to take advantage of setUpTestData to perform costly data creation _once_ for all the tests attached to the same TestCase.

Splitting the suite per test will likely force setUpTestData to be called once per process that has at least once one test of the TestCase assigned to it.

A better strategy might be to avoid splitting TestCases among processes but order the suite of TestCase by the number of tests attached to them to _larger_ TestCases are run first just like you would normally pack items in decreasing size to optimize for reduced aggregate space.

If you have metrics around the average cumulative duration of each TestCase (e.g. computed from previous CI runs) it should be an even better heuristic than len(TestCase.tests).

Cheers,

Simon

[Adam Johnson]

Hi Alan

I'd repeat what Simon says.

To write fast tests, you'll want to adopt per-testcase setup, which is shared between tests within that test case. Django has two mechanisms for this in TestCase: fixtures and setUpTestData. (I gave a talk about setUpTestData last year: https://www.youtube.com/watch?v=_8qLxaWMdzE , and wrote a blog post: https://adamj.eu/tech/2021/04/12/how-to-convert-a-testcase-from-setup-to-setuptestdata/ )

If you split testcases across multiple processes, such per-testcase setup needs repeating in each process. So it can add a lot more work to your test suite, and may even make it slower overall. It's for this reason I recommend those using pytest-xdist use the --dist=loadscope option, which copies what Django does.

Simon's suggestion of ordering test cases by size could indeed help runtime. But I would normally prefer to use the --shuffle option (or pytest-randomly) to randomize test order, in order to discover test isolation bugs.

I would just recommend trying to keep test cases within a reasonable size bound. If you benchmark and find one to be particularly large, try speeding it up (setUpTestData, avoid TransactionTestCase, etc.) or split it.

Thanks,

Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5709f600-25ce-4e0a-af34-0e453f67072bn%40googlegroups.com.