Making the test suite run faster

[Aymeric Augustin]

Hello,

As the test suite is growing, it’s getting slower. I’ve tried to make it faster by running tests in parallel.

The current state of my experiment is here: https://github.com/django/django/pull/4063

I’m distributing the tests to workers with the multiprocessing module. While the idea is simple, the unittest APIs make its implementation painful.

** Results **

Without the patch:

Ran 9016 tests in 350.610s
./runtests.py  355,86s user 20,48s system 92% cpu 6:48,23 total

With the patch

Ran 9016 tests in 125.778s
./runtests.py --parallel  512,31s user 29,92s system 300% cpu 3:00,73 total

Since it takes almost one minute to create databases, parallelization makes the execution of tests go from 6 minutes to 2 minutes.

This isn’t bad, but the x3 speedup is a bit disappointing given that I have 4 physical / 8 logical cores. Perhaps the IPC is expensive.

Does anyone have insights about scaling with multiprocessing?

** Limitations **

This technique works well with in-memory SQLite databases. Each process gets its own copy of the database in its memory space.

It fails with on-disk SQLite databases. SQLite can’t cope with this level of concurrency. It timeouts while attempting to lock the database.

It fails with PostgreSQL (and, I assume, other databases) because tests collide, for instance when they attempt to load the same fixture.

** Next steps **

At this point, the patch improves the common use case of running `./runtests.py` locally to check a database-independent change, and little else.

Do you think it would be useful to include it in Django anyway? Do you have concerns about the implementation? Charitably, I’ll say that “it works”…

Releasing it separately as a custom test runner may be more appropriate.

What do you think?

-- 

Aymeric.

[Michael Manfre]

Anything to make the test suite faster is a worthwhile effort. The speed up will be even better for those of us with slower dev systems. Getting the speed boost for in memory sqlite is a good start and Django is much more than an ORM. It'll take work to improve the database isolation for the test suite, but that is something that should probably happen regardless of parallelization.

I'm imagining database backends being able to control whether or not they support parallelization and if each process needs its own database.

Regards,

Michael Manfre

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/639C2955-7AAB-4BC6-940D-EA69F7F51280%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

[Russell Keith-Magee]

On Sat, Feb 7, 2015 at 12:05 AM, Aymeric Augustin <aymeric....@polytechnique.org> wrote:

Hello,

As the test suite is growing, it’s getting slower. I’ve tried to make it faster by running tests in parallel.

The current state of my experiment is here: https://github.com/django/django/pull/4063

I’m distributing the tests to workers with the multiprocessing module. While the idea is simple, the unittest APIs make its implementation painful.

** Results **

Without the patch:

Ran 9016 tests in 350.610s
./runtests.py  355,86s user 20,48s system 92% cpu 6:48,23 total

With the patch

Ran 9016 tests in 125.778s
./runtests.py --parallel  512,31s user 29,92s system 300% cpu 3:00,73 total

Since it takes almost one minute to create databases, parallelization makes the execution of tests go from 6 minutes to 2 minutes.

This isn’t bad, but the x3 speedup is a bit disappointing given that I have 4 physical / 8 logical cores. Perhaps the IPC is expensive.

Does anyone have insights about scaling with multiprocessing?

Are you resource locking on anything else? e.g., is disk access becoming the bottleneck? Even memory throughput could potentially be a bottleneck - are you hitting disk cache? 

 

** Limitations **

This technique works well with in-memory SQLite databases. Each process gets its own copy of the database in its memory space.

It fails with on-disk SQLite databases. SQLite can’t cope with this level of concurrency. It timeouts while attempting to lock the database.

It fails with PostgreSQL (and, I assume, other databases) because tests collide, for instance when they attempt to load the same fixture.

I've thought about (but never done anything) about this problem in the past - my thought for this problem was to use multiple test databases, so you have isolation. Yes this means you need to do more manual setup (createdb test_database_1; createdb test_database_2; etc), but it means you don't have any collision problems multiprocessing an on-disk database.

 

** Next steps **

At this point, the patch improves the common use case of running `./runtests.py` locally to check a database-independent change, and little else.

Do you think it would be useful to include it in Django anyway? Do you have concerns about the implementation? Charitably, I’ll say that “it works”…

It's definitely worth pursuing. Faster test suite == double plus good. Multiprocessing would seem to be an obvious approach, too. 

My only "concern" relates to end-of-test reporting - how are you reporting test success/failure? Do you get a single coherent test report at the end? Do you get progress reporting, or just "subprocess 1 has completed; 5 failures, 200 passes" at the end of a subprocess? My interest here isn't strictly about Django - it's about tooling, and integration of a parallelized test suite with IDEs, or tools like Cricket.

Releasing it separately as a custom test runner may be more appropriate.

 

For me, it depends on how much code we're talking about, and how invasive you need to be on the core APIs. If the multiprocessing bit is fairly minor (and I suspect it probably is), but you need to make bunch of invasive changes to the test infrastructure, then you might as well include it in Django's core as a utility. However, if the whole thing can stand alone with minimal (or no) internal modifications - or modifications that make sense in a general sense - then you might as well release as a standalone package.

Yours,

Russ Magee %-)

[Aymeric Augustin]

On 7 févr. 2015, at 01:02, Russell Keith-Magee <rus...@keith-magee.com> wrote:

I've thought about (but never done anything) about this problem in the past - my thought for this problem was to use multiple test databases, so you have isolation. Yes this means you need to do more manual setup (createdb test_database_1; createdb test_database_2; etc), but it means you don't have any collision problems multiprocessing an on-disk database.

The fastest way to do this is probably to create a database then clone it. Each backend would have to implement a duplication method:

- SQLite: os.copy(‘django_test.sqlite3’, ‘django_test_N.sqlite3’)

- PostgreSQL: CREATE DATABASE django_test_N WITH TEMPLATE django_test OWNER django_test;

- MySQL: mysqldump … | mysql …

- Oracle: apparently there’s a DUPLICATE command — I have a bad feeling about this one.

For optimal speed, this feature should support --keepdb.

My only "concern" relates to end-of-test reporting - how are you reporting test success/failure? Do you get a single coherent test report at the end? Do you get progress reporting, or just "subprocess 1 has completed; 5 failures, 200 passes" at the end of a subprocess? My interest here isn't strictly about Django - it's about tooling, and integration of a parallelized test suite with IDEs, or tools like Cricket.

Yes, I have a clean report. If you look at the pull request you’ll see two successive implementations:

1) Run tests in workers, pass “events" back to the master process, feed them to the master test runner. Unfortunately this technique is never going to be sufficiently robust because it involves passing tracebacks between processes and tracebacks aren’t pickleable in general.

2) Run tests in worker, buffer the output in workers, have the master test runner reconstruct proper output. This is less elegant. It only works with the TextTestRunner because it depends heavily on its internals. But it’s more robust and it suffices for running Django’s test suite.

What this really means is — you have to choose between being a proper unittest2 runner or being robust. That’s what I meant when I said the unittest2 APIs made the implementation painful.

-- 

Aymeric.

[Fabio Caritas Barrionuevo da Luz]

Parallel is different than concurrent, did you come to look at the package "testtools", I think it implements something similar to what you want to do


https://github.com/testing-cabal/testtools/blob/master/testtools/testsuite.py#L38-L198
http://stackoverflow.com/a/17059844/2975300

[Aymeric Augustin]

On 7 févr. 2015, at 15:09, Fabio Caritas Barrionuevo da Luz <bna...@gmail.com> wrote:

> Parallel is different than concurrent

Of course. I’m concerned with parallelism here because that’s what gives the performance improvement. Of course I have to make concurrency safe.

> did you come to look at the package "testtools", I think it implements something similar to what you want to do

Yes, I started by attempting to use it. Unfortunately it relies an threads. That can’t work for Django’s tests, primarily because of `override_settings`.

--
Aymeric.

[Aymeric Augustin]

**tl;dr** I can run the full test suite in 85 seconds on SQLite, a 4.8x speedup.

Hello,

Since I last wrote about this project, I improved parallelization by:

- reworking the IPC to avoid exchanging tracebacks

- implementing database duplication for SQLite, PostgreSQL and MySQL

The code is still rough. Several options of runtests.py don't work with

parallelization.

Initially performance was disappointing. I couldn’t max out my cores during

the whole run. With some basic monitoring I noticed that the CPU load

plummeted when there were spikes of disk writes. This led me to believe I was

disk I/O bound even with an in-memory database and a SSD.

So I started optimizing disk I/O, which means doing less I/O and doing it only

in a RAM-mounted temporary directory. Writing to RAM instead of writing to

disk helps a lot. It’s as simple as creating a RAMdisk (that depends on your

OS) and pointing the TMPDIR environment variable to the RAMdisk.

Unfortunately, i18n and migrations management commands write in the

application directories. I haven't found (yet) a way to point them to a

temporary directory instead.

My 2012 MacBook Pro with a 2.3 GHz Intel Core i7 (4 cores, 8 threads) takes:

- 30 seconds for creating the two databases

- 240 seconds to run the actual tests in a single process

- 72 seconds in 4 processes (3.3x faster)

- 60 seconds in 6 processes (4x faster)

- 55 seconds in 8 processes (4.4x faster)

That looks quite close to what my hardware can do. Hyperthreading doesn't help

as much as multiple cores when it comes to running multiple processes and the

synchronization costs increase with the number of processes.

Creating the database accounts for more than a third of the total runtime. I'm

not sure how much time is spent in the migrations framework and how much doing

the table creations. --keepdb helps but it requires an on-disk database. An

on-RAMdisk database would most likely be the best option. I haven't tried it

yet. It should work with at least SQLite and PostgreSQL (using tablespaces).

If you want to help, I'd be interested in:

- reports of whether parallelization works for test suites other than Django's

  own -- apply my pull request and run `django-admin test --parallel` or

  `django-admin test --parallel-num=N`

- a patch implementing database duplication on Oracle

Let me know if you have questions or concerns.

-- 

Aymeric.

[Aymeric Augustin]

Hello,

I polished my test parallelization patch a bit. I'd like to merge it before
1.9 alpha. I rewrote history heavily to make the changes easier to review:
https://github.com/django/django/pull/4761

Eventually I settled for the approach I dismissed in a previous email: run

tests in workers, pass "events" back to the master process, feed them to the

master test runner. I depend on the `tblib` package to make tracebacks
pickleable.

The patch is still missing support for Oracle, despite significant help I
received from Shai and others. I'm proposing to merge it anyway because I
don't want to delay it indefinitely. It's possible to implement the missing
methods on Oracle at any point in the future.

I used various techniques to make tests safe for concurrent execution on a
case by case basis. When possible, I moved all filesystem writes to a
temporary directory. When not possible, I serialized execution of conflicting
test cases by locking a file.

If you have concerns about adding this feature to Django or about the
implementation -- which isn't the most beautiful code I've written -- now is a
good time to bring them up. I’d like to merge the patch in one or two weeks.

Also it would be interesting to try parallelization on test suites other than
Django's own. Just apply my PR and run `django-admin test --parallel` or
`django-admin test --parallel-num=N`. (Unfortunately I suspect few projects
are compatible with Django's development version and have a large test suite.)

Thanks!

--
Aymeric.

[Andrew Godwin]

I'm on board merging this without Oracle support - if I understand right, it means we just have to run those non-parallel for now, so we're not losing anything, right?

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/F7BA7C0F-3214-42E2-AAD2-B90B62EC07E2%40polytechnique.org.

[Aymeric Augustin]

On 31 août 2015, at 06:12, Andrew Godwin <and...@aeracode.org> wrote:

> I'm on board merging this without Oracle support - if I understand right, it means we just have to run those non-parallel for now, so we're not losing anything, right?

Right.

--
Aymeric.

[Tino de Bruijn]

Hi Aymeric,

Really cool that you pushed this further. Out of curiosity I have two questions:

- What happens when two SerializeMixin tests try to lock the same file? Does one wait for the other (probably not), or is a lockfile exception raised?

- How does this work in combination with the --keepdb flag

Thanks,

Tino

--
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/72EC1F48-9CD7-496B-BFA0-017B9AA93B75%40polytechnique.org.

[Aymeric Augustin]

Hi Tino,

2015-08-31 8:35 GMT+02:00 Tino de Bruijn <tin...@gmail.com>:

- What happens when two SerializeMixin tests try to lock the same file? Does one wait for the other (probably not), or is a lockfile exception raised?

The second one waits for the first one to complete. This happens all the time because conflicting tests are often defined close to one another and the parallel test runner runs tests in order.

 

- How does this work in combination with the --keepdb flag

All clones of the test database are kept.

There's one edge case: if you run with --parallel-num=2 --keepdb and then --parallel-num=4 --keepdb, the test runner will crash because clones 3 and 4 won't exist. I don't think that's a big problem.

--

Aymeric.

[Tim Graham]

Aymeric, did you envision any changes to Django's CI setup? Currently we run 1 Jenkins executor per CPU, so I don't know that adding parallelization would have any benefit? (We are already using all 8 CPUs when we're running 8 concurrent builds from the matrix.)  If not, then I wonder how we can ensure that Django's test suite continues to work well in parallel.

[Aymeric Augustin]

On 31 août 2015, at 20:41, Tim Graham <timog...@gmail.com> wrote:

> Aymeric, did you envision any changes to Django's CI setup?

Glad you asked :-) I have some thoughts but no definitive opinion.

> Currently we run 1 Jenkins executor per CPU, so I don't know that adding parallelization would have any benefit? (We are already using all 8 CPUs when we're running 8 concurrent builds from the matrix.)

Indeed, we’re already parallelizing by running simultaneously tests against multiple databases. So we aren’t going to make order-of-magnitude gains.

The last processes of a given build may complete a bit faster if they’re run in parallel mode as they will make better use of the server’s resources once there are fewer active executors than CPUs. So the completion time for a matrix build may be slightly lower, unless the increased concurrency slows tests down sufficiently to negate this effect.

If increased concurrency is a problem, half the number of executors running tests with --parallel-num=2 should still complete faster, because it “packs” CPU usage more efficiently towards the end of the matrix build. It’s hard to tell what’s going to happen without trying.

> If not, then I wonder how we can ensure that Django's test suite continues to work well in parallel.

Perhaps we could make the --parallel option the default in runtests.py. You’d have to pass --parallel-num=1 to disable it. This would help occasional sprinters unfamiliar with the option. They could just run ./runtests.py, melt their laptops, and not even have time for a coffee break.

--
Aymeric.

[Aymeric Augustin]

> On 31 août 2015, at 20:59, Aymeric Augustin <aymeric....@polytechnique.org> wrote:
>
> Perhaps we could make the --parallel option the default in runtests.py. You’d have to pass --parallel-num=1 to disable it. This would help occasional sprinters unfamiliar with the option. They could just run ./runtests.py, melt their laptops, and not even have time for a coffee break.

I made this change in the latest version of the pull request. Barring unexpected problems, I’m going to merge it before the DjangoCon US sprints.

This will require ./runtests.py --no-parallel or ./runtests.py --parallel-num=1 to run tests under Oracle. I think it’s a good tradeoff because the defaults should be optimized for occasional contributors.

Tim, here’s my suggestion for the CI server:

- set --parallel-num=1 for builds of the master branch (django-master, django-master-trusty, django-oracle, django-selenium, master-trusty-reverse, master-ἥoἥascii-path, django-coverage, django-coverage-postgresql)
- set --parallel-num=2 for PR builds (pull-requests-trusty)

--
Aymeric.

[Shai Berger]

Hi,

On Sunday 06 September 2015 13:06:18 Aymeric Augustin wrote:
>
> This will require ./runtests.py --no-parallel or ./runtests.py
> --parallel-num=1 to run tests under Oracle. I think it’s a good tradeoff
> because the defaults should be optimized for occasional contributors.
>

Can we somehow make this default controlled by the database backend, so that
it only defaults to --parallel on backends which support it?

While not many, Oracle does have its own occasional contributors, and I'm not
sure this kind of change would be welcomed by the 3rd-party backends.

This could be done, I think, with a feature flag on the backend
("supports_parallel_tests"), defaulting to False, set to True on supporting
backends.

My 2 cents,
Shai.

[Michael Manfre]

I agree with Shai. The database backend needs to be able to control this feature.

Regards,

Michael Manfre

--

GPG Fingerprint: 74DE D158 BAD0 EDF8

keybase.io/manfre

[Aymeric Augustin]

Le 8 sept. 2015 à 18:42, Michael Manfre <mma...@gmail.com> a écrit :
>
> I agree with Shai. The database backend needs to be able to control this feature.

Yes, I'm implementing that. The feature will be opt-in for database backends.

--
Aymeric.

[Tim Graham]

Aymeric merged this on Thursday. Big thanks to him!

Now it's time to have a competition to build the machine that will run the Django test suite in the fastest time. :-)