runtest.py slowdowns?
Joseph Kocherhans
long it takes to run the django test suite? I used to get times of
about 14 seconds around March (at PyCon) but it's taking ~150 seconds
now. It's quite possibly something that's changed in my setup, but I
figured I'd see if anyone else has noticed the same thing. It doesn't
seem like there have been *that* many tests added to django in the
last couple of months.
Joseph
Russell Keith-Magee
Yes Virginia, there has been a test slowdown :-)
It happened around mid January, as a result of the tests that use
fixtures. Every time a fixture test is executed, it flushes the
contents of the database. Flushing is a slow operation. Unfortunately,
it's also a necessary operation for those tests.
I agree that the slowdown is less that ideal. I have a chat with
Malcolm recently about possible ways to speed things up, and we didn't
come up with any ideas that would yield any sort of significant
speedup. Any suggestions on how to improve the situation are welcome.
Yours,
Russ Magee %-)
Joseph Kocherhans
>
> On 5/15/07, Joseph Kocherhans <jkoch...@gmail.com> wrote:
> >
> > Has anyone noticed a fairly recent order of magnitude slowdown in how
> > long it takes to run the django test suite? I used to get times of
> > about 14 seconds around March (at PyCon) but it's taking ~150 seconds
> > now. It's quite possibly something that's changed in my setup, but I
> > figured I'd see if anyone else has noticed the same thing. It doesn't
> > seem like there have been *that* many tests added to django in the
> > last couple of months.
>
> Yes Virginia, there has been a test slowdown :-)
>
> It happened around mid January, as a result of the tests that use
> fixtures. Every time a fixture test is executed, it flushes the
> contents of the database. Flushing is a slow operation. Unfortunately,
> it's also a necessary operation for those tests.
Hmm... I guess I'm just remembering the times before I merged changes
from trunk into newforms-admin.
> I agree that the slowdown is less that ideal. I have a chat with
> Malcolm recently about possible ways to speed things up, and we didn't
> come up with any ideas that would yield any sort of significant
> speedup. Any suggestions on how to improve the situation are welcome.
Such is the price of testing :) I just wanted to make sure I wasn't
the only one seeing slowdowns. I'm surprised I didn't notice it
before. Unfortuantely I haven't really read the fixtures code, so I
definitely don't have any suggestions forthcoming. I've just treated
it as fabulous magic until I have time/a reason to look.
Joseph
Ivan Sagalaev
> It happened around mid January, as a result of the tests that use
> fixtures. Every time a fixture test is executed, it flushes the
> contents of the database. Flushing is a slow operation. Unfortunately,
> it's also a necessary operation for those tests.
>
> I agree that the slowdown is less that ideal. I have a chat with
> Malcolm recently about possible ways to speed things up, and we didn't
> come up with any ideas that would yield any sort of significant
> speedup. Any suggestions on how to improve the situation are welcome.
A wild suggestion: may be just drop an old database and create a new
one? Or only create new databases during tests and drop all of them at
once after?
Sorry if it already works like this, I'm not familiar very much with the
tests code and couldn't figure out where to look :-)
Malcolm Tredinnick
> On 5/15/07, Joseph Kocherhans <jkoch...@gmail.com> wrote:
> >
> > Has anyone noticed a fairly recent order of magnitude slowdown in how
> > long it takes to run the django test suite? I used to get times of
> > about 14 seconds around March (at PyCon) but it's taking ~150 seconds
> > now. It's quite possibly something that's changed in my setup, but I
> > figured I'd see if anyone else has noticed the same thing. It doesn't
> > seem like there have been *that* many tests added to django in the
> > last couple of months.
>
> Yes Virginia, there has been a test slowdown :-)
>
> It happened around mid January, as a result of the tests that use
> fixtures. Every time a fixture test is executed, it flushes the
> contents of the database. Flushing is a slow operation. Unfortunately,
> it's also a necessary operation for those tests.
Another data point: there's been almost a 100% slowdown -- from ~24s
with SQLite client to 45-50s now on my reasonably grunty desktop machine
-- with a bunch of the test client changes recently. I started seeing
gradual increases in the runtime around the time you changed login, so
in the last few weeks and it's gone up a bit more lately. I don't know
what the actual cause is here, just using those "moments" as a way to
place the changes on a timeline.
I'm absolutely not pointing fingers or criticising here, because the
functionality is very useful. I can usually write regression tests or
proof-of-concept tests for almost anything I need pretty easily these
days (I've been living inside the test suite a lot lately).
PostgreSQL and MySQL backend tests haven't blown out nearly as much,
which isn't too surprising since disk access and server overhead is much
greater there, but they've increased slightly.
Regards,
Malcolm
Russell Keith-Magee
>
> On Tue, 2007-05-15 at 12:29 +0800, Russell Keith-Magee wrote:
> > >
> > Yes Virginia, there has been a test slowdown :-)
> >
>
> with SQLite client to 45-50s now on my reasonably grunty desktop machine
> -- with a bunch of the test client changes recently. I started seeing
> gradual increases in the runtime around the time you changed login, so
> in the last few weeks and it's gone up a bit more lately. I don't know
> what the actual cause is here, just using those "moments" as a way to
> place the changes on a timeline.
Any recent slowdown will probably be due to the recent increase in the
number of test client tests and test assertion tests. Each of these
tests flushes the database.
Yours
Russ Magee %-)
Russell Keith-Magee
>
> Russell Keith-Magee wrote:
> > It happened around mid January, as a result of the tests that use
> > fixtures. Every time a fixture test is executed, it flushes the
> > contents of the database. Flushing is a slow operation. Unfortunately,
> > it's also a necessary operation for those tests.
> >
> > I agree that the slowdown is less that ideal. I have a chat with
> > Malcolm recently about possible ways to speed things up, and we didn't
> > come up with any ideas that would yield any sort of significant
> > speedup. Any suggestions on how to improve the situation are welcome.
>
> A wild suggestion: may be just drop an old database and create a new
> one? Or only create new databases during tests and drop all of them at
> once after?
It turns out to be slower. Flush only needs to delete data; if you
drop the database entirely, you need to destroy the old database,
recreate the database, and resync the database, too. Each of these is
a fairly expensive operation.
One thought Malcolm and I shared was that it _might_ be possible to
save the state of a 'synced' database, and restore that saved state at
the start of each test. My gut reaction says that this should be
faster, but its not an easy thing to implement. On file-backed SQLite
it would be a fairly simple file copy, but on any other backend it
becomes a lot more complex. Again, suggestions welcome.
Yours,
Russ Magee %-)