Testing: Fixtures vs Factories vs ???

[Paolo]

Hi folks,

I've been coding away on my latest Django project, and have recently started to think about how best to test the functionality I have so far.

Creating fixtures for my tests seems like a good idea initially, but from experience I find fixtures become harder and harder to maintain as the project ages and expands.

The literature appears to recommend fixtures as the way forward, but I also see there are projects such as 'factory boy' (inspired by the Rails world) offering an alternative (or complimentary?) method.  

Are there any recommendations/comments/examples from the Django community on what the best way to go is with regards to data for testing? 

Cheers,

Paul

[Shawn Milochik]

I don't think there's one true way. In other words, the answer is "all
of the above," depending on your project and the needs of each
individual test.

It also depends on your code. If you've done TDD, and therefore made
your code easier to test, you can probably do it the simplest way
possible. If you're trying to add coverage after the code is in place,
there's often a large amount of "setup" needed due to model
relationships, and it's easier to just dump everything into fixtures and
write tests to your known data.

There's also a third option, which is to have your tests (or the setUp)
dynamically create data for use by the tests. This is really handy for
things that are easy to test, but can be too much depending on the
design of the code overall.

Shawn

[Alvaro]

Hello,

You can try django-pyxtures [1] to generate data for running tests.
It's simply and clear. It also has an option to autogenerate data for
your models.

[1] https://github.com/igav/django-pyxtures

[Steve Bywater]

I work on a complex django project -- our test suite is over 1000
tests. We've begun converting them from fixtures to factories. Here's
why:

- Factories were measurably faster. At least in my local environment,
which may be more IO bound than others, but I was looking at the
results of tests and seeing tests run up to ten times faster using
factories. For example, I'd run a class including 10 tests that used
three fixtures and a total of maybe 30 instances a few times, and get
an average run time of about say 30 seconds. Then I would convert the
same test to use data from factories instead, and the same class would
run in under 4 seconds. The same speed gain remained in our jenkins
environment. My results may not be typical.

- At least in some cases, data loaded in factories doesn't get
unloaded. I don't know if that was by design or not but it caught me
by surprise. So if you have a test that loads a fixture with three
users, and assert that there are exactly three users, this test will
fail if another test happens to load a fixture that has some other
user in it has already run. A nightmare to debug when a test in an
unrelated part of the test suite starts failing after an innocuous
change, as running this test by itself will pass every time. Whereas
data your test creates with a factory will be blown out before the
next test runs.

We still use fixtures in some cases: static data that lots of tests
use and none of them change.

Best,
- Steve