Django testing strategy

[Daniele Procida]

I have started writing my first tests, for a project that has become pretty large (several thousand lines of source code).

What needs the most testing - where most of the bugs or incorrect appear emerge - are the very complex interactions between objects in the system.

To me, the intuitive way of testing would be this:

* to set up all the objects, in effect creating a complete working database
* run all the tests on this database

That's pretty much the way I test things without automated tests: is the output of the system, running a huge database of objects, correct?

However, I keep reading that I should isolate all my tests. So I have had a go at creating tests that do that, but it can mean setting up a dozen objects sometimes for a single tiny test, then doing exactly the same thing with one small difference for another test.

Often I have to run save() on these objects, because otherwise tests that depend on many-to-many and other database relations won't work.

That seems very inefficient, to create a succession of complex and nearly-identical test conditions for dozens if not hundreds of tests.

I'd appreciate any advice.

Daniele

[Thomas Orozco]

You can use `fixtures` for this purpose! You can have several of them to have exactly the data you need for a test.

Have a look there: https://docs.djangoproject.com/en/dev/topics/testing/#django.test.TestCase.fixtures

Cheers,

Thomas 

2012/10/4 Daniele Procida <dan...@vurt.org>

Daniele

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Evan Brumley]

django-dynamic-fixture can also help a lot in this situation:
http://paulocheque.github.com/django-dynamic-fixture/

Certainly beats having to futz around with fixtures.

[Daniele Procida]

On Thu, Oct 4, 2012, Evan Brumley <voight...@gmail.com> wrote:

>django-dynamic-fixture can also help a lot in this situation:
>http://paulocheque.github.com/django-dynamic-fixture/
>
>Certainly beats having to futz around with fixtures.

Thanks - there seem to be a lot of tools to generate test data in various ways, such as <https://github.com/dnerdy/factory_boy>.

However I am still puzzled by the question at a slightly higher level, about strategy - is it better to create a large collection of data once, and to test the behviour of the system against the various combinations of data in it, or to create a lengthy succession of smaller collections of data,containing only the material needed for the particular combination being tested each time?

Daniele

[Amirouche]

On Thursday, October 4, 2012 7:49:19 PM UTC+2, Daniele Procida wrote:

I have started writing my first tests, for a project that has become pretty large (several thousand lines of source code).

That is too, late! ;-)
 

What needs the most testing - where most of the bugs or incorrect appear emerge - are the very complex interactions between objects in the system.

To me, the intuitive way of testing would be this:

* to set up all the objects, in effect creating a complete working database
* run all the tests on this database

This method might work even if maintaining the fixture can be extra trouble. I never used fixture for doing tests, the useful case might not have presented itself but we had another way of doing it.
 

That's pretty much the way I test things without automated tests: is the output of the system, running a huge database of objects, correct?

Yes, but UT, should also be some kind of a documentation of how the projects works, having a pre-built database doesn't give much information about how to setup a correct database for the project whereas fine grained UT do.

 

However, I keep reading that I should isolate all my tests.

Yes, but there might be cases where tests fixtures are useful, I don't know them.

So I have had a go at creating tests that do that, but it can mean setting up a dozen objects sometimes for a single tiny test, then doing exactly the same thing with one small difference for another test.

Use factories for that. there is factory boy that seems to do the job, but I don't know its value, we were using our own factory and/or build the database manually, like I said earlier documenting/testing the way the database should look like  in the tests is IMO significant. You can also test the factory which might serve as documentation/tests of the proper building of the database something like, but it can be OT of UT if the factory is not used in the project code:

def test_build_base_site(self):
    Factory.build(Site)
    self.assertEquals(Site.objects.count(), 42)

def test_build_section(self):
    Factory.build(Section, name='Test Section')
    self.assertEquals(Section.objects.all(), 1)

    self.assertEquals(Section.objects.all(), 1)
def test_build_article(self):
    section = Factory.build('Article')
    self.assertEquals(Article.objects.all(), 42)

# etc...

The factory takes some arguments, builds the object you asked for
I don't know how complex is your schema, the  factory can be recursive so it's not problem.
 

 

[Amirouche]

Please ignore this message I've hit «Post» by mistake I will edit my post :)
 

[Amirouche]

On Thursday, October 4, 2012 7:49:19 PM UTC+2, Daniele Procida wrote:

I have started writing my first tests, for a project that has become pretty large (several thousand lines of source code).

That is too, late! ;-)
 

What needs the most testing - where most of the bugs or incorrect appear emerge - are the very complex interactions between objects in the system.

To me, the intuitive way of testing would be this:

* to set up all the objects, in effect creating a complete working database
* run all the tests on this database

This method might work even if maintaining the fixture can be extra trouble. I never used fixture for doing tests, the useful case might not have presented itself but we had another way of doing it.
 

That's pretty much the way I test things without automated tests: is the output of the system, running a huge database of objects, correct?

Yes, but UT, should also be some kind of a documentation of how the projects works, having a pre-built database doesn't give much information about how to setup a correct database for the project whereas fine grained UT do.

 

However, I keep reading that I should isolate all my tests.

Yes, but there might be cases where tests fixtures are useful, I don't know them.

So I have had a go at creating tests that do that, but it can mean setting up a dozen objects sometimes for a single tiny test, then doing exactly the same thing with one small difference for another test.

Use factories for that. there is factory boy that seems to do the job, but I don't know its value, we were using our own factory and/or build the database manually, like I said earlier documenting/testing the way the database should look like  in the tests is IMO significant. You can also test the factory which might serve as documentation/tests of the proper building of the database something like:

def test_build_base_site(self):
    Factory.build(Site, domain='foo', name='bar')
    self.assertEquals(Site.objects.count(), 1)
    site = Site.objects.all()[0]
    # test domain name and name

def test_build_section(self):
    Factory.build(Section, name='Test Section')  # other fields are generated ! You cannot test them outsite factory UT of course...
    self.assertEquals(Section.objects.all(), 1)
    section = Section.objects.all()[0]
    self.assertNotNone(section.site)

def test_build_article(self):
    section = Factory.build('Article')

    self.assertEquals(Article.objects.count(), 1)
    article = Article.objects.all()[0]
    self.assertNotNone(article.section)
    self.assertNotNone(article.section.site)

# etc...

The factory takes some arguments to populate the fields of the objects you asked for (you probably can use magic parameters to populate related fields too), builds the object you asked for.

I don't know how complex is your schema, the  factory can be recursive so it's not problem.
 
 

Often I have to run save() on these objects, because otherwise tests that depend on many-to-many and other database relations won't work.

I don't understand what you mean.
 

That seems very inefficient, to create a succession of complex and nearly-identical test conditions for dozens if not hundreds of tests.

Could you give an example of nearly identifical tests ?
 

I'd appreciate any advice.

I don't know how does your project setup looks like but here is the one I worked with:

For each feature (or group of features) we split it in two apps, a generic app and an integration/project app. The generic app can be re-used in other projects, whereas the integration app does the specific templates, model, signal hooking specific to the current project. Then the tests follows, in the generic apps you tests generics things (that don't need a particular project setup...) and in the integration/project app you tests everything else. Client side testing will go in integration app testing because you will most likely override templates in it and if you do client side testing in the generic app they will fail because of the overrides.

Gotchas:
- Django application template create a tests.py in the application directory, remove it and create a directory instead (or use another application template), you will most likely need several files to do the testing (factory, models, feature1, feature2) and mixing it with the application files is ugly (you can also have a tests.py with a loc > 1000). Don't forget to create tests/__init__.py and import every test classes you create so that they can be found by Django test runner
- If the generic apps needs some models, they should be defined in the tests/__init__.py (If I remember well)

- If you have A LOT of tests, or simply want to make the life of the developpers easier look for fsync=off (for PostgresSQL) or similar option in you database of choice or (prefered if possible) use in memory sqlite.

I'm not sure but if you use fixtures and they are created between all the tests (classes) the test run speed may increase quiet a bit making it quiet -- aweful -- to test the project hence developing it, that's also one of the reasons why generic apps are a good idea because they can be developped and properly tested separatly, can improve your productivity and  you can open-source the generic app ;).


Amirouche

[Amirouche]

The problem you raising, is I think the problematic of unit-tests which tests a specific feature of an application versus integration aka. several-applications-wide tests which tests that a user workflow works properly like I put create a model, celery process it, then I (or another user) change the model, then I send an email, I get an hardbounce etc...). Anyway, like I said in the other mail, I do think that manual is better.

HTH,

Amirouche

[Dan Gentry]

I've been using factory-boy as of late, and have found it to be a great way to setup each test exactly as I need it.  Fixtures aren't bad either, but working with a large amount of data can make it difficult to predict the proper output for a test, and changes to this data to accommodate a new test situation could affect others. 

You could setup a group of test records in the setup method of a test class to be used for several tests.

Keep your tests focused, and you'll find you won't need a huge batch of data to get good coverage.

 

[Lachlan Musicman]

This has been a very interesting thread - out of interest, does anyone
have a preference for one of factory-boy or django-dynamic-fixture and
why?

They look similarly up to date and useful, but I've no idea how to
differentiate them.

cheers
L.

--
...we look at the present day through a rear-view mirror. This is
something Marshall McLuhan said back in the Sixties, when the world
was in the grip of authentic-seeming future narratives. He said, “We
look at the present through a rear-view mirror. We march backwards
into the future.”

http://www.warrenellis.com/?p=14314

[Daniele Procida]

On Thu, Oct 4, 2012, Daniele Procida <dan...@vurt.org> wrote:

>I have started writing my first tests, for a project that has become
>pretty large (several thousand lines of source code).

>I'd appreciate any advice.

Many thanks for the advice and suggestions.

This is what I have produced so far: <https://github.com/evildmp/Arkestra/blob/develop/contacts_and_people/tests.py>

At the moment, all the tests (they are testing methods on a particular set of models) are in a single class and even a single test function. I want to break them up, to make them more manageable.

What would be a good way to do that? I will need the objects I set up available in lots of different test functions, and probably different test classes too.

For example, I would like to write some tests for views, and to test based on the objects I have created so far.

Daniele