Proposal: Do not use global settings.py when running unit tests

[Siddhi]

So here is the situation:

I have a middleware installed that looks at the HTTP_HOST header and
stores the domain and subdomain into the request object. Now when
running unit tests, the tests fail because the HTTP_HOST header is not
passed into the test Client object. Even if I modify my own tests, the
tests from django.contrib.auth still fail because this value is not
passed through. As a workaround I've modified the middleware to use
some test values if the key is not found, which works for this
particular case, but this is a specific problem from a more general
problem and the general problem still remains.

I'm going to use the auth framework as an example below, but it could
be any contrib or third party app.

The first problem is that any middleware that expects a particular
input will cause unit tests from contrib apps to fail because that
particular input is not being passed in.

A second problem is that certain templates must exist before the auth
unit test will pass - even if those templates are not used (for
example if you only use the User model, or you have renamed the
templates to something else). This is reported in ticket #7611
(currently marked as wontfix) - http://code.djangoproject.com/ticket/7611

A third problem is that some of the unit tests specify their own url
conf for the unit test, so if you have customised the registration
template to do a reverse lookup of your own urls, then the test will
fail (it cannot find those urls because urlconf has been overridden
for the test). This was brought up on the developers group here -
http://groups.google.com/group/django-developers/browse_thread/thread/ac888ea8567ba466

I guess what I am coming around to saying is that there are various
ways to modify the project environment through settings.py (urlconf,
templates, middleware etc) that could cause unit test failures. This
seems broken. It should not be the case that a developer has to modify
all the tests from contrib/third party apps when a new middleware is
installed or a template is customised.

What is needed is a way for a test to specify its environment so that
it is run in the same environment irrespective of the project
environment specified in settings.py.

Currently Django allows a TestCase to specify its own fixtures and
urlconf. I propose to extend this to other settings that might cause
problems. Currently I can think of templates and middleware because
I've run into these two, but it could be that other settings are also
causing problems.

Instead of allowing overrides for each setting one by one, it might
make more sense to have a more general solution to this problem.

What I had in mind is as follows

First start with blank settings. i.e only the default settings in
django.conf.global_settings are applied.

Then allow each TestCase class to define a settings.py override file.
Something like

settings = 'django.contrib.auth.test.settings'

This is optional, only if the test requires some specific settings
(for eg. auth framework needs the session middleware). Before running
the test, the settings are overridden and then rolled back after the
test is over.

In the test settings, the testcase can setup anything that is
required. It may be a custom urlconf, or custom template directory
pointing to some test templates, any middleware that the test requires
etc.

This will be a more robust solution, rather than using the global
project settings.py when running the tests.

If the developer wants to test something with the global settings.py,
then they can just set their settings override to that file in their
own tests without worrying that some third party tests are going to
fail.

Does this make sense? Any feedback would be welcome.

[Calvin Spealman]

+1

--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

[Carl Meyer]

On Nov 3, 3:24 am, Siddhi <siddharta.li...@gmail.com> wrote:
> What is needed is a way for a test to specify its environment so that
> it is run in the same environment irrespective of the project
> environment specified in settings.py.

This doesn't fix the third-party/contrib apps problem, but "a way for
a test to specify its environment" is pretty trivial in practice, see
snippet 1011 for one implementation:

http://www.djangosnippets.org/snippets/1011/

Carl

[Siddhi]

Hmm, perhaps some of this code could be moved into the TestCase setup
and teardown to provide the functionality to all test classes? I was
looking at the auth tests again and there is code like this -
http://dpaste.com/88465/ - to get custom settings in for the test, so
this kind of functionality will be useful. Plus it removes the
brittleness if there is a change in global settings an unexpected
place, eg: middleware

--
Siddharta Govindaraj
http://siddhi.blogspot.com
http://www.silverstripesoftware.com/blog/