Testing for various combinations of settings in settings.py

[Tim Chase]

For a project I'm building, I've got a number of knobs that can
be twiddled in the settings.py (well, in a local_settings.py, but
they get slurped into the settings.py, but it would also be
helpful for testing coverage against multiple DB backends) that
control the behavior of a deployed project. Some of these effect
how models are built and thus how the resulting database will be
defined.

Is there a sensible way to automate testing coverage of the
available permutations of options here? If possible, I'd like to
prevent the need to have N-factorial local_settings.py files,
each with its own custom permutation of the available settings,
and then script testing across each permutation. Many settings
are independent of each other, but do effect the outcome of my
tests (and some tests are unreachable unless you make changes to
the settings).

Thanks,

-tim

[Amit Ramon]

What you can do is read the settings you need to tweak from environment variables. For example, you can put in your settings.py file the following code:

import os
DATABASE_NAME = os.environ["DATABASE_NAME"]
DATABASE_USER = os.environ["DATABASE_USER"]
DATABASE_PASSWORD = os.environ["DATABASE_PASSWORD"]

Then for each different environment all you have to do is to set these variables. For example, for the local test server, I use a shell script that loads the environment variables and then starts the server. You can also set variables in apache2 configuration if you use mod_python, or in a script for fcgi. If you need more information about this, just ask.

--- Amit

* Tim Chase <django...@tim.thechases.com> [2008-03-20 05:35 -0500]:

[Russell Keith-Magee]

On Thu, Mar 20, 2008 at 7:35 PM, Tim Chase
<django...@tim.thechases.com> wrote:
>
> Is there a sensible way to automate testing coverage of the
> available permutations of options here? If possible, I'd like to
> prevent the need to have N-factorial local_settings.py files,
> each with its own custom permutation of the available settings,
> and then script testing across each permutation. Many settings
> are independent of each other, but do effect the outcome of my
> tests (and some tests are unreachable unless you make changes to
> the settings).

This could be a candidate for a custom test runner. The current test
runner build and executes the test suite with the settings provided at
the command line (either with --settings or DJANGO_SETTINGS_FILE);
however, it shouldn't be too hard to set up a loop around the test
suite execution that runs the suite many times with local settings
modifications (i.e., just overriding settings.XYZ = 'test setting' in
the test runner).

Documentation on setting up custom test runners is here:

http://www.djangoproject.com/documentation/testing/#defining-a-test-runner

Yours,
Russ Magee %-)

[Tim Chase]

>> Is there a sensible way to automate testing coverage of the
>> available permutations of options here? If possible, I'd like to
>> prevent the need to have N-factorial local_settings.py files,
>> each with its own custom permutation of the available settings,
>> and then script testing across each permutation. Many settings
>> are independent of each other, but do effect the outcome of my
>> tests (and some tests are unreachable unless you make changes to
>> the settings).
>
> This could be a candidate for a custom test runner.

[snip]

> Documentation on setting up custom test runners is here:
>
> http://www.djangoproject.com/documentation/testing/#defining-a-test-runner

Thanks, Russ! That's exactly what I was looking for. I had
started to poke around in the source for the "manage.py test" (in
core/managements/commands/test.py) command and wondered if I'd
have to implement my own custom command, but it looks like it's
already been thought of and all I need to do is take the existing
test-runner as a skeleton, and then run repeatedly with my custom
settings.

-tim