Suspected bug: testing with multiple references to same database

[Shai Berger]

Hi all,

I am developing an application which needs to write an audit log to a
database. I want the committing of log records to be independent of the
committing of other database writes (so user actions can be audited even if
they cause errors). So I figured the easiest way to do so would be to use the
new multiple-database support: The logging would go on a separate connection,
and I can then commit it separately.

The twist is, I don't want the log to go to a separate database -- just to a
separate connection. So I use two aliases with the same parameters. And
testing misbehaves.

At first, I did it with a trivial

DATABASES['auditlog'] = DATABASES['default']

in the settings, and with no explicit TEST_NAME for the database. Assume my
database is named "main"

Now, when I tried to run tests, the tests refuse to run unless the "main"
(i.e. non-test) database exists. While running, the test creates databases
"test_main" and "test_test_main"; when it ends, it removes "test_test_main"
and "main", so that only "test_main" remains. This is quite nasty. I can guess
where it comes from (the DATABASES values are manipulated for testing, and
nobody thought there could be two references to the same dict there), but
anyway, seems buggy.

Things get better if I either separate the dicts by

DATABASES['auditlog'] = DATABASES['default'].copy()

or explicitly specify the TEST_NAME in the definition; then I still get
misbehavior (the harness tries to create the database twice and fails to
destroy it the second time), but that's an "acceptable" misbehavior.

I anticipate that there will be some "that's not the way to achieve what you
want" responses, and I welcome any suggestions to achieve the same goal
otherwise, but even if what I am doing is considered utter misconfiguration, I
think it should be handled a little more gracefully than deleting the main
database -- please consider this before sending me off to django-users.

Barring any glaring omission on my part, I'd like to open a bug for this.

Thanks,
Shai.

[Russell Keith-Magee]

Hi Shai,

It certainly sounds to me like a legitimate bug (or group of bugs).

The "point two aliases at the same physical database" approach was
considered as a possible use case for multi-db; and in normal database
operation, it should work fine. You're just the first person to notice
the potential problems when it comes to constructing test databases.

It's possible that this problem might be able to be solved with a
tweak of the way TEST_MIRROR is handled. TEST_MIRROR was originally
intended to provide a way for master/slave setups to be easily tested,
making the test_slave point at the test_master so that the test system
didn't have to try and reproduce the master/slave replication
configuration. The side effect is that databases marked as a mirror
don't get a test database created -- they get pointed at the test
database that they are mirroring.

The class of setup you describe shouldn't require an explicit mirror
to be defined, but it should be able to be handled with the same sort
of logic. A pre-processing step that identifies duplicated database
connections and inserts an alias should do the trick.

So - yes, please open a ticket for this. Feel free to try your hand at
a patch, too -- this almost rates as an 'easy pickings' ticket. This
is a situation where you'll get a pass on having a test as part of the
patch; testing test setup is one of those areas that's almost
impossible to validate, so whoever commits (probably me) will just
take it for a very good walk before checking in.

Yours,
Russ Magee %-)

[Shai Berger]

On Thursday 07 October 2010 07:31:49 Russell Keith-Magee wrote:
>
> It certainly sounds to me like a legitimate bug (or group of bugs).
>

It is now bug #14415.

Thanks,
Shai.

[Filip Gruszczyński]

I am not very familiar with all this, but I took a look at the code
and have a question. Would it be enough to make a following validation
before setting up test databases: for each two connections using the
same backend names of the test databases are different?

--
Filip Gruszczyński

[Russell Keith-Magee]

2010/10/11 Filip Gruszczyński <grus...@gmail.com>:

That would remove the (potentially) data damaging side effects, but
doesn't really fix the problem. If I have a production database
configuration where alias X and Y point to the same database, then it
makes sense to me that the test database configuration should do the
same.

Yours,
Russ Magee %-)