spawning threads within test cases in django test framework

[Brian]

I've got a django app with a periodically scheduled background task that updates the database. I've written a bunch of tests for its principal class that are run as part of the django unit test framework. I want to convert the class to do its work using multiple threads, but I'm having trouble getting the tests to run against the multi-threaded version.

I've tried sqlite3 and postgres as back-ends, but to no avail. Is there a problem where the testing framework uses transactions to rollback any changes from one test to the next and so the evolving state of the database in that transaction is hidden from the other threads? Can the database connection (and hence transaction) be shared between the threads? Has anyone encountered this problem before? (And, ideally, came up with a really neat solution...)

Thanks,

Brian 

[Duane Griffin]

On 31 May 2011 22:01, Brian <brian....@gmail.com> wrote:
> I've got a django app with a periodically scheduled background task that
> updates the database. I've written a bunch of tests for its principal class
> that are run as part of the django unit test framework. I want to convert
> the class to do its work using multiple threads, but I'm having trouble
> getting the tests to run against the multi-threaded version.

I've had a similar problem trying to get some locking unit tests
working. Unfortunately I don't have a neat solution: I gave up in the
end :(

The first problem was, as you guessed, that the database connection
(or more specifically the cursor), can't be shared between threads.
IIRC it is stored as a thread-local variable and won't be correctly
initialised for the new threads. I managed to get it initialised by
doing the following:

import django.db
django.db.close_connection()
cursor = connection.cursor()

...however, that initialised it to use the *real* database, not the
test one. Very confusing test failures it produced, too :) Anyway, I
tried a few different things to try and get it all working, but gave
up after a while. I intend to come back to it at some point so if you
manage to get this working I'd be very interested in the solution!

> Thanks,
> Brian

Cheers,
Duane.

--
"I never could learn to drink that blood and call it wine" - Bob Dylan

[Kirill Spitsin]

On Tue, May 31, 2011 at 02:01:00PM -0700, Brian wrote:
> I've got a django app with a periodically scheduled background task that
> updates the database. I've written a bunch of tests for its principal class
> that are run as part of the django unit test framework. I want to convert
> the class to do its work using multiple threads, but I'm having trouble
> getting the tests to run against the multi-threaded version.
>
> I've tried sqlite3 and postgres as back-ends, but to no avail. Is there a
> problem where the testing framework uses transactions to rollback any
> changes from one test to the next and so the evolving state of the database
> in that transaction is hidden from the other threads?

Right.

> Can the database connection (and hence transaction) be shared between
> the threads? Has anyone encountered this problem before? (And,
> ideally, came up with a really neat solution...)

Try to use TransactionTestCase instead of TestCase (and don't use
in-memory SQLite datebase).

--
Kirill Spitsin

[Brian]

Thanks for the replies. I realised that I could reorganise the code so that all the database updating was done in the main thread. Re TransactionTestCases, I knew I'd read something related to that but didn't find it when I went looking yesterday evening; it sounds like that would have worked in this case.