No such table error when using threads

[BlueBird]

Hi,

I've got a strange behavior. My application is split between a web
frontend and a backend which use the DB to exchange tasks and results.

The problem is with the backend. It's just a regular program,
accessing the DB to fetch tasks, completing them and then storing the
result in a DB. The program may (depending on a config) use multiple
threads to perform the tasks and store the result.

When running the test suite that exercise the backend, I get an error
if I allow multiple threads of execution :

err( 'Number of urlResult: %d' % CheckResultUrl.objects.count
() )
File "c:\Python26\lib\site-packages\django\db\models\manager.py",
line 108, in
count
return self.get_query_set().count
()
File "c:\Python26\lib\site-packages\django\db\models\query.py", line
292, in
count
return self.query.get_count
()
File "c:\Python26\lib\site-packages\django\db\models\sql\query.py",
line 376,
in
get_count
number = obj.get_aggregation()
[None]
File "c:\Python26\lib\site-packages\django\db\models\sql\query.py",
line 348,
in
get_aggregation
result = query.execute_sql
(SINGLE)
File "c:\Python26\lib\site-packages\django\db\models\sql\query.py",
line 2369, in
execute_sql
cursor.execute(sql,
params)
File "c:\Python26\lib\site-packages\django\db\backends\util.py",
line 19, in
execute
return self.cursor.execute(sql,
params)
File "c:\Python26\lib\site-packages\django\db\backends
\sqlite3\base.py", line
193, in
execute
return Database.Cursor.execute(self, query,
params)
OperationalError: no such table: dynamic_checkresulturl

The DB is initialized with :
setup_test_environment()
connection.creation.create_test_db()

On the stdout, I can see the different tables created :
scheduler\tests\test_checks.py Creating test
database...
Creating table
auth_permission
Creating table
auth_group
Creating table
auth_user
Creating table
auth_message
Creating table
django_content_type
Creating table
django_session
Creating table
django_site
Creating table
django_admin_log
Creating table
django_flatpage
Creating table
dynamic_domain
Creating table
dynamic_checkresultbase
Creating table
dynamic_checkresultonepage
Creating table
dynamic_checkresultdomain
Creating table
dynamic_checkresulturl
Creating table
dynamic_checkresultparenturl
Creating table
dynamic_userprofile
Creating table
dynamic_schedulertodo
Creating table
dynamic_billingentry
Creating table
registration_registrationprofile
Installing index for auth.Permission
model
Installing index for auth.Message
model
Installing index for admin.LogEntry
model
Installing index for flatpages.FlatPage
model
Installing index for dynamic.Domain
model
Installing index for dynamic.CheckResultOnePage
model
Installing index for dynamic.CheckResultDomain
model
Installing index for dynamic.CheckResultUrl
model
Installing index for dynamic.CheckResultParentUrl
model
Installing index for dynamic.BillingEntry
model
Installing json fixture 'initial_data' from 'fixtures'.

So, the table is created and has an index.

Where does the error come from ? I am really stuck.

My investigation reveals that with one main thread doing nothing and
one "slave" thread accessing the DB, I get the error. If the main
thread does all the job, no error.

Any ideas ?

This is on windows XP, with python 2.6.1 , version of django above 1.1
tracked from svn .

cheers,

Philippe

[BlueBird]

On 24 oct, 21:52, BlueBird <p...@freehackers.org> wrote:
> Hi,
>
> I've got a strange behavior. My application is split between a web
> frontend and a backend which use the DB to exchange tasks and results.
>
> The problem is with the backend. It's just a regular program,
> accessing the DB to fetch tasks, completing them and then storing the
> result in a DB. The program may (depending on a config) use multiple
> threads to perform the tasks and store the result.
>
> When running the test suite that exercise the backend, I get an error
> if I allow multiple threads of execution :
>

I've reduced the problem to one script, using the Poll application
from the tutorial.

The problem seems to be that when running in test mode with threads,
the behavior is
different than when running in test mode without threads, and the DB
is not set correctly
for the background thread.

The demonstration with a simple script, to run in the Poll tutorial:

==================================
import os
os.environ[ 'DJANGO_SETTINGS_MODULE' ] = 'settings'
import settings
import datetime, threading

#django stuff
from polls.models import *
from django.core.mail import mail_admins
from django.test.utils import *
from django.db import connection


def create_object():
print 'Creating Poll'
p = Poll()
p.question = "What's up doc ?"
p.pub_date = datetime.date.today()
p.save()
print 'Poll object saved. Id: %d' % p.id


WITH_THREAD = True

if __name__ == '__main__':
setup_test_environment()
old_db_name = settings.DATABASE_NAME
new_db_name = connection.creation.create_test_db(verbosity=1)
print 'New DATABASE:', new_db_name

if WITH_THREAD:
t = threading.Thread( target=create_object )
t.start()
t.join()
else:
create_object()

teardown_test_environment()
connection.creation.destroy_test_db( old_db_name )
=====================================

With WITH_THREAD set to True:

=====================================
Philippe@pc-philippe /cygdrive/d/work/django/django-tuto/mysite $
python
run_with_threads.py

Creating test
database...
Creating table
auth_permission
Creating table
auth_group
Creating table
auth_user
Creating table
auth_message
Creating table

django_admin_log

Creating table
django_content_type
Creating table
django_session
Creating table
django_site
Creating table

polls_poll
Creating table
polls_choice

Installing index for auth.Permission
model
Installing index for auth.Message
model
Installing index for admin.LogEntry
model

Installing index for polls.Choice
model
New
DATABASE: :memory:
Creating
Poll
Exception in thread
Thread-1:
Traceback (most recent call
last):
File "c:\Python26\lib\threading.py", line 522, in
__bootstrap_inner
self.run
()
File "c:\Python26\lib\threading.py", line 477, in
run
[...]

File "c:\Python26\lib\site-packages\django\db\backends
\sqlite3\base.py", line
193, in
execute
return Database.Cursor.execute(self, query,
params)
OperationalError: no such table:

polls_poll

Destroying test
database...
Philippe@pc-philippe /cygdrive/d/work/django/django-tuto/mysite
$
==============================

With WITH_THREAD set to False:

==============================
Philippe@pc-philippe /cygdrive/d/work/django/django-tuto/mysite $
python run_wi
th_threads.py

Creating test
database...
Creating table
auth_permission
Creating table
auth_group
Creating table
auth_user
Creating table
auth_message
Creating table

django_admin_log

Creating table
django_content_type
Creating table
django_session
Creating table
django_site
Creating table

polls_poll
Creating table
polls_choice

Installing index for auth.Permission
model
Installing index for auth.Message
model
Installing index for admin.LogEntry
model

Installing index for polls.Choice
model
New
DATABASE: :memory:
Creating
Poll
Poll object saved. Id:
1
Destroying test
database...
Philippe@pc-philippe /cygdrive/d/work/django/django-tuto/mysite
$
===============================

I've tried to increase the verbosity of create_test_db() and compare
the output in the two
cases. No difference !

Any idea ? Looks like a django bug to me in the in-memory DB used for
tests.

Philippe

[BlueBird]

Ok, I found a way around the problem, in case anyone ever has the same
problem. If you specify TEST_DATABASE_NAME in your settings, it will
force sqllite to use a file database instead of in-memory database,
and then the problem goes.

cheers,

Philippe

[Elad Silberring]

Thank YOU!!!!!!!