[Django] #17998: database threading issue with dev server and sqlite
Django
----------------------------------------------+--------------------
Reporter: anonymous | Owner: nobody
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: 1.4
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
Steps to reproduce:
* Run dev server in the standard way "./manage.py runserver"
* Request a Django-served page in a browser
* Likely important that the page has links to static media files
served by the dev server so the browser initiates multiple
simultaneous
requests
* There is middleware on the site that hits the session table in the db
for every request, including the static media files
* You may need to flush the browser cache when trying to repeatedly see
the error
Expected behavior:
* All files served normally
Observed behavior:
* Non-deterministic, some requests cause an error
* Error response is the following: "A server error occurred. Please
contact the administrator."
* The exception in the error log reads:
{{{
DatabaseError: DatabaseWrapper objects created in a thread can only be
used in that same thread. The object with alias 'default' was created in
thread id 4374663168 and this is thread id 4385443840.
}}}
* Note that the Django site in question is only using the Django ORM, not
reaching deeper into the django.db library, and it does not explicitly
spawn any threads
* I'm including a full backtrace from the dev server below.
Environment:
* Mac OS X 10.6.8
* python27 @2.7.2_1 (MacPorts)
* py27-sqlite @2.6.3_0 (MacPorts)
* sqlite3 @3.7.11_0 (MacPorts)
* Django 1.4.0-final0 (pip)
* Firefox 11.0
Full backtrace:
{{{
DEBUG:django.db.backends:(0.003) SELECT "django_session"."session_key",
"django_session"."session_data", "django_session"."expire_date" FROM
"django_session" WHERE ("django_session"."session_key" =
02462878adc69e05312600990114b569 AND "django_session"."expire_date" >
2012-03-28 10:29:14.458986 ); args=('02462878adc69e05312600990114b569',
u'2012-03-28 10:29:14.458986')
DEBUG:django.db.backends:(0.000) SELECT (1) AS "a" FROM "django_session"
WHERE "django_session"."session_key" = 02462878adc69e05312600990114b569
LIMIT 1; args=('02462878adc69e05312600990114b569',)
DEBUG:django.db.backends:(0.000) UPDATE "django_session" SET
"session_data" =
NDY1YWVmMjI5ZTZiOGIxODUyMjVlMTAxOGQ2MjUzNTEzOWRmMWQ1NzqAAn1xAS4=
, "expire_date" = 2012-03-28 10:59:14.471961 WHERE
"django_session"."session_key" = 02462878adc69e05312600990114b569 ;
args=('NDY1YWVmMjI5ZTZiOGIxODUyMjVlMTAxOGQ2MjUzNTEzOWRmMWQ1NzqAAn1xAS4=\n',
u'2012-03-28 10:59:14.471961', '02462878adc69e05312600990114b569')
Traceback (most recent call last):
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py",
line 85, in run
self.result = application(self.environ, self.start_response)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/django/contrib/staticfiles/handlers.py", line 67, in
__call__
return self.application(environ, start_response)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/django/core/handlers/wsgi.py", line 243, in __call__
signals.request_finished.send(sender=self.__class__)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/django/dispatch/dispatcher.py", line 172, in send
response = receiver(signal=self, sender=sender, **named)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/django_digest/backend/db.py", line 54, in close_connection
self.connection.close()
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/django/db/backends/sqlite3/base.py", line 319, in close
self.validate_thread_sharing()
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/django/db/backends/__init__.py", line 136, in
validate_thread_sharing
% (self.alias, self._thread_ident, thread.get_ident()))
DatabaseError: DatabaseWrapper objects created in a thread can only be
used in that same thread. The object with alias 'default' was created in
thread id 4374663168 and this is thread id 4385443840.
[28/Mar/2012 10:29:14] "GET / HTTP/1.1" 500 59
}}}
Thanks very much, Django is great software!
--
Ticket URL: <https://code.djangoproject.com/ticket/17998>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by akaariai):
* cc: anssi.kaariainen@… (added)
* needs_better_patch: => 0
* stage: Unreviewed => Accepted
* needs_tests: => 0
* needs_docs: => 0
Comment:
I have looked into this a little, but I can't see where the connection is
shared between the threads. Any ideas? It should be impossible to get the
same `DatabaseWrapper` in different threads unless explicitly shared. Of
course, it is possible there is a but making it not so impossible, which
would be more serious.
While I haven't reproduced this bug, I am still going to mark this
accepted, as I have seen some strange threading errors, too (caused by
using ipdb in my case, though).
--
Ticket URL: <https://code.djangoproject.com/ticket/17998#comment:1>
Django
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Changes (by akaariai):
* stage: Accepted => Unreviewed
Comment:
I am going to revert the accepted triaging. I have been trying to
reproduce this, and I can't. The method I have been using is a test view
which does a database query and then sleeps a little. Then I hit that view
continuously with high-concurrency AB benchmark. No errors.
In addition I have been reading the code, and I can't see a way the
connection would get shared between threads.
So, more info is needed.
--
Ticket URL: <https://code.djangoproject.com/ticket/17998#comment:2>
Django
Reporter: anonymous | Owner: nobody
Type: Bug | Status: closed
(models, ORM) | Resolution:
Keywords: | Triage Stage:
Has patch: 0 | Unreviewed
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Changes (by akaariai):
* status: new => closed
* resolution: => worksforme
Comment:
I can't reproduce this and I haven't seen other complaints about this.
Closing this as worksforme.
--
Ticket URL: <https://code.djangoproject.com/ticket/17998#comment:3>
Django
Reporter: anonymous | Owner: nobody
Component: Database layer | Version: 1.4
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
Has patch: 0 | Unreviewed
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by trey9000):
The bug turned out to be in the django_digest package that we were using.
See some discussion here:
https://bitbucket.org/akoha/django-digest/issue/10/conflict-with-global-
databasewrapper
A patched version of django-digest that works for me can be found here:
https://github.com/trey0/django-digest
Sorry for reporting this in the wrong place...
--
Ticket URL: <https://code.djangoproject.com/ticket/17998#comment:4>
Django
Reporter: anonymous | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.4
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
Has patch: 0 | Unreviewed
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by anonymous):
got the same exception when enable
{{{
from gevent import monkey; monkey.patch_all()
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/17998#comment:5>
Django
Reporter: anonymous | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.4
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
Has patch: 0 | Unreviewed
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by anonymous):
Replying to [comment:5 anonymous]:
> got the same exception when enable
> {{{
> from gevent import monkey; monkey.patch_all()
> }}}
Could be fixed by not patching threads:
{{{
gevent.monkey.patch_all(socket=True, dns=True, time=True,
select=True,thread=False, os=True, ssl=True, httplib=False,
aggressive=True)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/17998#comment:6>
Django
Reporter: anonymous | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.4
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
Has patch: 0 | Unreviewed
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by idbill@…):
I am recently had this issue occur after I increased the logging level on
a mod_wsgi instance.... I log quite a bit in debug mode, but rarely with
that setting on anything other than in my development environment (which
is usually running via runserver).
--
Ticket URL: <https://code.djangoproject.com/ticket/17998#comment:7>