[Django] #21202: Atomic masks fatal database errors and instead propagates "connection already closed"

Django

unread,

Sep 30, 2013, 3:51:49 PM9/30/13

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"
----------------------------------------------+------------------------
Reporter: intgr | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer (models, ORM) | Version: 1.6-beta-1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 1
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+------------------------
I noticed this problem with the new Atomic code in Django 1.6: when the
PostgreSQL server raises a FATAL error (meaning that the server closes the
connection after the error), then the Atomic block will swallow the
original error and propagates a meaningless "InterfaceError: connection
already closed" instead. I don't know whether this also affects other
database backends.

I find that it's quite annoying to debug problems when the original cause
of an error is hidden by subsequent error recovery code. It's not a new
problem with Atomic, I believe commit_on_success (and Christophe Pettus's
xact) had the same problem.

The easiest way to reproduce this is to terminate your own connection. (In
PostgreSQL 9.1 and earlier you need to be superuser to do this):

{{{#!python
from django.db import connection
from django.db.transaction import atomic

with atomic():
cur = connection.cursor()
cur.execute("select pg_terminate_backend(pg_backend_pid())")
cur.fetchall()
}}}

With current Django 1.6 this raises an error like:

{{{
Traceback (most recent call last):
File "term1.py", line 7, in <module>
cur.fetchall()
File "/home/marti/src/django/django/db/transaction.py", line 351, in
__exit__
connection.set_autocommit(True)
File "/home/marti/src/django/django/db/backends/__init__.py", line 335,
in set_autocommit
self._set_autocommit(autocommit)
File
"/home/marti/src/django/django/db/backends/postgresql_psycopg2/base.py",
line 184, in _set_autocommit
self.connection.autocommit = autocommit
psycopg2.InterfaceError: connection already closed
}}}

It tells you that the connection is closed, but not why. And there's a
second problem too: a psycopg2 exception is propagated into user code,
without using Django's exception wrappers.

To solve this problem, I submitted pull request 1696:
https://github.com/django/django/issues/1696

1. Changed BaseDatabaseWrapper.set_autocommit to wrap backend-specific
errors.
2. Wrapped `Atomic.__exit__` in another layer of try/except and re-
raising the original exception in case of an InterfaceError.

Now the original exception is properly raised:

{{{
django.db.utils.OperationalError: terminating connection due to
administrator command
}}}

I can also write unit tests if someone can help me figure out how to test
this (the example I provided above requires PostgreSQL, with either
superuser privs or version 9.2+).

Ran all 'transactions' and 'db_backends' tests on these changes with both
SQLite (OK (skipped=42)) and psycopg2 (OK (skipped=1))

--
Ticket URL: <https://code.djangoproject.com/ticket/21202>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Sep 30, 2013, 4:06:40 PM9/30/13

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

-------------------------------------+-------------------------------------
Reporter: intgr | Owner: aaugustin
Type: | Status: assigned
Cleanup/optimization | Version:
Component: Database layer | 1.6-beta-1
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 1
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by aaugustin):

* status: new => assigned
* needs_better_patch: => 1
* needs_tests: => 0
* owner: nobody => aaugustin
* needs_docs: => 0
* type: Uncategorized => Cleanup/optimization
* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:1>

Django

unread,

Oct 1, 2013, 5:05:53 AM10/1/13

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by intgr):

Hi, I see you set the "Patch needs improvement" flag, but I can't see a
review anywhere. Am I missing something?

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:2>

Django

unread,

Oct 1, 2013, 6:43:18 AM10/1/13

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by aaugustin):

The report only covers SQLite and PostgreSQL. We need to check and
possibly make changes for MySQL and Oracle.

The diff in `atomic` is very scary, but maybe that's just a side effect of
indenting everything.

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:3>

Django

unread,

Oct 1, 2013, 6:58:00 AM10/1/13

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by intgr):

Yeah, the diff in atomic is not really scary at all, here's a whitespace-
independent diff for 2nd commit:

{{{#!patch
diff --git a/django/db/transaction.py b/django/db/transaction.py
index 86a357f..2a4f16d 100644
--- a/django/db/transaction.py
+++ b/django/db/transaction.py
@@ -18,7 +18,7 @@ from functools import wraps

from django.db import (
connections, DEFAULT_DB_ALIAS,
- DatabaseError, ProgrammingError)
+ DatabaseError, ProgrammingError, InterfaceError)
from django.utils.decorators import available_attrs

@@ -312,6 +312,7 @@ class Atomic(object):
connection.in_atomic_block = False

try:
+ try:
if exc_type is None and not connection.needs_rollback:
if connection.in_atomic_block:
# Release savepoint if there is one
@@ -354,6 +355,14 @@ class Atomic(object):
elif not connection.savepoint_ids and not
connection.commit_on_exit:
connection.in_atomic_block = False

+ except InterfaceError:
+ if exc_type is not None:
+ # We tried to clean up from an exception, but ran into an
+ # InterfaceError, meaning the connection is unusable from
now
+ # on. Propagate up the original error instead of masking
it
+ # with InterfaceError ("connection already closed")
+ return
+
def __call__(self, func):
@wraps(func, assigned=available_attrs(func))
def inner(*args, **kwargs):
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:4>

Django

unread,

Feb 20, 2014, 9:19:13 AM2/20/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by dji):

Whatever happened to this? Seeing this happen a lot on production with
1.6. Even worse, the connection then doesn't get closed, and so all
future requests error as well. So should we also be closing the bad
connection... somewhere?

Longer explanation: As it is with this patch applied (to 1.6, I can't seem
to get dev to run), `django.db.close_old_connections()` fails on
`conn.rollback()` (which is called by `conn.abort()`) with the same
`InterfaceError`. This means that `conn.close_if_unusable_or_obsolete()`
never gets called, and the broken connection is reused for later requests,
causing all future requests on that connection to error.

You can replicate this behavior pretty easily as described above, by
enabling connection pooling and terminating your own connection in a
request, and then trying to make a subsequent request. I suspect this is
still the behavior in dev as well, nothing appears to have changed.

My "solution" is to squelch `InterfaceError` as well as `DatabaseError` in
`close_old_connections`, but I'm new to the Django internals so I'm not
sure if this is best, or if it would be better to just close the
connection straight away here in `__exit__` before we return, or what.
Happy to help sort this but I'm not sure of the correct approach.

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:5>

Django

unread,

Mar 23, 2014, 6:41:35 PM3/23/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Has patch: 1 | Patch needs improvement: 0

Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------

Changes (by aaugustin):

* needs_better_patch: 1 => 0

Comment:

https://github.com/django/django/pull/2468

I believe that patch works for the case when the database closes the
connection and also adresses comment 5.

It also prevents further queries until the exit of the outermost atomic
block, thus preserving atomicity.

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:6>

Django

unread,

Apr 8, 2014, 8:20:17 AM4/8/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Changes (by depaolim):

* cc: depaolim@… (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:7>

Django

unread,

Apr 10, 2014, 4:04:05 PM4/10/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by Aymeric Augustin <aymeric.augustin@…>):

In [changeset:"3becac84845cee8f12a5fb7f68c87cbaf029c6a0"]:
{{{
#!CommitTicketReference repository=""
revision="3becac84845cee8f12a5fb7f68c87cbaf029c6a0"
Fixed #22321 -- Wrapped exceptions in _set_autocommit.

Refs #21202.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:8>

Django

unread,

Apr 10, 2014, 4:04:06 PM4/10/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

-------------------------------------+-------------------------------------
Reporter: intgr | Owner: aaugustin

Type: | Status: closed

Cleanup/optimization | Version:
Component: Database layer | 1.6-beta-1

(models, ORM) | Resolution: fixed

Changes (by Aymeric Augustin <aymeric.augustin@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"8176150850b2e34b2afe1dc107e184eb4c6cd668"]:
{{{
#!CommitTicketReference repository=""
revision="8176150850b2e34b2afe1dc107e184eb4c6cd668"
Fixed #21202 -- Maintained atomicity when the server disconnects.

Thanks intgr for the report.

This commit doesn't include a test because I don't know how to emulate a
database disconnection in a cross-database compatible way.

Also simplified a 'backends' test that was constrained by this problem.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:9>

Django

unread,

Apr 10, 2014, 5:24:02 PM4/10/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by Aymeric Augustin <aymeric.augustin@…>):

In [changeset:"746cded010c7ba3e2bf2df4cde32dcc9b75cb0d3"]:
{{{
#!CommitTicketReference repository=""
revision="746cded010c7ba3e2bf2df4cde32dcc9b75cb0d3"
[1.6.x] Fixed #22321 -- Wrapped exceptions in _set_autocommit.

Refs #21202.

Backport of 3becac84 from master
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:10>

Django

unread,

Apr 10, 2014, 5:24:03 PM4/10/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by Aymeric Augustin <aymeric.augustin@…>):

In [changeset:"1d3d2b9a2458cbf11b7e828fb59b3f2cb73d14fe"]:
{{{
#!CommitTicketReference repository=""
revision="1d3d2b9a2458cbf11b7e828fb59b3f2cb73d14fe"
[1.6.x] Fixed #21202 -- Maintained atomicity when the server disconnects.

Thanks intgr for the report.

This commit doesn't include a test because I don't know how to emulate a
database disconnection in a cross-database compatible way.

Also simplified a 'backends' test that was constrained by this problem.

Backport of 81761508 from master
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:11>

Django

unread,

Apr 10, 2014, 5:24:49 PM4/10/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by Aymeric Augustin <aymeric.augustin@…>):

In [changeset:"2ad0bc132a6f70a128d8e1fec689c6a474ddde89"]:
{{{
#!CommitTicketReference repository=""
revision="2ad0bc132a6f70a128d8e1fec689c6a474ddde89"
[1.7.x] Fixed #21202 -- Maintained atomicity when the server disconnects.

Thanks intgr for the report.

This commit doesn't include a test because I don't know how to emulate a
database disconnection in a cross-database compatible way.

Also simplified a 'backends' test that was constrained by this problem.

Backport of 81761508 from master
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/21202#comment:13>

Django

unread,

Apr 10, 2014, 5:24:48 PM4/10/14

to django-...@googlegroups.com

#21202: Atomic masks fatal database errors and instead propagates "connection
already closed"

Comment (by Aymeric Augustin <aymeric.augustin@…>):

In [changeset:"5f22bda3827f32289793b90a387d8c77d602a859"]:
{{{
#!CommitTicketReference repository=""
revision="5f22bda3827f32289793b90a387d8c77d602a859"
[1.7.x] Fixed #22321 -- Wrapped exceptions in _set_autocommit.