[Django] #31717: WSGI calling close causes premature close_if_unusable_or_obsolete calls during transaction?
Django
during transaction?
-----------------------------------------+------------------------
Reporter: ylilarry | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
I'm having an issue with uwsgi+postgresql+django combo. This issue does
not exist when I use the manage runserver development server.
The issue is, I'm constantly receiving this error from django
{{{
prod_1 | Traceback (most recent call last):
prod_1 | File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/base/base.py", line 253, in _cursor
prod_1 | return self._prepare_cursor(self.create_cursor(name))
prod_1 | File "/usr/local/lib/python3.8/dist-
packages/django/utils/asyncio.py", line 26, in inner
prod_1 | return func(*args, **kwargs)
prod_1 | File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/postgresql/base.py", line 231, in
create_cursor
prod_1 | cursor = self.connection.cursor()
prod_1 | psycopg2.InterfaceError: connection already closed
}}}
Possibly related:
https://code.djangoproject.com/ticket/25362
https://code.djangoproject.com/ticket/15802
However, I backtraced the stack of this error and found out Django is
calling {{{BaseDatabaseWrapper.close()}}} right before this error.
Specifically, this line:
{{{
def close_if_unusable_or_obsolete(self):
"""
Close the current connection if unrecoverable errors have occurred
or if it outlived its maximum age.
"""
if self.connection is not None:
# If the application didn't restore the original autocommit
setting,
# don't take chances, drop the connection.
if self.get_autocommit() != self.settings_dict["AUTOCOMMIT"]:
self.close()
return
}}}
And when the error happens, we have {{{self.get_autocommit()}}} returning
{{{False}}}, and {{{self.settings_dict["AUTOCOMMIT"]}}} returning
{{{True}}}.
Then I backtraced where {{{self.set_autocommit(False)}}} was called and
found out that this happened in {{{Atomic.__enter__()}}}
{{{
def __enter__(self):
connection = get_connection(self.using)
if not connection.in_atomic_block:
# Reset state when entering an outermost atomic block.
connection.commit_on_exit = True
connection.needs_rollback = False
if not connection.get_autocommit():
# Pretend we're already in an atomic block to bypass the
code
# that disables autocommit to enter a transaction, and
make a
# note to deal with this case in __exit__.
connection.in_atomic_block = True
connection.commit_on_exit = False
if connection.in_atomic_block:
# We're already in a transaction; create a savepoint, unless
we
# were told not to or we're already waiting for a rollback.
The
# second condition avoids creating useless savepoints and
prevents
# overwriting needs_rollback until the rollback is performed.
if self.savepoint and not connection.needs_rollback:
sid = connection.savepoint()
connection.savepoint_ids.append(sid)
else:
connection.savepoint_ids.append(None)
else:
connection.set_autocommit(False,
force_begin_transaction_with_broken_autocommit=True)
connection.in_atomic_block = True
}}}
I think something bad is happening when wsgi calls
{{{HttpResponseBase.close()}}} during a transaction, because
{{{
def close_old_connections(**kwargs):
for conn in connections.all():
conn.close_if_unusable_or_obsolete()
signals.request_started.connect(close_old_connections)
signals.request_finished.connect(close_old_connections)
}}}
and
{{{
# The WSGI server must call this method upon completion of the
request.
# See http://blog.dscpl.com.au/2012/10/obligations-for-calling-close-
on.html
def close(self):
for closer in self._resource_closers:
try:
closer()
except Exception:
pass
# Free resources that were still referenced.
self._resource_closers.clear()
self.closed = True
signals.request_finished.send(sender=self._handler_class)
}}}
causes {{{close_if_unusable_or_obsolete()}}} to be called during this
transaction, so {{{self.get_autocommit() !=
self.settings_dict["AUTOCOMMIT"]}}} closed the connection.
--
Ticket URL: <https://code.djangoproject.com/ticket/31717>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
during transaction?
Reporter: ylilarry | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by ylilarry):
I commented out
{{{signals.request_finished.connect(close_old_connections)}}} and the
problem immediately goes away.
Also manage runserver does not seem to call
{{{HttpResponseBase.close()}}}.
This gives me more confidence that where the issue lays.
--
Ticket URL: <https://code.djangoproject.com/ticket/31717#comment:1>
Django
during transaction?
Reporter: Yu Li | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by rm_):
* cc: rm_ (added)
Comment:
@ylilarry which uwsgi version? Can you share your uwsgi config?
--
Ticket URL: <https://code.djangoproject.com/ticket/31717#comment:2>
Django
during transaction?
Reporter: Yu Li | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by rm_):
@ylilarry please add a new comment next time please :) Could you reproduce
with ''enable-threads = true'' ?
--
Ticket URL: <https://code.djangoproject.com/ticket/31717#comment:3>
Django
during transaction?
Reporter: Yu Li | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: new => closed
* resolution: => needsinfo
Comment:
There's not enough to go on here. If you could upload a minimal reproduce
project so we can investigate that would be awesome. Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/31717#comment:4>