[Django] #33085: Random errors - django.db.utils.InterfaceError: connection already closed
Django
-----------------------------------------+----------------------------
Reporter: Rich | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Severity: Normal | Keywords: postgresql
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+----------------------------
Some background - I've been using this stack, deployed in AWS in various
applications for many years without issue:
- Ubuntu (in this case 20.04 LTS)
- Nginx
- Uwsgi
- Postgresql (v12 in RDS - tried v13 but same errors)
An AWS load balancer sends traffic to the Ubuntu instance, which is
handled by Nginx, which forwards on to Django (3.2.6) running in Uwsgi.
Django connects to the database using psycopg2 (2.9.1). Normally this
setup works perfectly for me.
The issue I have is that in one of my new projects using this setup, the
database connection seems to be closing randomly. Django reports errors
like this:
{{{
Traceback (most recent call last):
[my code...]
for answer in q.select_related('entry__session__player'):
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 280, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size)
File "/usr/local/lib/python3.8/dist-
packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/base/base.py", line 259, in cursor
return self._cursor()
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/base/base.py", line 237, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line
90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/base/base.py", line 237, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/postgresql/base.py", line 236, in
create_cursor
cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed
}}}
The location in my code varies. Sometimes (less frequently) I get this
too:
{{{
Traceback (most recent call last):
[my code...]
group = contest.groups.create(restaurant = restaurant, supergroup =
supergroup)
File "/usr/local/lib/python3.8/dist-
packages/django/db/models/fields/related_descriptors.py", line 677, in
create
return super(RelatedManager, self.db_manager(db)).create(**kwargs)
File "/usr/local/lib/python3.8/dist-
packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 453, in create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/base.py",
line 726, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/local/lib/python3.8/dist-packages/django/db/models/base.py",
line 763, in save_base
updated = self._save_table(
File "/usr/local/lib/python3.8/dist-packages/django/db/models/base.py",
line 868, in _save_table
results = self._do_insert(cls._base_manager, using, fields,
returning_fields, raw)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/base.py",
line 906, in _do_insert
return manager._insert(
File "/usr/local/lib/python3.8/dist-
packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/usr/local/lib/python3.8/dist-
packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False,
executor=self._execute)
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/utils.py", line 78, in _execute
self.db.validate_no_broken_transaction()
File "/usr/local/lib/python3.8/dist-
packages/django/db/backends/base/base.py", line 447, in
validate_no_broken_transaction
raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the
current transaction. You can't execute queries until the end of the
'atomic' block.
}}}
Again, the location in my code varies, and it's not always in a simple
''create'' call - sometimes it's ''bulk_create'', sometimes
''get_or_create''. I'm guessing that the underlying cause is probably the
same as the 'connection_closed' error but I'm not sure.
So that's what Django tells me. The Postgresql log doesn't contain any
errors that coincide in time with the errors reported by Django. The only
errors in the log are of this form:
{{{
LOG: could not receive data from client: Connection reset by peer
}}}
These coincide with uwsgi killing off worker processes (I have a 1000
request limit set for each worker to avoid any potential memory leak
issues), so they aren't related.
So Postgresql is reporting no relevant errors - I can only assume that the
connection was closed properly, and Django wasn't expecting that. There
are no errors in the systemd journal at all on the Ubuntu instance.
So I'm unsure how to proceed. I doubt this is a bug in Django but no other
component in the system is complaining and it must be a low-level issue.
This happens quite rarely, but enough to be a concern - something like 1
in 1000 requests.
Any suggestions of how to investigate this further would be gratefully
accepted - I'm happy to jump into the Django source to add some logging or
try some tweaks. Thanks in advance!
--
Ticket URL: <https://code.djangoproject.com/ticket/33085>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Rich | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 3.2
Severity: Normal | Resolution: invalid
Keywords: postgresql | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* status: new => closed
* resolution: => invalid
Comment:
See TicketClosingReasons/UseSupportChannels for ways to get help. If you
confirm a bug in Django, please reopen with details.
--
Ticket URL: <https://code.djangoproject.com/ticket/33085#comment:1>