[Django] #35455: psycopg3 warns about connections not being closed

[Django]

#35455: psycopg3 warns about connections not being closed
-------------------------------------+-------------------------------------
Reporter: HTErik | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 5.0
layer (models, ORM) |
Severity: Normal | Keywords: psycopg
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
When running
* a Django 5.0.6 application
* psycopg 3.1.19
* with db `"ENGINE": "django.db.backends.postgresql"`
* with Python warnings as errors enabled

This occasionally prints following error from psycopg:


{{{
ResourceWarning: connection <psycopg.Connection [IDLE] (REDACTED) at
0x710d29abe120> was deleted while still open. Please use 'with' or
'.close()' to close the connection
}}}

This warning comes from the `__del__` function in
`psycopg.connection.BaseConnection`.

I have no idea how to reproduce this.
I tried patching psycopg to print a stack trace whenever a connection is
opened and closed. This tells me that the connection that was deleted
before being closed, was opened through the following code path:


{{{
File "blah.py", line 193, in __heartbeat
MyModel.objects.bulk_update(items_to_run, ["last_update"])
File "python3.12/site-packages/django/db/models/manager.py", line 87, in
manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "python3.12/site-packages/django/db/models/query.py", line 922, in
bulk_update
with transaction.atomic(using=self.db, savepoint=False):
File "python3.12/site-packages/django/db/transaction.py", line 198, in
__enter__
if not connection.get_autocommit():
File "python3.12/site-packages/django/db/backends/base/base.py", line
450, in get_autocommit
self.ensure_connection()
File "python3.12/site-packages/django/utils/asyncio.py", line 26, in
inner
return func(*args, **kwargs)
File "python3.12/site-packages/django/db/backends/base/base.py", line
275, in ensure_connection
self.connect()
File "python3.12/site-packages/django/utils/asyncio.py", line 26, in
inner
return func(*args, **kwargs)
File "python3.12/site-packages/django/db/backends/base/base.py", line
256, in connect
self.connection = self.get_new_connection(conn_params)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35455>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35455: psycopg3 warns about connections not being closed
-------------------------------------+-------------------------------------
Reporter: HTErik | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: psycopg | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by HTErik):

I have identified the problem now.
It happens when one performs Django queries from a separate thread. The DB
connections in Django are thread local inside the `ConnectionHandler`, so
when the thread finishes and is eventually GCd, the thread locals holding
the DB connections in Django are also GCd, without any prior `close()`.

To solve it, one must call this in a `finally` clause of all threads that
run Django queries.

{{{
for conn in django.db.connections.all():
conn.close()
}}}

I'm not sure if this is expected from users or if this is something Django
should handle internally.
The documentation on this topic could use some more details.
https://docs.djangoproject.com/en/5.0/ref/databases/#connection-management
--
Ticket URL: <https://code.djangoproject.com/ticket/35455#comment:1>

[Django]

#35455: psycopg3 warns about connections not being closed
-------------------------------------+-------------------------------------
Reporter: HTErik | Owner: nobody

Type: Bug | Status: closed

Component: Database layer | Version: 5.0
(models, ORM) |

Severity: Normal | Resolution: invalid

Keywords: psycopg | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* resolution: => invalid
* status: new => closed

Comment:

Hi HTErik, I recommend you share more details of this in the async section
of the forum: https://forum.djangoproject.com/c/internals/async/8, as this
is the place for discussion and support.
If there's an agreement that there is a bug in Django or that the docs
need updating, come back and I'll be happy to accept improvements 👍
--
Ticket URL: <https://code.djangoproject.com/ticket/35455#comment:2>