[Django] #35583: asgiref.sync.sync_to_async cannot be affected by close_old_connections

[Django]

#35583: asgiref.sync.sync_to_async cannot be affected by close_old_connections
------------------------------+-----------------------------------------
Reporter: AlexandrOnuf | Type: Uncategorized
Status: new | Component: Uncategorized
Version: | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------+-----------------------------------------
Rel. to https://code.djangoproject.com/ticket/34914#comment:3
partially rel. to this implementation
https://github.com/django/channels/blob/main/channels/db.py

1. Let's assume, that we have django command, that creates a "server" that
listening for a message from queue, or periodically do some job.
2. Second point - our service have async function and requires DB calls
(dummy example in ''_test'' function)
3. let's assume. that we have Postgres DB running in docker compose - and
to reproduce this bug, we should call **docker compose restart db**

Similar to Django itself and, for example, to Django channels
https://github.com/django/channels/blob/main/channels/db.py we want to be
able to close closed connection (simplest way to reproduce - DB was
restarted).

**Expected behavior:** we are assuming, that we do same steps as we can
see in Django project examples and we should see **success** message in
logs.
**Actual**: we will see ''failure: the connection is closed''

**What we can see in code itself:**
1. sync_to_async class will use its own isolated ThreadPoolExecutor
2. django connections will be created withing this ThreadPoolExecutor and
will be unavailable for us (this is my assumption)

**Found workaround**
Calling this code will implicitly have access to hidden DB pool and will
close closed connections

{{{
syncio.run(sync_to_async(close_old_connections)())
}}}


Notes:
1. I'm not really sure - is it a feature request or a bug (taking into
account, how hard to find theoretical reason of it)
2. I didn't dive into details of **async_to_sync** function/class

Code snippet for Django command
{{{
import asyncio
import time

from asgiref.sync import sync_to_async
from django.db import close_old_connections
from django.core.management.base import BaseCommand

# This can be any model
from django.contrib.auth.models import User


class Command(BaseCommand):
def handle(self, *args, **options):
while True:
process_message()
time.sleep(3)


def process_message():
print("close_old_connections")
close_old_connections()

print("run test")
try:
test()
print("success")
except Exception as e:
print("failure:", str(e))
print("done test")


def test():
asyncio.run(_test())


async def _test():
"""
Synthetic case - most likely, we will use ``sync_to_async`` as
decorator for part of functions
and will use it directly for other calls ``sync_to_async``.
"""
await User.objects.afirst()
await sync_to_async(
User.objects.first
)()
await sync_to_async(
User.objects.first,
thread_sensitive=False,
)()
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35583>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35583: asgiref.sync.sync_to_async cannot be affected by close_old_connections

-------------------------------+--------------------------------------
Reporter: AlexandrOnuf | Owner: (none)

Type: Uncategorized | Status: new
Component: Uncategorized | Version:

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------+--------------------------------------
Description changed by AlexandrOnuf:

Old description:

New description:

Rel. to https://code.djangoproject.com/ticket/34914#comment:3
partially rel. to this implementation
https://github.com/django/channels/blob/main/channels/db.py

1. Let's assume, that we have django command, that creates a "server" that
listening for a message from queue, or periodically do some job.
2. Second point - our service have async function and requires DB calls
(dummy example in ''_test'' function)
3. let's assume. that we have Postgres DB running in docker compose - and
to reproduce this bug, we should call **docker compose restart db**

Similar to Django itself and, for example, to Django channels
https://github.com/django/channels/blob/main/channels/db.py we want to be
able to close closed connection (simplest way to reproduce - DB was
restarted).

**Expected behavior:** we are assuming, that we do same steps as we can
see in Django project examples and we should see **success** message in
logs.
**Actual**: we will see ''failure: the connection is closed''

**What we can see in code itself:**
1. sync_to_async class will use its own isolated ThreadPoolExecutor
2. django connections will be created withing this ThreadPoolExecutor and
will be unavailable for us (this is my assumption)

**Found workaround**
Calling this code will implicitly have access to hidden DB pool and will
close closed connections

{{{

asyncio.run(sync_to_async(close_old_connections)())

--
Ticket URL: <https://code.djangoproject.com/ticket/35583#comment:1>

[Django]

#35583: asgiref.sync.sync_to_async cannot be affected by close_old_connections

-------------------------------------+-------------------------------------
Reporter: Alexandr Onufrienko | Owner: (none)
Type: Bug | Status: closed
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):

* cc: Carlton Gibson (added)
* component: Uncategorized => Database layer (models, ORM)
* resolution: => needsinfo
* status: new => closed
* type: Uncategorized => Bug

Comment:

Hello Alexandr, thank you for your report.

From the provided description, I can't quite understand the exact issue
you are reporting. Could you please describe the use case and what you're
trying to achieve in detail? This will help us understand the problem
better and find the right solution.

There are a few things to consider:

1. When you said "''we have django command, that creates a "server" that
listening for a message from queue, or periodically do some job''". This
is in general against the recommended Django pattern for management
commands. If you need to have a long running process, the ideal solution
is to use a tool that integrates with Django and also provides the queue
functionality such as Celery or any other task queue. See
[https://forum.djangoproject.com/t/debugging-the-client-was-disconnected-
by-the-server-because-of-inactivity/21730 this relevant discussion] or
[https://forum.djangoproject.com/t/when-are-db-connections-closed/16347 or
this one] in the Forum.

2. We would also need details about your database configuration
(`CONN_MAX_AGE` specifically), Django version, etc. I have tried the code
sample provided with a postgresql database and I don't get the error you
have reported (I do get ''ResourceWarning: connection <psycopg.Connection
[IDLE] ...> was deleted while still open. Please use 'with' or '.close()'
to close the connection'').

Initially I thought this could be similar to #31905 but after some further
thinking I don't think it's the case.
--
Ticket URL: <https://code.djangoproject.com/ticket/35583#comment:2>