Re: [Django] #35651: The number of Redis connections has been continuously increasing. (was: django redis cache not really using connection pooling)

[Django]

#35651: The number of Redis connections has been continuously increasing.
-------------------------------------+-------------------------------------
Reporter: gojuukaze | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by gojuukaze):

* summary: django redis cache not really using connection pooling => The
number of Redis connections has been continuously increasing.


Old description:

> I'm using `django.core.cache.backends.redis.RedisCache` as a backend for
> redis, and I recently observed that the number of Redis connections has
> been continuously increasing.
>
> After debugging, I found that whenever the cache method is called, the
> `RedisCacheClient `class is reinitialized, and in the `__init__` function
> of `RedisCacheClient`, the connection pool is set to empty.
>
> This causes the `_get_connection_pool` method of `RedisCacheClient` to
> always create a new connection pool instead of using the existing one.
>
> {{{
> class RedisCacheClient:
> def __init__(
> self,
> servers,
> serializer=None,
> pool_class=None,
> parser_class=None,
> **options,
> ):
> import redis
>
> self._lib = redis
> self._servers = servers
> self._pools = {} # << === set pools
>
> def _get_connection_pool(self, write):
> index = self._get_connection_pool_index(write)
>
> #
> # self._pools is is always empty.
> #
>
> if index not in self._pools:
> self._pools[index] = self._pool_class.from_url(
> self._servers[index],
> **self._pool_options,
> )
> return self._pools[index]
> }}}
>

> One solution is to put `_pools` outside of `__init__` , for example：
>

> {{{
> class RedisCacheClient:
>
> # init pool
> _pools = {}
>
> def __init__(
> self,
> servers,
> serializer=None,
> pool_class=None,
> parser_class=None,
> **options,
> ):
> import redis
>
> self._lib = redis
> self._servers = servers
> ## self._pools={}
>
> }}}
>
> By the way, I am using Django 5.0.7 and running it in asynchronous mode.

New description:

I'm using `django.core.cache.backends.redis.RedisCache` as a backend for
redis, and I recently observed that the number of Redis connections has
been continuously increasing.

After debugging, I found that whenever the cache method is called, the
`RedisCacheClient `class is reinitialized, and in the `__init__` function
of `RedisCacheClient`, the connection pool is set to empty.

This causes the `_get_connection_pool` method of `RedisCacheClient` to
always create a new connection pool and a new connection instead of using
the existing one.

{{{
class RedisCacheClient:
def __init__(
self,
servers,
serializer=None,
pool_class=None,
parser_class=None,
**options,
):
import redis

self._lib = redis
self._servers = servers
self._pools = {} # << === set pools

def _get_connection_pool(self, write):
index = self._get_connection_pool_index(write)

#
# self._pools is is always empty.
#

if index not in self._pools:
self._pools[index] = self._pool_class.from_url(
self._servers[index],
**self._pool_options,
)
return self._pools[index]
}}}


One solution is to put `_pools` outside of `__init__` , for example：


{{{
class RedisCacheClient:

# init pool
_pools = {}

def __init__(
self,
servers,
serializer=None,
pool_class=None,
parser_class=None,
**options,
):
import redis

self._lib = redis
self._servers = servers
## self._pools={}

}}}

By the way, I am using Django 5.0.7 and running it in asynchronous mode.

--
--
Ticket URL: <https://code.djangoproject.com/ticket/35651#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.