[Django] #31373: Django 3.x / uwsgi is leaking connections

41 views
Skip to first unread message

Django

unread,
Mar 17, 2020, 11:04:21 AM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-----------------------------------------+------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

After few hours all posgtress connections are used (too many connections
error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,
Mar 17, 2020, 11:05:54 AM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Changes (by Marcin Nowak):

* Attachment "Firefox_Screenshot_2020-03-17T15-05-35.104Z.png" added.

Django

unread,
Mar 17, 2020, 11:06:50 AM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

After few hours all posgtress connections are used (too many connections
error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:1>

Django

unread,
Mar 17, 2020, 3:44:17 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Changes (by Marcin Nowak):

* Attachment "Firefox_Screenshot_2020-03-17T19-42-34.977Z.png" added.

Django

unread,
Mar 17, 2020, 3:55:15 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database


> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

After few hours all posgtress connections are used (too many connections
error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

Facts:
- setting up CONN_MAX_AGE > 0 does not help
- starting `runserver` and making many calls with `ab -c 10 -n 100
http://127.0.0.1:8000/` uses all available connections and ends with error
500 (due to no available connections)
- starting `runserver --nothreading` and making calls with `ab -c 100 -n
100 http://127.0.0.1:8000/` works fine

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:2>

Django

unread,
Mar 17, 2020, 3:58:06 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database


> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - starting `runserver` and making many calls with `ab -c 10 -n 100
> http://127.0.0.1:8000/` uses all available connections and ends with
> error 500 (due to no available connections)
> - starting `runserver --nothreading` and making calls with `ab -c 100 -n
> 100 http://127.0.0.1:8000/` works fine

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

UWSGi threads are enabled (`--enable-threads`)

After few hours all posgtress connections are used (too many connections
error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

Facts:
- setting up CONN_MAX_AGE > 0 does not help
- starting `runserver` and making many calls with `ab -c 10 -n 100
http://127.0.0.1:8000/` uses all available connections and ends with error
500 (due to no available connections)
- starting `runserver --nothreading` and making calls with `ab -c 100 -n
100 http://127.0.0.1:8000/` works fine

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:3>

Django

unread,
Mar 17, 2020, 3:59:17 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


> UWSGi threads are enabled (`--enable-threads`)
>

> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - starting `runserver` and making many calls with `ab -c 10 -n 100
> http://127.0.0.1:8000/` uses all available connections and ends with
> error 500 (due to no available connections)
> - starting `runserver --nothreading` and making calls with `ab -c 100 -n
> 100 http://127.0.0.1:8000/` works fine

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


UWSGi threads are enabled (`--enable-threads`)

Python 3.8.1 and 3.7.3

After few hours all posgtress connections are used (too many connections
error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

Facts:
- setting up CONN_MAX_AGE > 0 does not help
- starting `runserver` and making many calls with `ab -c 10 -n 100
http://127.0.0.1:8000/` uses all available connections and ends with error
500 (due to no available connections)
- starting `runserver --nothreading` and making calls with `ab -c 100 -n
100 http://127.0.0.1:8000/` works fine

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:4>

Django

unread,
Mar 17, 2020, 5:03:08 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>

> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - starting `runserver` and making many calls with `ab -c 10 -n 100
> http://127.0.0.1:8000/` uses all available connections and ends with
> error 500 (due to no available connections)
> - starting `runserver --nothreading` and making calls with `ab -c 100 -n
> 100 http://127.0.0.1:8000/` works fine

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


UWSGi threads are enabled (`--enable-threads`)
Python 3.8.1 and 3.7.3

After few hours all posgtress connections are used (too many connections


error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

Facts:
- setting up CONN_MAX_AGE > 0 does not help

- CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab


-c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
ends with error 500 (due to no available connections)

- CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls


with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine


CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
install. View must call database.
CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:5>

Django

unread,
Mar 17, 2020, 5:13:04 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>

> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
> -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
> ends with error 500 (due to no available connections)
> - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
> with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
>

> CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
> install. View must call database.
> CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

UWSGi threads are enabled (`--enable-threads`)
Python 3.8.1 and 3.7.3

After few hours all posgtress connections are used (too many connections


error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

Facts:
- setting up CONN_MAX_AGE > 0 does not help
- CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
-c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
ends with error 500 (due to no available connections)
- CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine

- CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
install. Called view must access database.
- CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
- Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
connections (still there are several errors on high traffic, but app
server can "recover" and handle bunch of incoming requests)
- Looks like Django 3.x cannot reuse connections

`ab -c 10 -n 200 ` Django 3.0.4:
Complete requests: 200
Failed requests: 108

`ab -c 10 -n 1000` Django 2.2.11:
Complete requests: 1000
Failed requests: 37

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:6>

Django

unread,
Mar 17, 2020, 5:17:33 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>

> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
> -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
> ends with error 500 (due to no available connections)
> - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
> with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
> - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
> install. Called view must access database.
> - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
> - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
> connections (still there are several errors on high traffic, but app
> server can "recover" and handle bunch of incoming requests)
> - Looks like Django 3.x cannot reuse connections
>
> `ab -c 10 -n 200 ` Django 3.0.4:
> Complete requests: 200
> Failed requests: 108
>
> `ab -c 10 -n 1000` Django 2.2.11:
> Complete requests: 1000
> Failed requests: 37

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

UWSGi threads are enabled (`--enable-threads`)
Python 3.8.1 and 3.7.3

After few hours all posgtress connections are used (too many connections


error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

----

Facts:
- setting up CONN_MAX_AGE > 0 does not help
- CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
-c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
ends with error 500 (due to no available connections)
- CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
- CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
install. Called view must access database.
- CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
- Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
connections (still there are several errors on high traffic, but app
server can "recover" and handle bunch of incoming requests)
- Looks like Django 3.x cannot reuse connections

----

Runserver (with threads) test results:

`ab -c 10 -n 200 ` Django 3.0.4:
Complete requests: 200
Failed requests: 108

`ab -c 10 -n 1000` Django 2.2.11:
Complete requests: 1000
Failed requests: 37

----

There may be two separate cases:
1. not reusing connection when CONN_MAX_AGE is greater than 0
2. leaking (not closing?) connections on uwsgi with enabled threads and
CONN_MAX_AGE set to 0

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:7>

Django

unread,
Mar 17, 2020, 5:41:28 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>

> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

UWSGi threads are enabled (`--enable-threads`)
Python 3.8.1 and 3.7.3

After few hours all posgtress connections are used (too many connections


error).
pg_stat_activity is reporting idle connections with part of queries
visible.

I would try to deliver more details later.

[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

----

Facts:
- setting up CONN_MAX_AGE > 0 does not help
- CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
-c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
ends with error 500 (due to no available connections)
- CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
- CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
install. Called view must access database.
- CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
- Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
connections (still there are several errors on high traffic, but app
server can "recover" and handle bunch of incoming requests)
- Looks like Django 3.x cannot reuse connections

- uwsgi + threads + CONN_MAX_AGE=0 on localhost -> cannot reproduce

----

Runserver (with threads) test results:

`ab -c 10 -n 200 ` Django 3.0.4:
Complete requests: 200
Failed requests: 108

`ab -c 10 -n 1000` Django 2.2.11:
Complete requests: 1000
Failed requests: 37

----

There may be two separate cases:
1. not reusing connection when CONN_MAX_AGE is greater than 0

2. leaking (not closing?) connections on uwsgi (?) with enabled threads
(?) and CONN_MAX_AGE set to 0

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:8>

Django

unread,
Mar 17, 2020, 5:42:49 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0


> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>

> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>

New description:

Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

UWSGi threads are enabled (`--enable-threads`)
Python 3.8.1 and 3.7.3

After few hours all posgtress connections are used (too many connections


error).
pg_stat_activity is reporting idle connections with part of queries
visible.


[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

----

----

----

I'm not sure about the 2nd case.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:9>

Django

unread,
Mar 17, 2020, 6:53:05 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Changes (by Simon Charette):

* status: new => closed
* resolution: => needsinfo


Comment:

I understand your desire to see this addressed but without a simplified
test project that reproduces your issue it's impossible to determine
whether or not Django is at fault and not one of your projects third-party
app is.

Please reopen this issue if you can provide a reproduction case else it's
simply not actionable by triagers that don't have access to the project
you're trying to upgrade. Thanks!

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:10>

Django

unread,
Mar 17, 2020, 7:08:10 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------

Comment (by Marcin Nowak):

Replying to [comment:10 Simon Charette]:


> I understand your desire to see this addressed but without a simplified
test project that reproduces your issue it's impossible to determine

whether or not Django is at fault and its not one of your projects third-
party app that is.


>
> Please reopen this issue if you can provide a reproduction case else
it's simply not actionable by triagers that don't have access to the
project you're trying to upgrade. Thanks!

The case about not reusing db connections is reproducible.
1. Just create a new project
2. Setup postgresql_psycopg2 db, set CONN_MAX_AGE=30,
3. create view hitting the database (User.objects.count() for example),
bind it to "/" route
4. make sure your postgres has connection limit set to quite low value
(mine is 100)
5. manage.py runserver
6. run `ab -c 10 -n 200 https://127.0.0.1:8000/`

My case from production server is quite different, maybe related somehow.
There is a chance that my project is causing the issue, but it looks that
Django 3.x has own issue too, which maybe increases my problem. The most
significant change in my project was upgrade of Django, from 2.x to 3.x

Sorry for chaotic description. It's been 16 hour of my today's work, for
last few hours I was trying to solve the issue, find the cause (I gave up,
sorry). Now I'm working on getting Django 2.x back, backporting some 3.x
features to 2.x, so I cannot focus anymore on reproducing my issue in
isolated environment (I know how important it is).

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:11>

Django

unread,
Mar 17, 2020, 7:16:11 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Changes (by Marcin Nowak):

* Attachment "connmaxage.zip" added.

Example project with CONN_MAX_AGE issue

Django

unread,
Mar 17, 2020, 7:22:09 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------

Comment (by Marcin Nowak):

{{{
createdb connmaxage
python -m venv /tmp/env
. /tmp/env/bin/activate
pip install django==3.0.4
pip install psycopg2-binary
cd /tmp/
unzip connmaxage.zip
cd connmaxage
./manage.py runserver
}}}

then (on the 2nd terminal)

{{{
ab -c 10 -n 1000 http://127.0.0.1:8000/
}}}

Result:

{{{
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
Internal Server Error: /
Traceback (most recent call last):
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/base/base.py", line 220, in ensure_connection
self.connect()
File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/base/base.py", line 197, in connect
self.connection = self.get_new_connection(conn_params)
File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/postgresql/base.py", line 185, in
get_new_connection
connection = Database.connect(**conn_params)
File "/tmp/env/lib/python3.8/site-packages/psycopg2/__init__.py", line
126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: sorry, too many clients already


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/tmp/env/lib/python3.8/site-
packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/tmp/env/lib/python3.8/site-
packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/tmp/env/lib/python3.8/site-
packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args,
**callback_kwargs)
File "/tmp/connmaxage/connmaxage/main/views.py", line 7, in main
return HttpResponse(str(User.objects.count()))
File "/tmp/env/lib/python3.8/site-packages/django/db/models/manager.py",
line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/tmp/env/lib/python3.8/site-packages/django/db/models/query.py",
line 397, in count
return self.query.get_count(using=self.db)
File "/tmp/env/lib/python3.8/site-
packages/django/db/models/sql/query.py", line 517, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/tmp/env/lib/python3.8/site-
packages/django/db/models/sql/query.py", line 502, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/tmp/env/lib/python3.8/site-
packages/django/db/models/sql/compiler.py", line 1149, in execute_sql
cursor = self.connection.cursor()
File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/base/base.py", line 260, in cursor
return self._cursor()
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/base/base.py", line 236, in _cursor
self.ensure_connection()
File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/base/base.py", line 220, in ensure_connection
self.connect()
File "/tmp/env/lib/python3.8/site-packages/django/db/utils.py", line 90,
in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/base/base.py", line 220, in ensure_connection
self.connect()
File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/base/base.py", line 197, in connect
self.connection = self.get_new_connection(conn_params)
File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
line 26, in inner
return func(*args, **kwargs)
File "/tmp/env/lib/python3.8/site-
packages/django/db/backends/postgresql/base.py", line 185, in
get_new_connection
connection = Database.connect(**conn_params)
File "/tmp/env/lib/python3.8/site-packages/psycopg2/__init__.py", line
126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL: sorry, too many clients already

[17/Mar/2020 23:20:23] "GET / HTTP/1.0" 500 166202
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:12>

Django

unread,
Mar 17, 2020, 7:27:18 PM3/17/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------

Comment (by Marcin Nowak):

Ab results for Django 3:

{{{
Complete requests: 1000
Failed requests: 901
}}}

Re-test with Django 2:

1. in the same env `pip install django==2.2.11`
2. do same `ab` test:

{{{
Complete requests: 1000
Failed requests: 19
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:13>

Django

unread,
Mar 18, 2020, 1:07:08 AM3/18/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* status: closed => new
* severity: Normal => Release blocker
* cc: Andrew Godwin (added)
* type: Uncategorized => Bug
* component: Uncategorized => Database layer (models, ORM)
* resolution: needsinfo =>
* stage: Unreviewed => Accepted


Comment:

Thanks for the sample project.

I managed to bisect the regression at
a415ce70bef6d91036b00dd2c8544aed7aeeaaed which introduced async support in
Django.

I didn't investigate why connections kept piling up but I suspect
[https://github.com/django/django/commit/a415ce70bef6d91036b00dd2c8544aed7aeeaaed
#diff-c50ed26e574fb41592a832e0b37c5713R143-R148 these lines] are to blame.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:14>

Django

unread,
Mar 18, 2020, 1:39:28 AM3/18/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Andrew Godwin):

That would not surprise me - next things to investigate are:

- Of course, reverting that one change and seeing if it fixes it.
- Does uwsgi override any part of the threading module? This happens with
gevent and, I think, mod_wsgi, and it causes havoc as they ruin thread
IDs.
- Checking a couple of recent major versions of asgiref within Django's
compatibility range, to see if the change was there (doubtful)

Likely cause is that the close_old_connections() task is never being run
in what Local thinks is the same thread as before. I have a potential
workaround we can do if it _is_ overriding thread IDs, but if not, we'll
have to dive deeper and see what we can fix about it.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:15>

Django

unread,
Mar 18, 2020, 1:47:46 AM3/18/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* cc: Simon Charette (added)


Comment:

Andrew what about `runserver`?

I understand it's not meant to be used as a production http server but the
fact the issue can be reproduced pretty easily makes me believe something
might be wrong with this commit or `asgiref.local.Local`.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:16>

Django

unread,
Mar 18, 2020, 2:11:11 AM3/18/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Andrew Godwin):

Oh, I missed the `runserver` comment - that would indeed indicate a severe
problem, as it uses bog-standard WSGI threading.

I suspect the problem will be with Local, as it is likely not precisely
replicating threading.local enough - working out how would be my line of
investigation. On the plus side, if it is that, we can ship a fix to
asgiref quickly out-of-cycle with Django and everyone will benefit.

I'd volunteer to look, but starting a new job and the current world
situation means I won't have time to do the investigation for a little
bit, but if we can trace it down to a direct behavioural difference I can
likely write the fix.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:17>

Django

unread,
Mar 18, 2020, 11:41:14 AM3/18/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson):

Same results.

Running on `stable/2.2.x`, just with `runserver`, I see some
`psycopg2.OperationalError: FATAL: sorry, too many clients already`
errors, but it does OK — it recovers for a while, before an error again,
and so on. (It's only runserver...)

Upping to `stable/3.0.x`, the limit is reached and then it never recovers.

On `stable/3.0.x`, (and `master`), reverting **just** the use of
`asgiref.local.Local` to `threading.local` in the lines Simon identified
in `django/db/utils.py` leads to a better-than-stable/2.2.x behaviour.
(It's essentially the same, but errors slightly less often, so presumably
something else changed to account for that.)

(Disabling the auto-reloader makes no apparent difference.)

> I suspect the problem will be with Local, as it is likely not precisely
replicating threading.local enough

This looks favourite then.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:18>

Django

unread,
Mar 18, 2020, 12:29:32 PM3/18/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Reupen Shah):

The root cause is probably different, but I think it's worth pointing out
that asgiref.local.Local also has problems with Gunicorn + gevent:
https://github.com/django/asgiref/issues/144

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:19>

Django

unread,
Mar 18, 2020, 7:15:37 PM3/18/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Andrew Godwin):

OK, I have attempted to rewrite Local to more closely match how
`threading` does it - I've pushed a new branch to `asgiref` called
`new_local`. Could you test against that?
https://github.com/django/asgiref/tree/new_local

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:20>

Django

unread,
Mar 19, 2020, 10:30:24 AM3/19/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson):

I gave the asgiref new_local branch a run against runserver (with Django
stable/3.0.x).
This again exhibits some errors but then recovers (similar to 2.2)

@Marcin are you able to try in your project?

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:21>

Django

unread,
Mar 25, 2020, 3:08:57 AM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------

Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed

Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by felixxm):

* status: new => closed

* resolution: => invalid


Comment:

We confirmed that it was an issue in `asgiref` that was fixed in
`asgiref==3.2.7` (Thanks Andrew!).

With `asgiref==3.2.5`:
{{{
Time taken for tests: 47.321 seconds
Complete requests: 500
Failed requests: 400
}}}


With `asgiref==3.2.7`:
{{{
Time taken for tests: 4.413 seconds
Complete requests: 500
Failed requests: 0
}}}

Note: `asgiref==3.2.6` introduced a regression and shouldn't be used.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:22>

Django

unread,
Mar 25, 2020, 5:19:56 AM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Marcin Nowak):

> @Marcin are you able to try in your project?

I think I will, but give me a some time.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:23>

Django

unread,
Mar 25, 2020, 5:36:58 AM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson):

Hi Marcin, thanks.

Since it’s been addressed in `asgiref`, a `pip install -U asgiref` (or
equivalent, given that you probably don’t deploy by hand) should do it.

Do report back! :)

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:24>

Django

unread,
Mar 25, 2020, 8:49:09 AM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

Should we bump `asgi` to `>=3.2.7` in the next 3.0 release and on master?

https://github.com/django/django/blob/ab5720ad80b48fac746021e826fe76d0869f9c02/setup.cfg#L42

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:25>

Django

unread,
Mar 25, 2020, 11:37:32 AM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Andrew Godwin):

I would rather put it as `~=3.2` as that will ensure that installations
auto-upgrade to an asgiref 3.3 and beyond if they are released, but not a
potentially-API-incompatible asgiref 4. If we use a simple greater-than,
we'd have to also put a `< 4`.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:26>

Django

unread,
Mar 25, 2020, 11:59:17 AM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

Andrew, agreed. As long as it reduces the amount of reports related to
this hard to diagnose issue I think its worth doing.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:27>

Django

unread,
Mar 25, 2020, 12:21:00 PM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Andrew Godwin):

Right. I don't think there's a situation other than an explicit version
pin where someone will pull in an old version of asgiref (the only one I
can think of is an out-of-date internal mirror).

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:28>

Django

unread,
Mar 25, 2020, 2:56:23 PM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by felixxm):

Replying to [comment:26 Andrew Godwin]:


> I would rather put it as `~=3.2` as that will ensure that installations
auto-upgrade to an asgiref 3.3 and beyond if they are released, but not a
potentially-API-incompatible asgiref 4. If we use a simple greater-than,
we'd have to also put a `< 4`.

It is set that way in Django 3.0, see
[https://github.com/django/django/blob/2306f8c9bb2bc9539cfe77e38affc01b09f0bfb4/setup.py#L86
setup.py#L86]. We can decide to pin a specific version of `asgiref` to
Django 3.1. after a feature freeze ~ May, 2020.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:29>

Django

unread,
Mar 25, 2020, 7:43:47 PM3/25/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Marcin Nowak):

Hi Carlton.

I'll report back, but this is more complicated than just pip install or
something like that.

I've backported few things and I'm monkey patching Django 2.x. I must
revert these changes, switch to 3.x, and deploy a new release.
And because there is still a little risk, I have to be careful and deploy
a new release at the time of low traffic, then I must have some time to
observe metrics.

But I'll do this and give you a feedback, for sure.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:30>

Django

unread,
Mar 26, 2020, 3:06:36 AM3/26/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson):

Hi Marcin. Thanks! (Sounds fun... :)

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:31>

Django

unread,
Mar 26, 2020, 5:58:12 AM3/26/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Marcin Nowak):

> Sounds fun... :)

Yes, I'm not strong in English.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:32>

Django

unread,
Mar 27, 2020, 5:07:46 PM3/27/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Marcin Nowak):

I've updated to Django 3.0.4 and asgiref 3.2.7 - PosgreSQL connections
graph looks good.
I think the issue was fixed. Thank you for fast reaction and bugfix.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:33>

Django

unread,
Mar 30, 2020, 6:37:04 AM3/30/20
to django-...@googlegroups.com
#31373: Django 3.x / uwsgi is leaking connections.
-------------------------------------+-------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: invalid
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Marcin Nowak):

After few days I can finally confirm that new asgiref version solves the
issue. Thanks.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:34>

Reply all
Reply to author
Forward
0 new messages