[Django] #28334: contrib.postgresql overwhelms database with "select from pg_type" queries on each request
Django
on each request
------------------------------------------------+------------------------
Reporter: Igor Gumenyuk | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: contrib.postgres | Version: 1.11
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
On each newly created database connection `django.contrib.postgresql`
tries to register hstore type for this connection via `connection_created`
signal and `register_hstore` function.
https://github.com/django/django/blob/master/django/contrib/postgres/apps.py#L20
https://github.com/django/django/blob/stable/1.11.x/django/contrib/postgres/signals.py#L16
Behind the scenes `register_hstore` runs sql query to get `oid` of
`hstore` type:
https://github.com/psycopg/psycopg2/blob/master/lib/extras.py#L885
This happens on every(!) request (unless persistent connections enabled)
SELECT t.oid, %s
FROM pg_type t JOIN pg_namespace ns
ON typnamespace = ns.oid
WHERE typname = 'hstore';
This is just huge overhead, since `register_hstore` accepts `oid` argument
to avoid hitting database every call.
We have seen significant latency spikes because of this.
Possible solution would be have configurable `GET_HSTORE_OID_FUNC` in
`DATABASES->OPTIONS` setting which can be cached in any way.
This will also maintain backwards compatibility.
I can send patch for this if proposed solution is acceptable to be merged
in master.
--
Ticket URL: <https://code.djangoproject.com/ticket/28334>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
on each request
Reporter: Igor Gumenyuk | Owner: (none)
Cleanup/optimization |
Component: contrib.postgres | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Igor Gumenyuk:
Old description:
> On each newly created database connection `django.contrib.postgresql`
> tries to register hstore type for this connection via
> `connection_created` signal and `register_hstore` function.
>
> https://github.com/django/django/blob/master/django/contrib/postgres/apps.py#L20
> https://github.com/django/django/blob/stable/1.11.x/django/contrib/postgres/signals.py#L16
>
> Behind the scenes `register_hstore` runs sql query to get `oid` of
> `hstore` type:
> https://github.com/psycopg/psycopg2/blob/master/lib/extras.py#L885
>
> This happens on every(!) request (unless persistent connections enabled)
> SELECT t.oid, %s
> FROM pg_type t JOIN pg_namespace ns
> ON typnamespace = ns.oid
> WHERE typname = 'hstore';
>
> This is just huge overhead, since `register_hstore` accepts `oid`
> argument to avoid hitting database every call.
> We have seen significant latency spikes because of this.
>
> Possible solution would be have configurable `GET_HSTORE_OID_FUNC` in
> `DATABASES->OPTIONS` setting which can be cached in any way.
> This will also maintain backwards compatibility.
>
> I can send patch for this if proposed solution is acceptable to be merged
> in master.
New description:
On each newly created database connection `django.contrib.postgresql`
tries to register hstore type for this connection via `connection_created`
signal and `register_hstore` function.
https://github.com/django/django/blob/master/django/contrib/postgres/apps.py#L20
https://github.com/django/django/blob/stable/1.11.x/django/contrib/postgres/signals.py#L16
Behind the scenes `register_hstore` runs sql query to get `oid` of
`hstore` type:
https://github.com/psycopg/psycopg2/blob/master/lib/extras.py#L885
This happens on every(!) request (unless persistent connections enabled)
SELECT t.oid, %s
FROM pg_type t JOIN pg_namespace ns
ON typnamespace = ns.oid
WHERE typname = 'hstore';
This is just huge overhead, since `register_hstore` accepts `oid` argument
to avoid hitting database every call.
We have seen significant latency spikes because of this.
Possible solution would be to have configurable `GET_HSTORE_OID_FUNC` in
`DATABASES->OPTIONS` setting which can be cached in any way.
This will also maintain backwards compatibility.
I can send patch for this if proposed solution is acceptable to be merged
in master.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:1>
Django
on each request
Reporter: Igor Gumenyuk | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: contrib.postgres | Version: 1.11
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:2>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Component: contrib.postgres | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Igor Gumenyuk):
* status: new => assigned
* owner: (none) => Igor Gumenyuk
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:3>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Igor Gumenyuk):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:4>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* needs_better_patch: 0 => 1
Comment:
There are test failures on the pull request.
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:5>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Igor Gumenyuk):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:6>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Igor Gumenyuk):
All issues resolved, there are no failed tests
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:7>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* needs_tests: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:8>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Igor Gumenyuk):
* needs_tests: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:9>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:10>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:11>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Component: contrib.postgres | Version: 1.11
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"86a18dc46aae3a4a6410e3f3d864fa0ec4e5b2cd" 86a18dc]:
{{{
#!CommitTicketReference repository=""
revision="86a18dc46aae3a4a6410e3f3d864fa0ec4e5b2cd"
Fixed #28334 -- Added caching for hstore/citext OIDs.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:12>
Django
on each request
Reporter: Igor Gumenyuk | Owner: Igor
Type: | Gumenyuk
Cleanup/optimization | Status: closed
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"f7b0532ec0cf703849ef50e513e3dc6b5483a6b0" f7b0532]:
{{{
#!CommitTicketReference repository=""
revision="f7b0532ec0cf703849ef50e513e3dc6b5483a6b0"
Refs #28334 -- Fixed crash in hstore/citext oid caching during test db
creation.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:13>