[Django] #32015: BUG: psycopg2 json is not loaded with connections.cursor

6 views

Skip to first unread message

Django

unread,

Sep 17, 2020, 3:22:05 PM9/17/20

to django-...@googlegroups.com

#32015: BUG: psycopg2 json is not loaded with connections.cursor
-------------------------------------+-------------------------------------
Reporter: mblance | Owner: nobody
Type: | Status: new
Uncategorized |
Component: | Version: 3.1
Uncategorized | Keywords: psycopg2, database,
Severity: Normal | postgresql, json, jsonb, jsonfield
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Since the upgrade from v3->v3.1, I discovered this regression in the
psycopg2 cursor. jsonb results from sql queries are not loaded into
python data structures. This causes issues when dumping and loading
again, the json fields do not get loaded on the front. The offending line
of code can be found here
https://github.com/django/django/blob/master/django/db/backends/postgresql/base.py#L206
in the postgresql backend.

This code was introduced to work around issues with specifying a
JSONDecoder in the new JSONField. Attempting to implement a workaround by
creating additional connections sort of works, but causes freezing during
cleanup of test cases, and maybe more serious issues elsewhere.

--
Ticket URL: <https://code.djangoproject.com/ticket/32015>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Sep 18, 2020, 1:09:27 AM9/18/20

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by felixxm):

* status: new => closed
* type: Uncategorized => Bug
* component: Uncategorized => Database layer (models, ORM)
* resolution: => duplicate

Old description:

> Since the upgrade from v3->v3.1, I discovered this regression in the
> psycopg2 cursor. jsonb results from sql queries are not loaded into
> python data structures. This causes issues when dumping and loading
> again, the json fields do not get loaded on the front. The offending
> line of code can be found here
> https://github.com/django/django/blob/master/django/db/backends/postgresql/base.py#L206
> in the postgresql backend.
>
> This code was introduced to work around issues with specifying a
> JSONDecoder in the new JSONField. Attempting to implement a workaround
> by creating additional connections sort of works, but causes freezing
> during cleanup of test cases, and maybe more serious issues elsewhere.
>

> This is also causing issues with a ValuesQuerySet, the output from jsonb
> fields is always string.

New description:

Since the upgrade from v3->v3.1, I discovered this regression in the
psycopg2 cursor. jsonb results from sql queries are not loaded into
python data structures. This causes issues when dumping and loading
again, the json fields do not get loaded on the front. The offending line
of code can be found here
https://github.com/django/django/blob/master/django/db/backends/postgresql/base.py#L206
in the postgresql backend.

This is also causing issues with a ValuesQuerySet, the output from jsonb
fields is always string.

Comment:

> Since the upgrade from v3->v3.1, I discovered this regression in the
psycopg2 cursor. jsonb results from sql queries are not loaded into
python data structures.

This is an intended change, see #31991.

> This causes issues when dumping and loading again, the json fields do
not get loaded on the front.

I cannot reproduce any issue with `dumpdata`/`loaddata`.

> This is also causing issues with a ValuesQuerySet, the output from jsonb
fields is always string.

Using `QuerySet.values()` works for me.

Duplicate of #31991. I'm happy to reopen if you can provide a sample
project that reproduce described issues with builtin mechanisms.