[Django] #30335: TypeError: unhashable type: 'list' when paginating queryset with KeyTransform annotation
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 2.2
layer (models, ORM) | Keywords: postgresql jsonb
Severity: Normal | keytransform annotation pagination
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
We have encountered a unexpected error when trying to upgrade an
application from Django 2.1 to 2.2.
In our application we have the requirement to order a queryset on a value
that's found in a nested datastructure stored in a JSON field. To do this
we use `KeyTransform`s to annotate the queryset with this value and order
on this annotation. This queryset is then paginated.
This used to work in Django 2.1, but now raises a `TypeError` in Django
2.2.
I created a minimal example to reproduce the error:
`models.py`:
{{{
from django.db import models
from django.contrib.postgres.fields import JSONField
class Example(models.Model):
data = JSONField(default=dict)
}}}
`tests.py`:
{{{
from django.test import TestCase
from django.core.paginator import Paginator
from django.contrib.postgres.fields.jsonb import KeyTextTransform,
KeyTransform
from .models import Example
class TestKeyTransformPagination(TestCase):
def setUp(self):
Example.objects.create(data={
'translations': [
{'title': 'Ladies and gentleman'},
{'title': 'Dames en heren'}
]
})
Example.objects.create(data={
'translations': [
{'title': 'Apples and pears'},
{'title': 'Appels en peren'}
]
})
# The next example queryset raises: TypeError: unhashable type:
'list'
# qs = Example.objects.order_by('data__translations__0__title')
# qs[0]
#
# This is a workaround to achieve the desired ordering
extract_title = KeyTextTransform('title', KeyTransform('0',
KeyTransform('translations', 'data')))
self.qs =
Example.objects.annotate(title=extract_title).order_by('title')
def test_queryset(self):
self.assertEqual('Apples and pears',
self.qs[0].data['translations'][0]['title'])
def test_pagination(self):
# Works on 2.1.x and master, raises TypeError: unhashable type:
'list' on 2.2
paginator = Paginator(self.qs, per_page=10)
page = paginator.page(1)
self.assertEqual('Apples and pears',
page.object_list[0].data['translations'][0]['title'])
}}}
While investigating this issue I discovered that this error does not
happen on the current master.
After bisecting I can tell that commit
3767c7ff391d5f277e25bca38ef3730ddf9cea9c (Fixed #29244) introduces (or
exposes) the exception and 3f32154f40a855afa063095e3d091ce6be21f2c5 (Fixed
#30188) fixes the error.
Is it possible to backport the fix for #30188 to Django 2.2 so we can
upgrade our application?
--
Ticket URL: <https://code.djangoproject.com/ticket/30335>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: postgresql jsonb | Triage Stage:
keytransform annotation | Unreviewed
pagination |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jaap Roes):
By the way, I'm not sure if
`Example.objects.order_by('data__translations__0__title')[0]` is supposed
to work? It currently raises a `TypeError` which is why we are using
`annotate` with `KeyTransform`s as a work around.
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:1>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Keywords: postgresql jsonb | Triage Stage: Accepted
keytransform annotation |
pagination |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
Thanks for the report. I can reproduce with the provided example. (Passes
on 2.1.x, fails on 2.2.x)
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:2>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: postgresql jsonb | Triage Stage: Accepted
keytransform annotation |
pagination |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Can Sarıgöl (added)
* has_patch: 0 => 1
Comment:
Hi, is this can be a solution?
[https://github.com/django/django/pull/11181 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:3>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: postgresql jsonb | Triage Stage: Accepted
keytransform annotation |
pagination |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jaap Roes):
Replying to [comment:3 Can Sarıgöl]:
> Hi, is this can be a solution?
[https://github.com/django/django/pull/11181 PR]
That looks good, it seems that it would also remove the need for our
workaround, which I'm always in favor of :-)
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:4>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
Reporter: Jaap Roes | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: postgresql jsonb | Triage Stage: Accepted
keytransform annotation |
pagination |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by felixxm):
Related ticket #29139.
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:5>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
| Sarıgöl
Type: Bug | Status: assigned
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: postgresql jsonb | Triage Stage: Accepted
keytransform annotation |
pagination |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => assigned
* owner: nobody => Can Sarıgöl
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:6>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
| Sarıgöl
Type: Bug | Status: assigned
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Release blocker | Resolution:
keytransform annotation | checkin
pagination |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:7>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
| Sarıgöl
Component: Database layer | Version: 2.2
(models, ORM) |
Keywords: postgresql jsonb | Triage Stage: Ready for
keytransform annotation | checkin
pagination |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"d87bd29c4f8dfcdf3f4a4eb8340e6770a2416fe3" d87bd29]:
{{{
#!CommitTicketReference repository=""
revision="d87bd29c4f8dfcdf3f4a4eb8340e6770a2416fe3"
Fixed #30335, #29139 -- Fixed crash when ordering or aggregating over a
nested JSONField key transform.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:8>
Django
KeyTransform annotation
-------------------------------------+-------------------------------------
| Sarıgöl
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: postgresql jsonb | Triage Stage: Ready for
keytransform annotation | checkin
pagination |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"e85317d73113382c96e87f36f0d732e3084df145" e85317d7]:
{{{
#!CommitTicketReference repository=""
revision="e85317d73113382c96e87f36f0d732e3084df145"
[2.2.x] Fixed #30335, #29139 -- Fixed crash when ordering or aggregating
over a nested JSONField key transform.
Backport of d87bd29c4f8dfcdf3f4a4eb8340e6770a2416fe3 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30335#comment:9>