[Django] #29643: Hashing list in Q objects when using __in lookup
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Type: | Status: new
Uncategorized |
Component: Utilities | Version: 2.0
Severity: Normal | Keywords: hash, tuple, list,
Triage Stage: | drf
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
When using djangorestframework with django 2.0.X with lookup __in like
id__in=[1,2,3] will raise: unhashable type: 'list'
traceback ends to this file:
/django/utils/tree.py" in __hash__
74. return hash((self.__class__, self.connector, self.negated) +
tuple(self.children))
When using tuples with __in everything works fine. But when using
.values_list() on query I need to use tuple(query) to use it in Q object
with __in lookup.
Code worked before with 1.11 version just fine. Also when using __in
without Q() everything seems to work just fine.
--
Ticket URL: <https://code.djangoproject.com/ticket/29643>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Component: Utilities | Version: 2.0
Severity: Normal | Resolution:
Keywords: hash, tuple, list, | Triage Stage:
drf | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> When using djangorestframework with django 2.0.X with lookup __in like
> id__in=[1,2,3] will raise: unhashable type: 'list'
>
> traceback ends to this file:
> /django/utils/tree.py" in __hash__
> 74. return hash((self.__class__, self.connector, self.negated)
> + tuple(self.children))
> When using tuples with __in everything works fine. But when using
> .values_list() on query I need to use tuple(query) to use it in Q object
> with __in lookup.
> Code worked before with 1.11 version just fine. Also when using __in
> without Q() everything seems to work just fine.
New description:
When using djangorestframework with django 2.0.X with lookup _in like
{{{id__in=[1,2,3]}}} will raise: unhashable type: 'list'
traceback ends to this file:
{{{
/django/utils/tree.py" in __hash__
74. return hash((self.__class__, self.connector, self.negated) +
tuple(self.children))
}}}
When using tuples with {{{__in}}} everything works fine. But when using
.values_list() on query I need to use tuple(query) to use it in Q object
with {{{__in}}} lookup.
Code worked before with 1.11 version just fine. Also when using _in
without Q() everything seems to work just fine.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Component: Utilities | Version: 2.0
Severity: Normal | Resolution:
Keywords: hash, tuple, list, | Triage Stage:
drf | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
Please include a minimal example to reproduce the crash. My guess of
`Model.objects.filter(Q(id__in=[1,2,3]))` didn't crash. Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Component: Utilities | Version: 2.0
Severity: Normal | Resolution:
Keywords: hash, tuple, list, | Triage Stage:
drf | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by rhyre):
{{{
Model.objects.prefetch_related('types').exclude(status=1).annotate(
category=Case(
When(Q(status__in=[1,2,3]) & Q(id__in=[1,2,3]),
then=Value(2)),
default=Value(1),
output_field=IntegerField()
)
).only('category')
}}}
then it is used in viewsets.ReadOnlyModelViewSet of rest framework in
get_queryset.
When using
{{{
When(status__in=[1,2,3], id__in=[1,2,3], then=Value(2)),
}}}
everything works fine. Also when using Q with tuple as:
{{{
When(Q(status__in=(1,2,3)) & Q(id__in=(1,2,3)), then=Value(2)),
}}}
its ok.
Only with list it will crash. I've workarounded this to not use Q but it
wasn't a problem before.
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Component: Utilities | Version: 2.0
Severity: Normal | Resolution:
Keywords: hash, tuple, list, | Triage Stage:
drf | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Carlton Gibson):
Hi. Thanks for the follow-up.
This still isn’t really a minimal reproduce.
Can you provide a test-case that demonstrates the issue? Or a small
project?
Any chance you could use git bisect to identify the commit which broke
your usage?
> Without DRF it seems to work just fine. There is a problem with
serialization
What exactly do you mean here? It’s not quite clear. (Do you mean that the
ORM call works fine until embedded with the DRF request-response handling?
If so you’ll need to provide a sample project I think…(?))
Could you paste the entire traceback for the error?
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Component: Utilities | Version: 2.0
Severity: Normal | Resolution:
Keywords: hash, tuple, list, | Triage Stage:
drf | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: felixxm (added)
Comment:
It is probably a regression introduced by regression fix (see
https://github.com/django/django/commit/fc6528b25ab1834be1a478b405bf8f7ec5cf860c).
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Component: Utilities | Version: 2.0
Severity: Normal | Resolution:
Keywords: hash, tuple, list, | Triage Stage:
drf | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by rhyre):
Replying to [comment:5 felixxm]:
> It is probably a regression introduced by regression fix (see
https://github.com/django/django/commit/fc6528b25ab1834be1a478b405bf8f7ec5cf860c).
In 2.0.8 still not working. Haven't tried 2.1 yet. Too much to refactor to
use 2.1.
I'll try to create a demo when I'll have some time.
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: rhyre | Owner: nobody
Component: Utilities | Version: 2.0
Severity: Release blocker | Resolution:
Keywords: hash, tuple, list, | Triage Stage: Accepted
drf |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* type: Uncategorized => Bug
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
Crash can be reproduce at the `Node` level with `hash(Node([['a', 1],
['b', 2]]))`.
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:7>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Component: Utilities | Version: 2.0
Severity: Release blocker | Resolution:
Keywords: hash, tuple, list, | Triage Stage: Accepted
drf |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => assigned
* owner: nobody => felixxm
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:8>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Severity: Release blocker | Resolution:
Keywords: hash, tuple, list, | Triage Stage: Accepted
drf |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by rhyre):
Full traceback:
{{{
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/core/handlers/exception.py" in inner
35. response = get_response(request)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/core/handlers/base.py" in _get_response
128. response = self.process_exception_by_middleware(e,
request)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/core/handlers/base.py" in _get_response
126. response = wrapped_callback(request,
*callback_args, **callback_kwargs)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/views/decorators/csrf.py" in wrapped_view
54. return view_func(*args, **kwargs)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/rest_framework/viewsets.py" in view
95. return self.dispatch(request, *args, **kwargs)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/rest_framework/views.py" in dispatch
494. response = self.handle_exception(exc)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/rest_framework/views.py" in handle_exception
454. self.raise_uncaught_exception(exc)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/rest_framework/views.py" in dispatch
491. response = handler(request, *args, **kwargs)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/rest_framework/mixins.py" in list
40. queryset = self.filter_queryset(self.get_queryset())
File "/home/rhyre/-/testline_rest.py" in get_queryset
109. return LegacyTestlineService.list_testlines()
File "/home/rhyre/-/testline.py" in list_testlines
162. return cache_query(key='testline/list', func=query,
args=['id', 'name', 'status', 'add_date'], timeout=600)
File "/home/rhyre/-/cache.py" in cache_query
40. cache.set(key, dill.dumps(result), timeout)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/dill/_dill.py" in dumps
293. dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/dill/_dill.py" in dump
286. pik.dump(obj)
File "/usr/lib64/python3.6/pickle.py" in dump
409. self.save(obj)
File "/usr/lib64/python3.6/pickle.py" in save
496. rv = reduce(self.proto)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/query.py" in __getstate__
224. self._fetch_all()
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/query.py" in _fetch_all
1179. self._result_cache = list(self._iterable_class(self))
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/query.py" in __iter__
53. results =
compiler.execute_sql(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/sql/compiler.py" in execute_sql
1055. sql, params = self.as_sql()
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/sql/compiler.py" in as_sql
448. extra_select, order_by, group_by = self.pre_sql_setup()
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/sql/compiler.py" in pre_sql_setup
55. group_by = self.get_group_by(self.select + extra_select,
order_by)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/sql/compiler.py" in get_group_by
129. expressions = self.collapse_group_by(expressions,
having_group_by)
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/sql/compiler.py" in collapse_group_by
180. expr for expr in expressions if expr in pks or
getattr(expr, 'alias', None) not in aliases
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/sql/compiler.py" in <listcomp>
180. expr for expr in expressions if expr in pks or
getattr(expr, 'alias', None) not in aliases
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/expressions.py" in __hash__
390. for key, value in kwargs.items()
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/db/models/expressions.py" in __hash__
390. for key, value in kwargs.items()
File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-
packages/django/utils/tree.py" in __hash__
74. return hash((self.__class__, self.connector, self.negated) +
tuple(self.children))
Exception Type: TypeError at /testline/list/ajax
Exception Value: unhashable type: 'list'
Request information:
USER: AnonymousUser
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:9>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Severity: Release blocker | Resolution:
Keywords: hash, tuple, list, | Triage Stage: Accepted
drf |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/10275 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:10>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Severity: Release blocker | Resolution:
Keywords: hash, tuple, list, | Triage Stage: Ready for
drf | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:11>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Utilities | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: hash, tuple, list, | Triage Stage: Ready for
drf | checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"9fee229874367beafd532dad6d0f9ff9676ded0b" 9fee2298]:
{{{
#!CommitTicketReference repository=""
revision="9fee229874367beafd532dad6d0f9ff9676ded0b"
Fixed #29643 -- Fixed crash when combining Q objects with __in lookups and
lists.
Regression in fc6528b25ab1834be1a478b405bf8f7ec5cf860c.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:12>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Severity: Release blocker | Resolution: fixed
Keywords: hash, tuple, list, | Triage Stage: Ready for
drf | checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"2bf766cedca413826d599ac190fd6715429616f6" 2bf766ce]:
{{{
#!CommitTicketReference repository=""
revision="2bf766cedca413826d599ac190fd6715429616f6"
[2.1.x] Fixed #29643 -- Fixed crash when combining Q objects with __in
lookups and lists.
Regression in fc6528b25ab1834be1a478b405bf8f7ec5cf860c.
Backport of 9fee229874367beafd532dad6d0f9ff9676ded0b from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:13>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Severity: Release blocker | Resolution: fixed
Keywords: hash, tuple, list, | Triage Stage: Ready for
drf | checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
This didn't completely solve the issue, see #29838 for follow up.
--
Ticket URL: <https://code.djangoproject.com/ticket/29643#comment:14>