[Django] #31926: missing FROM-clause entry when unpickling queries with FilteredRelation
Django
-------------------------------------+-------------------------------------
Reporter: Beda | Owner: nobody
Kosata |
Type: Bug | Status: new
Component: Database | Version: 2.2
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I am pickling query objects (queryset.query) for later re-evaluation as
per https://docs.djangoproject.com/en/2.2/ref/models/querysets/#pickling-
querysets. However, when I tried to rerun a query that contains a {{{
FilteredRelation }}} inside a filter, I get an {{{
psycopg2.errors.UndefinedTable: missing FROM-clause entry for table "t3"
}}} error.
I created a minimum reproducible example.
**models.py**
{{{
from django.db import models
class Publication(models.Model):
title = models.CharField(max_length=64)
class Session(models.Model):
TYPE_CHOICES = (('A', 'A'), ('B', 'B'))
publication = models.ForeignKey(Publication, on_delete=models.CASCADE)
session_type = models.CharField(choices=TYPE_CHOICES, default='A',
max_length=1)
place = models.CharField(max_length=16)
value = models.PositiveIntegerField(default=1)
}}}
**The actual code to cause the crash**:
{{{
import pickle
from django.db.models import FilteredRelation, Q, Sum
from django_error.models import Publication, Session
p1 = Publication.objects.create(title='Foo')
p2 = Publication.objects.create(title='Bar')
Session.objects.create(publication=p1, session_type='A', place='X',
value=1)
Session.objects.create(publication=p1, session_type='B', place='X',
value=2)
Session.objects.create(publication=p2, session_type='A', place='X',
value=4)
Session.objects.create(publication=p2, session_type='B', place='X',
value=8)
Session.objects.create(publication=p1, session_type='A', place='Y',
value=1)
Session.objects.create(publication=p1, session_type='B', place='Y',
value=2)
Session.objects.create(publication=p2, session_type='A', place='Y',
value=4)
Session.objects.create(publication=p2, session_type='B', place='Y',
value=8)
qs = Publication.objects.all().annotate(
relevant_sessions=FilteredRelation('session',
condition=Q(session__session_type='A'))
).annotate(x=Sum('relevant_sessions__value'))
# just print it out to make sure the query works
print(list(qs))
qs2 = Publication.objects.all()
qs2.query = pickle.loads(pickle.dumps(qs.query))
# the following crashes with an error
# psycopg2.errors.UndefinedTable: missing FROM-clause entry for table
"t3"
# LINE 1: ...n"."id" = relevant_sessions."publication_id" AND
(T3."sessio...
print(list(qs2))
}}}
In the crashing query, there seems to be a difference in the {{{ table_map
}}} attribute - this is probably where the {{{ t3 }}} table is coming
from.
Please let me know if there is any more info required for hunting this
down.
Cheers
Beda
p.s.- I also tried in Django 3.1 and the behavior is the same.
p.p.s.- just to make sure, I am not interested in ideas on how to rewrite
the query - the above is a very simplified version of what I use, so it
would probably not be applicable anyway.
--
Ticket URL: <https://code.djangoproject.com/ticket/31926>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
used in aggregation.
-------------------------------------+-------------------------------------
Reporter: Beda Kosata | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
Thanks for this ticket, I was able to reproduce this issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/31926#comment:1>
Django
used in aggregation.
-------------------------------------+-------------------------------------
Reporter: Beda Kosata | Owner: nobody
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "test-31926.diff" added.
Tests.
Django
used in aggregation.
-------------------------------------+-------------------------------------
Reporter: Beda Kosata | Owner: nobody
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Beda Kosata):
Just a note, the failing queryset does not have to be constructed using
the
{{{
qs2 = Publication.objects.all()
qs2.query = pickle.loads(pickle.dumps(qs.query))
}}}
The same problem occurs even if the whole queryset is pickled and
unpickled and then a copy is created by calling `.all()`.
{{{
qs2 = pickle.loads(pickle.dumps(qs)).all()
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31926#comment:2>
Django
used in aggregation.
-------------------------------------+-------------------------------------
| Wobrock
Type: Bug | Status: assigned
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: David Wobrock (added)
* owner: nobody => David Wobrock
* has_patch: 0 => 1
* status: new => assigned
Comment:
Hi,
I started from the test Mariusz wrote, and followed the code down to where
it diverged, comparing the pickling case to the expected QuerySet.
Details can be found in the PR:
https://github.com/django/django/pull/13484
--
Ticket URL: <https://code.djangoproject.com/ticket/31926#comment:3>
Django
used in aggregation.
-------------------------------------+-------------------------------------
Reporter: Beda Kosata | Owner: David
| Wobrock
Type: Bug | Status: assigned
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
Comment:
Left some comments on the PR regarding the `__hash__` implementations but
it looks like David identified the underlying issue appropriately.
--
Ticket URL: <https://code.djangoproject.com/ticket/31926#comment:4>
Django
used in aggregation.
-------------------------------------+-------------------------------------
Reporter: Beda Kosata | Owner: David
| Wobrock
Type: Bug | Status: assigned
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
Comment:
Integrated the suggested changes, thanks!
I'm wondering if the patch should be backported to 2.2 and 3.0 - I guess
I'll let you consider and handle this.
--
Ticket URL: <https://code.djangoproject.com/ticket/31926#comment:5>
Django
used in aggregation.
-------------------------------------+-------------------------------------
Reporter: Beda Kosata | Owner: David
| Wobrock
Component: Database layer | Version: 2.2
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"0ef04fdd7ab75daa59536261bbfc5da4c4e31079" 0ef04fd]:
{{{
#!CommitTicketReference repository=""
revision="0ef04fdd7ab75daa59536261bbfc5da4c4e31079"
Refs #31926 -- Fixed reverse related identity crash on Q()
limit_choices_to.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31926#comment:7>