[Django] #20257: QuerySet that prefetches related object with a ManyToMany field cannot be pickled.

Django

unread,

Apr 13, 2013, 11:15:51 PM4/13/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.
-------------------------------------+--------------------------------
Reporter: bryced | Owner: nobody
Type: Bug | Status: new
Component: Core (Cache system) | Version: 1.5
Severity: Release blocker | Keywords: ORM, pickle, cache
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+--------------------------------
After upgrading from 1.4 to 1.5, exceptions were thrown when trying to
pickle certain querysets. This means that the caching framework doesn't
work for these querysets.

In 1.4 the following code runs fine. In 1.5 this error occurs:
{{{
PicklingError: Can't pickle <class 'people.models.SocialProfile_friends'>:
it's not found as people.models.SocialProfile_friends
}}}

models.py
{{{
from django.db import models

class Person(models.Model):
name = models.CharField(max_length=200)

class SocialProfile(models.Model):
person = models.ForeignKey(Person)
friends = models.ManyToManyField('self')

}}}

tests.py
{{{
from django.test import TestCase
from people.models import Person
import pickle

class SimpleTest(TestCase):

def test_pickle_failure(self):
bob = Person(name="Bob")
bob.save()

people =
Person.objects.all().prefetch_related('socialprofile_set')
pickle.dumps(people)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/20257>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Apr 15, 2013, 12:24:17 PM4/15/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.

-------------------------------------+-------------------------------------

Reporter: bryced | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 1.5
(models, ORM) | Resolution:
Severity: Release blocker | Triage Stage: Accepted
Keywords: ORM, pickle, cache | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by bmispelon):

* cc: bmispelon@… (added)
* needs_better_patch: => 0
* component: Core (Cache system) => Database layer (models, ORM)
* needs_tests: => 0
* needs_docs: => 0
* stage: Unreviewed => Accepted

Comment:

Hi,

Thanks for the detailed report.

I can reproduce the issue in the `stable/1.5.x` branch but it appears to
have been fixed in `master`.

The testcase works on the `stable/1.4.x` branch so it is a regression.

Using `git bisect`, I found that the problem was introduced by commit
056ace0f395a58eeac03da9f9ee7e3872e1e407b.

--
Ticket URL: <https://code.djangoproject.com/ticket/20257#comment:1>

Django

unread,

Apr 15, 2013, 1:31:24 PM4/15/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.

-------------------------------------+-------------------------------------

Reporter: bryced | Owner: nobody
Type: Bug | Status: new

Comment (by akaariai):

The problem is that fields for automatic models (for example the
intermediary m2m table) can't be pickled in 1.5. In master this is fixed
by introducing Field.reduce. In 1.4 the field wasn't used directly, only
the field.name was used so there was no problems. So, to fix this, one of
the following three things needs to be done:
- alter `QuerySet.__getstate__` and `__setstate__` to do something
similar that sql.query:Query's state methods do. That is, store the
field's name in getstate and restore back the real field instance in
setstate.
- remove direct storage of the field in QuerySet
- backpatch Field.reduce changes from master

I think the Field.reduce changes are too risky to backpatch, and I assume
there was a good reason to store the field directly in QuerySet instead of
field.name. So, changes to getstate and setstate seem like the best choice
to me.

--
Ticket URL: <https://code.djangoproject.com/ticket/20257#comment:2>

Django

unread,

Apr 24, 2013, 12:22:20 PM4/24/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.

-------------------------------------+-------------------------------------

Reporter: bryced | Owner: nobody
Type: Bug | Status: new

Comment (by jamesmfriedman@…):

This bug is duplicate of #20157. I've been tracking on it since it is
causing me some serious headaches in production since upgrading to 1.5. I
do hope for a patch in 1.5.2.

--
Ticket URL: <https://code.djangoproject.com/ticket/20257#comment:3>

Django

unread,

May 21, 2013, 4:44:41 AM5/21/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.

-------------------------------------+-------------------------------------
Reporter: bryced | Owner: nobody
Type: Bug | Status: closed

Component: Database layer | Version: 1.5

(models, ORM) | Resolution: fixed

Severity: Release blocker | Triage Stage: Accepted
Keywords: ORM, pickle, cache | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------

Changes (by Anssi Kääriäinen <akaariai@…>):

* status: new => closed
* resolution: => fixed

Comment:

In [changeset:"bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5"]:
{{{
#!CommitTicketReference repository=""
revision="bac187c0d8e829fb3ca2ca82965eabbcbcb6ddd5"
[1.5.x] Fixed prefetch_related + pickle regressions

There were a couple of regressions related to field pickling. The
regressions were introduced by QuerySet._known_related_objects caching.

The regressions aren't present in master, the fix was likely in
f403653cf146384946e5c879ad2a351768ebc226.

Fixed #20157, fixed #20257. Also made QuerySets with model=None
picklable.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/20257#comment:4>

Django

unread,

May 30, 2013, 9:05:15 AM5/30/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.

-------------------------------------+-------------------------------------
Reporter: bryced | Owner: nobody

Comment (by michaelmior):

This is currently blocking our team from upgrading to Django 1.5. Just
wondering if this will be included in the 1.5.2 release? Thanks for the
fix!

--
Ticket URL: <https://code.djangoproject.com/ticket/20257#comment:5>

Django

unread,

May 30, 2013, 10:10:19 AM5/30/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.

-------------------------------------+-------------------------------------
Reporter: bryced | Owner: nobody

Comment (by timo):

@michaelmior, Yes it will be. You can see the commit above is prefixed
with [1.5.x] indicating it was committed to the stable/1.5.x branch.

--
Ticket URL: <https://code.djangoproject.com/ticket/20257#comment:6>

Django

unread,

May 30, 2013, 1:19:49 PM5/30/13

to django-...@googlegroups.com

#20257: QuerySet that prefetches related object with a ManyToMany field cannot be
pickled.

-------------------------------------+-------------------------------------
Reporter: bryced | Owner: nobody

Comment (by michaelmior):

Thanks @timo! I noticed that after. I apologize if it's bad form to ask,
but any ETA on the 1.5.2 release? I'm happy to pitch in if there are
release blockers that need to be resolved.

--
Ticket URL: <https://code.djangoproject.com/ticket/20257#comment:7>

Reply all

Reply to author

Forward