[Django] #25426: pickling SimpleLazyObject fails just after accessing related object of wrapped model instance.
Django
wrapped model instance.
--------------------------------------+--------------------
Reporter: iru | Owner: nobody
Type: Bug | Status: new
Component: Core (Serialization) | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+--------------------
Reproduction step :
{{{
import pickle
import django
django.setup()
from django.contrib.auth.models import User
from django.utils.functional import SimpleLazyObject
u = User.objects.select_related('profile').get(id=153)
o = SimpleLazyObject(lambda :u)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/usr/local/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/usr/local/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
File "/usr/local/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/local/lib/python2.7/pickle.py", line 661, in _batch_setitems
for k, v in items:
RuntimeError: dictionary changed size during iteration
pickle.dumps(o) # SUCCEEDS
}}}
This also fails :
{{{
u = User.objects.get(id=153)
o = SimpleLazyObject(lambda :u)
o.profile
pickle.dumps(o)
}}}
This looks to be related with change in Model.__reduce__ in django 1.8
adding DJANGO_VERSION_PICKLE_KEY in __dict__
{{{
def __reduce__(self):
"""
Provides pickling support. Normally, this just dispatches to
Python's
standard handling. However, for models with deferred field
loading, we
need to do things manually, as they're dynamically created classes
and
only module-level classes can be pickled by the default path.
"""
data = self.__dict__
data[DJANGO_VERSION_PICKLE_KEY] = get_version()
if not self._deferred:
class_id = self._meta.app_label, self._meta.object_name
return model_unpickle, (class_id, [], simple_class_factory),
data
defers = []
for field in self._meta.fields:
if isinstance(self.__class__.__dict__.get(field.attname),
DeferredAttribute):
defers.append(field.attname)
model = self._meta.proxy_for_model
class_id = model._meta.app_label, model._meta.object_name
return (model_unpickle, (class_id, defers,
deferred_class_factory), data)
}}}
This is quite hard to patch if we just pass
--
Ticket URL: <https://code.djangoproject.com/ticket/25426>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by iru):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Old description:
New description:
--
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:1>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Description changed by iru:
Old description:
New description:
Reproduction step :
{{{
import pickle
import django
django.setup()
from django.contrib.auth.models import User
from django.utils.functional import SimpleLazyObject
u = User.objects.select_related('profile').get(id=153)
o = SimpleLazyObject(lambda :u)
pickle.dumps(o) # FAILS
--
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:2>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by iru):
* severity: Normal => Release blocker
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:3>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by iru):
This is how this happens :
0. pickling lazy object starts.
1. lazy object is pickled without DJANGO_VERSION_PICKLE_KEY in wrapping
user object.
2. related object "profile" is pickled. It also has a reference to the
user object. ( profile.user, due to one-to-one relationship)
3. the user object(profile.user) is pickled again with same dict used at
step 1. DJANGO_VERSION_PICKLE_KEY is set.
4. pickling the user object fails due to dictionary changed size during
iteration.
5. pickling lazy object fails.
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:4>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by iru):
This happens always when cPickle is used. Does not happen when proto >=2
with python implementation of pickle.
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:5>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by iru):
* Attachment "patch_25426.diff" added.
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by iru):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:6>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by iru):
* Attachment "patch_25426.diff" added.
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by iru):
the patch is for saving django version info as unpickler's argument
instead of setting it as model instance's attribute which results in
RuntimeError depending on pickling protocol.
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:7>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Release blocker | Resolution:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by iru):
updated model regression test :
https://github.com/knifenomad/django/commit/7cb818b33a0f9b8ebb658661916cefa39d438109
test case for this issue :
https://github.com/knifenomad/django/commit/bf7f68f5897e5cf1b6ca2fa39bb6e901e6aaa53f
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:8>
Django
wrapped model instance.
Reporter: iru | Owner: nobody
Type: Bug | Status: new
Component: Core (Serialization) | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:9>
Django
wrapped model instance.
Reporter: iru | Owner: iru
Type: Bug | Status: assigned
Component: Core (Serialization) | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by iru):
* owner: nobody => iru
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:10>
Django
wrapped model instance.
Reporter: iru | Owner: iru
Type: Bug | Status: closed
Component: Core | Version: 1.8
(Serialization) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by iru):
* status: assigned => closed
* resolution: => duplicate
Comment:
closing since this duplicates https://code.djangoproject.com/ticket/25389
--
Ticket URL: <https://code.djangoproject.com/ticket/25426#comment:11>