[Django] #24007: Unable to unpickle models from an external script
Django
----------------------------------------+------------------------
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
----------------------------------------+------------------------
I'm using Python rq stack to delegate some tasks from my Django app to a
queued system. In some of those tasks, models are pickled and then
unpickled by rq. Unfortunately, with Django 1.7, I'm getting
`AppRegistryNotReady` errors:
{{{
Traceback (most recent call last):
File "/usr/local/bin/rqworker", line 9, in <module>
load_entry_point('rq==0.4.6', 'console_scripts', 'rqworker')()
File "/usr/local/lib/python2.7/dist-packages/rq/scripts/rqworker.py",
line 100, in main
w.work(burst=args.burst)
File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 358, in
work
self.execute_job(job)
File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 422, in
execute_job
self.main_work_horse(job)
File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 457, in
main_work_horse
success = self.perform_job(job)
File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 473, in
perform_job
job.func_name,
File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 226, in
func_name
self._unpickle_data()
File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 194, in
_unpickle_data
self._func_name, self._instance, self._args, self._kwargs =
unpickle(self.data)
File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 48, in
unpickle
obj = loads(pickled_string)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py",
line 1464, in model_unpickle
model = apps.get_model(*model_id)
File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py",
line 199, in get_model
self.check_models_ready()
File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py",
line 131, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: (AppRegistryNotReady("Models
aren't loaded yet.",), <function model_unpickle
}}}
As you can see in the stack trace, my code has no chance to fix this as it
is not called before the unpickling process.
One idea could be to check the app registry state in `model_unpickle` and
populate it if needed. Thoughts?
--
Ticket URL: <https://code.djangoproject.com/ticket/24007>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:1>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* has_patch: 0 => 1
Comment:
https://github.com/django/django/pull/3747
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:2>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
Oh, and I'd really like to backport this to 1.7 as it appears to be a
regression.
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:3>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by timgraham):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:4>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"108b8bf852c76855ed98f5abe55db1da845598e7"]:
{{{
#!CommitTicketReference repository=""
revision="108b8bf852c76855ed98f5abe55db1da845598e7"
Fixed #24007 -- Ensure apps registry's ready before unpickling models
This prevents AppRegistryNotReady errors when unpickling Django
models from an external script.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:5>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Claude Paroz <claude@…>):
In [changeset:"47912d9f2b8f55044aef5b8ed9b006e8dd8ce976"]:
{{{
#!CommitTicketReference repository=""
revision="47912d9f2b8f55044aef5b8ed9b006e8dd8ce976"
[1.7.x] Fixed #24007 -- Ensure apps registry's ready before unpickling
models
This prevents AppRegistryNotReady errors when unpickling Django
models from an external script.
Backport of 108b8bf85 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:6>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by collinanderson):
Regression in the test.
https://github.com/django/django/pull/3749
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:7>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham <timograham@…>):
In [changeset:"4fb38b73073dcd54f4af41a2af14dbc36f5d9f33"]:
{{{
#!CommitTicketReference repository=""
revision="4fb38b73073dcd54f4af41a2af14dbc36f5d9f33"
Allowed Python executable names other than python in test from refs
#24007.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:8>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: closed => new
* has_patch: 1 => 0
* resolution: fixed =>
* stage: Ready for checkin => Accepted
Comment:
Another problem with the test when running it outside the tests directory:
{{{
$ ./tests/runtests.py model_regress
Testing against Django installed in '/home/tim/code/django/django'
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.............s..Traceback (most recent call last):
File "/home/tim/code/django/tmpCkJ7D2.py", line 9, in <module>
article = pickle.loads(data)
File "/usr/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
value = func(*args)
File "/home/tim/code/django/django/db/models/base.py", line 1662, in
model_unpickle
apps.populate(settings.INSTALLED_APPS)
File "/home/tim/code/django/django/apps/registry.py", line 85, in
populate
app_config = AppConfig.create(entry)
File "/home/tim/code/django/django/apps/config.py", line 87, in create
module = import_module(entry)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in
import_module
__import__(name)
ImportError: No module named model_regress
F.
======================================================================
FAIL: test_unpickling_when_appregistrynotready
(model_regress.test_pickle.ModelPickleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tim/code/django/tests/model_regress/test_pickle.py", line
88, in test_unpickling_when_appregistrynotready
self.fail("Unable to reload model pickled data")
AssertionError: Unable to reload model pickled data
----------------------------------------------------------------------
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:9>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
One solution: https://github.com/django/django/pull/3758, but I may miss
some simpler resolution.
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:10>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:11>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Claude Paroz <claude@…>):
In [changeset:"1d9fc5caa947ff4ee72180185e91a9a145171712"]:
{{{
#!CommitTicketReference repository=""
revision="1d9fc5caa947ff4ee72180185e91a9a145171712"
Made model_regress unpickling test CWD-independent
Refs #24007. Thanks Tim Graham for his help with the patch.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:12>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"995be4a1375340a53668dd80444756d77302000d"]:
{{{
#!CommitTicketReference repository=""
revision="995be4a1375340a53668dd80444756d77302000d"
Fixed refs #24007 test on Windows.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:13>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"ea1865223886295674422923a2a9d38bacb0564d"]:
{{{
#!CommitTicketReference repository=""
revision="ea1865223886295674422923a2a9d38bacb0564d"
[1.7.x] Made model_regress unpickling test CWD-independent
Refs #24007. Thanks Tim Graham for his help with the patch.
Backport of 1d9fc5caa947ff4ee72180185e91a9a145171712 and
995be4a1375340a53668dd80444756d77302000d from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:14>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => fixed
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:15>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"5338ff4808c822a8b00e90154b884b7be3011e60"]:
{{{
#!CommitTicketReference repository=""
revision="5338ff4808c822a8b00e90154b884b7be3011e60"
Fixed #24160 -- Fixed model_regress test on Windows; refs #24007.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:16>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"cef3f805c21c029d017e6704565f9f2f5746a1e5"]:
{{{
#!CommitTicketReference repository=""
revision="cef3f805c21c029d017e6704565f9f2f5746a1e5"
[1.7.x] Fixed #24160 -- Fixed model_regress test on Windows; refs #24007.
Backport of 5338ff4808c822a8b00e90154b884b7be3011e60 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:17>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Ready for checkin => Accepted
Comment:
Reopening per [https://groups.google.com/d/topic/django-
developers/YvEzdQU4qUU/discussion discussion on django-dev] which suggests
this should be reverted.
[https://github.com/django/django/pull/6044 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:18>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: closed => new
* resolution: fixed =>
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:19>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"10a162809fa4de3a56bb7f2d3cb12b6b82a6f826" 10a16280]:
{{{
#!CommitTicketReference repository=""
revision="10a162809fa4de3a56bb7f2d3cb12b6b82a6f826"
Refs #24007 -- Removed an apps.populate() call in model unpickling that
can cause deadlocks.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:20>
Django
Reporter: claudep | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => wontfix
--
Ticket URL: <https://code.djangoproject.com/ticket/24007#comment:21>