[Django] #19820: Make loaddata error messages less cryptic
Django
--------------------------------------+--------------------
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+--------------------
At present, error messages when running loaddata don't include any clue
about the part of the serialised data which caused the problem.
In the Deserialiser function (in core.serializers.json.py) there is a
try/except block on line 40 to 47 which could be amended to aid the user:
{{{
def Deserializer(stream_or_string, **options):
"""
Deserialize a stream or string of JSON data.
"""
if isinstance(stream_or_string, basestring):
stream = StringIO(stream_or_string)
else:
stream = stream_or_string
try:
for obj in PythonDeserializer(simplejson.load(stream), **options):
yield obj
except GeneratorExit:
raise
except Exception, e:
# Map to deserializer error
raise DeserializationError(e)
}}}
If raise DeserializationError(e) were changed to raise
DeserializationError(e, obj) then the function would return a tuple
containing the current error message (which isn't that informative) and
the object which causes the error.
This has been a big issue for me recently in migrating a pilot
installation which used sqlite over to postgres. There were a number of
data integrity issues which using postgres highlighted, but it was only by
applying this change that it became possible to identify the records which
were causing the problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: Core | Resolution:
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by claudep):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
Better error messages are always welcome. However, you mention integrity
issues, while the fix you suggest is at deserialization stage. Maybe some
actual example could help here?
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:1>
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Core | Resolution: needsinfo
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => needsinfo
Comment:
Closing needsinfo pending clarification of whether the better error
messages are really needed at deserialization or database-insert, and an
example clearly demonstrating the problem that the proposed fix would
ameliorate.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:2>
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Core | Resolution: needsinfo
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Naddiseo):
If there is, for example, an initialiser for `auth.Permission` with a
content_type that does not exist, the DeserializationError is raised:
django.core.serializers.base.DeserializationError: Problem installing
fixture 'testapp/initial_data/testing_data.json': ContentType matching
query does not exist.
This is a problem with big files, since it doesn't tell you where in the
file the error occurred. (Reproduced on 1.7c3)
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:3>
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Core | Resolution: needsinfo
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: reames@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:4>
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Core | Resolution: needsinfo
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Naddiseo):
Proposed fix: https://github.com/django/django/pull/3098
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:5>
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: Core | Resolution:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Needs tests: 1 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by claudep):
* status: closed => new
* has_patch: 0 => 1
* resolution: needsinfo =>
* needs_tests: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:6>
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: Core | Resolution:
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 1 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Naddiseo):
I've add a simple patch which tests that an error message is raised, and
that the message contains the offending model name, it's PK, and what the
offending field's value is.
Let me know if there's anything else I can add.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:7>
Django
Reporter: bigfudge | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: Core | Resolution:
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by timgraham):
* needs_tests: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:8>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:9>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
Comment:
I left comments for improvement on the PR. Please uncheck "Patch needs
improvement" when you update it, thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:10>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Naddiseo):
* needs_better_patch: 1 => 0
Comment:
New PR https://github.com/django/django/pull/3223
I left some questions I need some feedback on.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:11>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
Comment:
Patch needs a rebase.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:12>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Naddiseo):
Rebased, and responded to question.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:13>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 1 => 0
Comment:
Don't forget to uncheck "Patch needs improvement" so the ticket appears in
the review queue.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:14>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
Comment:
Tests need to fixed on Python 3.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:15>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Naddiseo):
* needs_better_patch: 1 => 0
Comment:
Fixed python 3 error: it was due to python3 "fixing" variables escaping
generator expressions.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:16>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
Comment:
Left some more comments.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:17>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Naddiseo):
* needs_better_patch: 1 => 0
Comment:
Rebased and updated patch.
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:18>
Django
Reporter: bigfudge | Owner: nobody
Type: Cleanup/optimization | Status: closed
Component: Core (Serialization) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"727e40c879f893a2c336e396aafdcad60b5d224c" 727e40c8]:
{{{
#!CommitTicketReference repository=""
revision="727e40c879f893a2c336e396aafdcad60b5d224c"
Fixed #19820 -- Added more helpful error messages to Python deserializer.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/19820#comment:19>