[Django] #28117: loaddata raises ValueError with psycopg2 backend
Django
-------------------------------------+-------------------------------------
Reporter: Nicolas | Owner: nobody
Kuttler |
Type: | Status: new
Cleanup/optimization |
Component: Core | Version: 1.11
(Management commands) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I have a site that currently uses mysql. The `admin.logentry` table
contains `object_repr` values from spam submissions like
`\u589f0\u0005\u0001\u0000\u0000: Nice article`
While migrating the site to postgres importing fixtures created with
dumpdata raises a `ValueError` originating in (I think)
https://github.com/psycopg/psycopg2/blob/2_7_1/psycopg/utils.c#L58 and not
handled in
https://github.com/django/django/blob/stable/1.11.x/django/core/management/commands/loaddata.py#L181
One possible fix, to give users a more informative error message, would be
to catch ValueErrors as well on that line.
{{{
Traceback (most recent call last):
File "./src/manage.py", line 8, in <module>
execute_from_command_line(sys.argv)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/management/__init__.py", line 367, in
execute_from_command_line
utility.execute()
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/management/commands/loaddata.py", line 64, in
handle
self.loaddata(fixture_labels)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/management/commands/loaddata.py", line 104, in
loaddata
self.load_label(fixture_label)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/management/commands/loaddata.py", line 167, in
load_label
obj.save(using=self.using)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/core/serializers/base.py", line 201, in save
models.Model.save_base(self.object, using=using, raw=True, **kwargs)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/db/models/base.py", line 824, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/db/models/base.py", line 889, in _save_table
forced_update)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/db/models/base.py", line 939, in _do_update
return filtered._update(values) > 0
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/db/models/query.py", line 654, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/db/models/sql/compiler.py", line 1148, in
execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
cursor.execute(sql, params)
File
"/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7
/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
ValueError: Problem installing fixture '/srv/www/project/data.json': A
string literal cannot contain NUL (0x00) characters.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28117>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: | Status: closed
Cleanup/optimization |
Component: Core (Management | Version: 1.11
commands) |
Severity: Normal | Resolution: duplicate
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => duplicate
Comment:
As discussed in #28201, the solution might be to have `CharField` and
`TextField` remove null bytes from their values. I'm tentatively closed
this as a duplicate but will reopen if the solution to that ticket doesn't
address this.
--
Ticket URL: <https://code.djangoproject.com/ticket/28117#comment:1>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Core (Management | Version: 1.11
commands) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: closed => new
* resolution: duplicate =>
* stage: Unreviewed => Accepted
Comment:
The [https://groups.google.com/d/topic/django-
developers/D1gvXYCezEc/discussion django-developers discussion] yielded a
consensus to go with a fix in the direction of what's suggested in the
ticket description.
--
Ticket URL: <https://code.djangoproject.com/ticket/28117#comment:2>
Django
-------------------------------------+-------------------------------------
Type: | Reddy Thatiparthy
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: 1.11
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Srinivas Reddy Thatiparthy
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/28117#comment:3>
Django
NUL characters in data
-------------------------------------+-------------------------------------
Reporter: Nicolas Kuttler | Owner: Srinivas
Type: | Reddy Thatiparthy
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: 1.11
commands) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
* stage: Accepted => Ready for checkin
Comment:
[https://github.com/django/django/pull/9500 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/28117#comment:4>
Django
NUL characters in data
-------------------------------------+-------------------------------------
Reporter: Nicolas Kuttler | Owner: Srinivas
Type: | Reddy Thatiparthy
Component: Core (Management | Version: 1.11
commands) |
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"58ec55b157b3c4cc9dc0a944804f8a719ff4e12f" 58ec55b1]:
{{{
#!CommitTicketReference repository=""
revision="58ec55b157b3c4cc9dc0a944804f8a719ff4e12f"
Fixed #28117 -- Added a helpful message in loaddata when psycopg2 can't
load a fixture due to NUL characters.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28117#comment:5>