[Django] #20572: locale fr_FR.UTF-8 breaks loaddata
Django
--------------------------------+--------------------
Reporter: gerard.henry@… | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: 1.5
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------+--------------------
Doing loaddata:
{{{
python manage.py loaddata -v 3 --traceback /tmp/mc1.json
...
Traceback (most recent call last):
File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-
packages/django/core/management/base.py", line 222, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-
packages/django/core/management/base.py", line 255, in execute
output = self.handle(*args, **options)
File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-
packages/django/core/management/commands/loaddata.py", line 226, in handle
e.args = ("Problem installing fixtures: %s" % e,)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 126:
ordinal not in range(128)
}}}
changing line 226 in loaddata.py (replace %s by %r), i see the error:
{{{
IntegrityError: Problem installing fixtures: IntegrityError('insert or
update on table "eventsmgmt_mycalendar" violates foreign key constraint
"eventsmgmt_mycalendar_created_by_id_fkey"\nD\xc3\x89TAIL : Key
(created_by_id)=(46) is not present in table "auth_user".\n',)
}}}
in the terminal, the locale is:
{{{
$localeLANG="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_CTYPE="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_ALL=
}}}
the workaround is to change the locale to en_US:
{{{
$ export LANG=en_US.UTF-8
}}}
and it works
--
Ticket URL: <https://code.djangoproject.com/ticket/20572>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: gerard.henry@… | Owner: nobody
Type: Bug | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by claudep):
* needs_better_patch: => 0
* needs_docs: => 0
* component: Uncategorized => Core (Management commands)
* needs_tests: => 0
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:1>
Django
Reporter: gerard.henry@… | Owner: nobody
Type: Bug | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
Note that I can currently reproduce the error with
`fixtures_regress.tests.TestFixtures.test_loaddata_raises_error_when_fixture_has_invalid_foreign_key`
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:2>
Django
Reporter: gerard.henry@… | Owner: nobody
Type: Bug | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
Also note that this is a Python2-only issue.
Some possible solutions:
* using `%r` instead `%s`. Drawback: the resulting message is not quite
the same:
{{{
e.args = ("Problem installing fixtures: %s" % e,)
(u"Problem installing fixtures: The row in table 'fixtures_regress_book'
with primary key '1' has an invalid
foreign key: fixtures_regress_book.author_id contains a value '3' that
does not have a corresponding value in
fixtures_regress_person.id.",)
e.args = ("Problem installing fixtures: %r" % e,)
(u'Problem installing fixtures: IntegrityError(u"The row in table
\'fixtures_regress_book\' with primary key
\'1\' has an invalid foreign key: fixtures_regress_book.author_id contains
a value \'3\' that does not have a
corresponding value in fixtures_regress_person.id.",)',)
}}}
And we should still track this pattern all over Django code.
* using `force_text` in `DatabaseErrorWrapper` to convert message to
unicode.[[BR]]
Drawback: non-ascii unicode args for exceptions are not well supported,
notably in unittest2 `assertRaisesRegexp` where `str(exc_value)` will
raise a `UnicodeEncodeError`.
* using `force_text(s, 'ascii', errors='ignore')` in
`DatabaseErrorWrapper` to strip out any non-ascii char from the exception
message.[[BR]]
Drawback: the error message will be altered.
Opinions or other solutions welcome.
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:3>
Django
Reporter: gerard.henry@… | Owner: nobody
Type: Uncategorized | Status: closed
Component: contrib.admin | Version: master
Severity: Normal | Resolution: fixed
Keywords: 1 | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 1
--------------------------------+--------------------------------------
Changes (by ogpcludi <sample@…>):
* status: new => closed
* needs_better_patch: 0 => 1
* component: Core (Management commands) => contrib.admin
* needs_tests: 0 => 1
* version: 1.5 => master
* easy: 0 => 1
* type: Bug => Uncategorized
* keywords: => 1
* needs_docs: 0 => 1
* has_patch: 0 => 1
* ui_ux: 0 => 1
* resolution: => fixed
* stage: Accepted => Unreviewed
Comment:
1
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:5>
Django
Reporter: gerard.henry@… | Owner: nobody
Type: Uncategorized | Status: new
Component: contrib.admin | Version: master
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 1
--------------------------------+--------------------------------------
Changes (by ogpcludi <sample@…>):
* needs_better_patch: 0 => 1
* component: Core (Management commands) => contrib.admin
* needs_tests: 0 => 1
* version: 1.5 => master
* easy: 0 => 1
* keywords: => 1
* needs_docs: 0 => 1
* has_patch: 0 => 1
* ui_ux: 0 => 1
* type: Bug => Uncategorized
* stage: Accepted => Unreviewed
Comment:
1
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:4>
Django
Reporter: gerard.henry@… | Owner: anonymous
Type: Uncategorized | Status: assigned
Component: contrib.admin | Version: master
Severity: Normal | Resolution:
Keywords: 1 | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 1
--------------------------------+--------------------------------------
Changes (by ogpcludi <sample@…>):
* status: new => assigned
* needs_better_patch: 0 => 1
* component: Core (Management commands) => contrib.admin
* needs_tests: 0 => 1
* keywords: => 1
* version: 1.5 => master
* easy: 0 => 1
* owner: nobody => anonymous
* needs_docs: 0 => 1
* has_patch: 0 => 1
* ui_ux: 0 => 1
* type: Bug => Uncategorized
* stage: Accepted => Unreviewed
Comment:
1
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:6>
Django
Reporter: gerard.henry@… | Owner: nobody
Type: Uncategorized | Status: new
Component: contrib.admin | Version: master
Severity: Normal | Resolution:
Keywords: 1 | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 1
--------------------------------+--------------------------------------
Changes (by ogpcludi <sample@…>):
* keywords: => 1
* type: Bug => Uncategorized
* version: 1.5 => master
* component: Core (Management commands) => contrib.admin
Comment:
1
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: gerard.henry@… | Owner: nobody
Type: Bug | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:4>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by EmilStenstrom):
There are lots of other cases where printing error messages in ascii could
lead to problems. I'm not sure if there are more bugs covering this, so
I'll add a small description here:
I have a model who's __unicode__() method returns:
{{{
<MailComment: \n\nK\xc3\xb6r jobbet nu och det verkar bli fel...>
}}}
When loading my fixture I get: ValueError('Cannot assign [unicode
representation of MailComment here]') which get propagated to fixture
loading and makes that crash with:
{{{
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32:
ordinal not in range(128)
}}}
Using %r instead of %s seems reasonable, as it doesn't remove any of the
offending characters (they are often the reason for the exception in the
first place), and it will likely work well with all sorts of environments.
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:5>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
#22882 reports a similar issue with the `flush` command.
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:6>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by timgraham):
Can we add `from __future__ import unicode_literals` to `dumpdata.py`?
This strategy seems to fixed the issue for `flush`.
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:7>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
commands) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
#25719 was a duplicate.
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:8>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Core (Management | Version: 1.5
commands) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* status: new => closed
* resolution: => wontfix
Comment:
Django master is now Python 3 only, therefore I'll close this ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/20572#comment:9>