[Django] #25677: compilemessages throws an exception and does not report msgformat errors correctly
Django
correctly
--------------------------------------+--------------------
Reporter: gavinwahl | Owner: nobody
Type: Uncategorized | Status: new
Component: Internationalization | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+--------------------
I have a django.po with errors. When I run compilemessages, instead of
getting a sensible error, I get a UnicodeDecodeError.
{{{
Traceback (most recent call last):
File "/usr/lib/python3.4/pdb.py", line 1661, in main
pdb._runscript(mainpyfile)
File "/usr/lib/python3.4/pdb.py", line 1542, in _runscript
self.run(statement)
File "/usr/lib/python3.4/bdb.py", line 431, in run
exec(cmd, globals, locals)
File "<string>", line 1, in <module>
File "manage.py", line 2, in <module>
import os
File "django/core/management/__init__.py", line 350, in
execute_from_command_line
utility.execute()
File "django/core/management/__init__.py", line 342, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "django/core/management/commands/compilemessages.py", line 98, in
handle
self.compile_messages(locations)
File "django/core/management/commands/compilemessages.py", line 122, in
compile_messages
output, errors, status = popen_wrapper(args)
File "django/core/management/utils.py", line 27, in popen_wrapper
output, errors = p.communicate()
File "/usr/lib/python3.4/subprocess.py", line 962, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.4/subprocess.py", line 1664, in _communicate
self.stderr.encoding)
File "/usr/lib/python3.4/subprocess.py", line 888, in
_translate_newlines
data = data.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc5 in position 179:
invalid continuation byte
}}}
I should get the error from msgformat:
{{{
$ msgfmt --check-format -o locale/es/LC_MESSAGES/django.mo
locale/es/LC_MESSAGES/django.po
locale/es/LC_MESSAGES/django.po:112: 'msgstr' is not a valid Python brace
format string, unlike 'msgid'. Reason: In the directive number 0, '�'
cannot start a field name.
msgfmt: found 1 fatal error
}}}
The problem is that the output of msgformat is _not_ a utf-8 string. It's
bytes. Any attempt to decode it into unicode is futile.
The exact output of msgformat is b"locale/es/LC_MESSAGES/django.po:112:
'msgstr' is not a valid Python brace format string, unlike 'msgid'.
Reason: In the directive number 0, '\xc5' cannot start a field
name.\nmsgfmt: found 1 fatal error\n".
--
Ticket URL: <https://code.djangoproject.com/ticket/25677>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
correctly
Reporter: gavinwahl | Owner: nobody
Type: Bug | Status: new
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* needs_better_patch: => 0
* needs_tests: => 0
* version: 1.8 => master
* needs_docs: => 0
* type: Uncategorized => Bug
* stage: Unreviewed => Accepted
Comment:
Could you please provide the faulty msgid/msgstr that caused the error?
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:1>
Django
correctly
Reporter: gavinwahl | Owner: nobody
Type: Bug | Status: new
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by gavinwahl):
* Attachment "django.po" added.
Django
correctly
Reporter: gavinwahl | Owner: nobody
Type: Bug | Status: new
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
It's interesting to look at the traceback on Python 3:
{{{
...
File "/home/claude/virtualenvs/djangogit34/lib/python3.4/site-
packages/django/core/management/commands/compilemessages.py", line 97, in
handle
self.compile_messages(locations)
File "/home/claude/virtualenvs/djangogit34/lib/python3.4/site-
packages/django/core/management/commands/compilemessages.py", line 121, in
compile_messages
output, errors, status = popen_wrapper(args)
File "/home/claude/virtualenvs/djangogit34/lib/python3.4/site-
packages/django/core/management/utils.py", line 27, in popen_wrapper
output, errors = p.communicate()
File "/usr/lib/python3.4/subprocess.py", line 960, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.4/subprocess.py", line 1662, in _communicate
self.stderr.encoding)
File "/usr/lib/python3.4/subprocess.py", line 888, in
_translate_newlines
data = data.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc5 in position 214:
invalid continuation byte
}}}
We see here that decoding (and error) happens in the standard lib.
Should we blame gettext which outputs non-utf-8-decodable characters?
Should we blame Python by not falling back to some human-readable
representation of the offending byte? Can Django do something to prevent
that?
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:2>
Django
correctly
Reporter: gavinwahl | Owner: nobody
Type: Bug | Status: new
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by gavinwahl):
The error is in the standard lib because it was passed
universal_newlines=True, which works on unicode. Popen will work in bytes
otherwise.
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:3>
Django
correctly
Reporter: gavinwahl | Owner: nobody
Type: Bug | Status: new
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/5547 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:4>
Django
correctly
Reporter: gavinwahl | Owner: jgeskens
Type: Bug | Status: assigned
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by jgeskens):
* owner: nobody => jgeskens
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:5>
Django
correctly
Reporter: gavinwahl | Owner:
Type: Bug | Status: new
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by jgeskens):
* owner: jgeskens =>
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:6>
Django
correctly
Reporter: gavinwahl | Owner:
Type: Bug | Status: new
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:7>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: closed
Component: | Version: master
Internationalization |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz <claude@…>):
* status: new => closed
* owner: => Claude Paroz <claude@…>
* resolution: => fixed
Comment:
In [changeset:"fa08d27fb714534670b431fde0cd04a17d637585" fa08d27f]:
{{{
#!CommitTicketReference repository=""
revision="fa08d27fb714534670b431fde0cd04a17d637585"
Fixed #25677 -- Prevented decoding errors in/after Popen calls
Thanks Gavin Wahl for the report and Tim Graham for the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:8>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: closed
Component: | Version: master
Internationalization |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"db8763fb239137e9f750d26df221483a8ae5c97c" db8763fb]:
{{{
#!CommitTicketReference repository=""
revision="db8763fb239137e9f750d26df221483a8ae5c97c"
Refs #25677 -- Fixed Python 2 i18n test failure on non-ASCII path.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:9>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: closed => new
* has_patch: 1 => 0
* resolution: fixed =>
* stage: Ready for checkin => Accepted
Comment:
Oops, I haven't installed gettext on the Windows CI machine so the tests
aren't running there. Here's a sample error from my local machine which
does have gettext installed:
{{{
======================================================================
ERROR: test_no_wrap_enabled (i18n.test_extraction.NoWrapExtractorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "c:\Users\Tim\code\django\tests\i18n\test_extraction.py", line 597,
in te
st_no_wrap_enabled
File "c:\users\tim\code\django\django\core\management\__init__.py", line
117,
in call_command
return command.execute(*args, **defaults)
File "c:\users\tim\code\django\django\core\management\base.py", line
341, in e
xecute
output = self.handle(*args, **options)
File
"c:\users\tim\code\django\django\core\management\commands\makemessages.py
", line 307, in handle
self.write_po_file(potfile, locale)
File
"c:\users\tim\code\django\django\core\management\commands\makemessages.py
", line 539, in write_po_file
"errors happened while running msgmerge\n%s" % errors)
django.core.management.base.CommandError: errors happened while running
msgmerge
c:\Users\Tim\code\django\tests\i18n\commands\locale\de\LC_MESSAGES\django.po:2:4
7: syntax error
msgmerge: found 1 fatal error
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:10>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Claude, any ideas? I can add gettext on the Windows CI machine if you want
to try a pull request with some solution (of course, we'd see the test
failures for other contributors too in the meantime).
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:11>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
I have no Windows system to test with, and don't like blind debugging. If
Windows users care for Django, let them step in.
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:12>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* cc: ramiro (added)
Comment:
Ramiro, maybe you could help with this issue at your convenience (before
Django 1.10)?
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:13>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Claude, the new test is also failing on Ubuntu 12.04 and Python 3
(possibly due to msgfmt 0.18.1 there vs 0.18.3 on 14.04?). Also seen this
on my Windows setup (again, Python 3 only) which uses msgfmt 0.17.
{{{
Traceback (most recent call last):
File "/mnt/jenkinsdata/workspace/django-
master/database/sqlite3/python/python3.4/tests/i18n/test_compilation.py",
line 177, in test_msgfmt_error_including_non_ascii
call_command('compilemessages', locale=['ko'], verbosity=0)
File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
next(self.gen)
File "/mnt/jenkinsdata/workspace/django-
master/database/sqlite3/python/python3.4/django/test/testcases.py", line
586, in _assert_raises_message_cm
yield cm
AssertionError: CommandError not raised
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:14>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
Are you able to install a more recent gettext on Windows to see if it's
gettext-related?
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:15>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
I was able to reproduce that on Linux with gettext 0.18.1. I guess that
python-brace-format is a "recent" gettext addition. A solution might be to
simply skip `test_msgfmt_error_including_non_ascii` with older gettext
versions. Not sure if that fixes all failures.
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:16>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
[https://github.com/django/django/pull/5772 Skipping the test] for older
versions of gettext seems to work. This doesn't solve the other issues on
Windows.
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:17>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"93be2f7dea51f7a063d9114660f5f1028a849cf0" 93be2f7]:
{{{
#!CommitTicketReference repository=""
revision="93be2f7dea51f7a063d9114660f5f1028a849cf0"
Refs #25677 -- Skipped an i18n test on older gettext versions.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:18>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* keywords: => 1.10
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:19>
Django
correctly
Reporter: gavinwahl | Owner: Claude
| Paroz <claude@…>
Type: Bug | Status: new
Component: | Version: master
Internationalization |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* keywords: 1.10 => 1.10 windows
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:20>
Django
correctly
Reporter: gavinwahl | Owner: claudep
Type: Bug | Status: assigned
Component: Internationalization | Version: master
Keywords: 1.10 windows | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* owner: Claude Paroz <claude@…> => claudep
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:21>
Django
correctly
Reporter: gavinwahl | Owner:
Type: Bug | Status: new
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: 1.10 windows | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* owner: claudep =>
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:22>
Django
correctly
Reporter: gavinwahl | Owner: ramiro
Type: Bug | Status: assigned
Component: Internationalization | Version: master
Severity: Normal | Resolution:
Keywords: 1.10 windows | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by ramiro):
* owner: => ramiro
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:23>
Django
correctly
Reporter: gavinwahl | Owner: ramiro
Component: Internationalization | Version: master
Severity: Normal | Resolution: fixed
Keywords: 1.10 windows | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by ramiro):
* status: assigned => closed
* resolution: => fixed
Comment:
I've opened #26645 to track these comment:10 errors. I suspect they aren't
related to the original issue reported here which was fixed by Claude on
fa08d27fb714534670b431fde0cd04a17d637585.
This, I'm re-closing this one as fixed.
--
Ticket URL: <https://code.djangoproject.com/ticket/25677#comment:24>