makemessages failing with non ascii characters in templates

[Bastian]

Hi, I am trying to create a .po file to start a translation. I use the command './manage.py makemessages -l ru' to try to create the Russian locale. The script seems to be working for a while and then crashes with:

processing language ru
Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_manager(settings)
  File "/home/env/project/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/home/env/project/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/env/project/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/env/project/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/home/env/project/lib/python2.7/site-packages/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/home/env/project/lib/python2.7/site-packages/django/core/management/commands/makemessages.py", line 365, in handle_noargs
    make_messages(locale, domain, verbosity, process_all, extensions, symlinks, ignore_patterns, no_wrap, no_obsolete)
  File "/home/env/project/lib/python2.7/site-packages/django/core/management/commands/makemessages.py", line 233, in make_messages
    f.write(templatize(src, orig_file[2:]))
  File "/home/env/project/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 127, in templatize
    return _trans.templatize(src, origin)
  File "/home/env/project/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 450, in templatize
    content = u''.join(comment)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

From what I understand makemessages will crash in such a way when encountering a non ascii character in a template, whether marked for translation or not.
So I have been looking for non ascii characters in the templates of this project and indeed there are quite a lot of them.
Now my question is, if it really is a problem with non ascii characters in template files, then what can I do about those characters? Is there a way
to mark a template file as utf8 as in a .py script? Or what am I missing?

thanks!

[Ramiro Morales]

Upgrade your copy of Django 1.3.x to a supported version (1.3 isn't
for this kind
of fixes) or apply this fix manually:

https://github.com/django/django/commit/0defa7dc9dc97a11baea3b91158fbc7c57e5c3a6

Django versions 1.4 or newer already include it.

As a general rule, if you find a problem with an ancient version it's
better to test
with a newer version because chanes are it has been fixed.

--
Ramiro Morales
@ramiromorales

[Bastian]

Thanks Ramiro,

I don't know the actual burden of upgrading a whole project from 1.3 to 1.4 but it does sound terrifying :) I might go the patch way.

[Bastian]

If I understand well, your fix is used to allow non ascii characters in the comments of a template. I found non ascii characters outside comments too, won't this break also?

On Tuesday, March 26, 2013 12:19:36 PM UTC+1, Ramiro Morales wrote:

[Ramiro Morales]

On Tue, Mar 26, 2013 at 9:23 AM, Bastian <bastien....@gmail.com> wrote:
> If I understand well, your fix is used to allow non ascii characters in the
> comments of a template. I found non ascii characters outside comments too,
> won't this break also?

Unfortunately we don't have information about which template file is the one
triggering the issue.

it would be great if you could isolate it and create a simple test case.

Non ASCII characters in non-comment template content should be handled
transparently.

--
Ramiro Morales
@ramiromorales

[Bastian]

I have to search again to filter only the non ascii chars in the comments. Right now I have a list of all non ascii characters. But I am considering upgrading to 1.4 before that not only for the translations. If not, or if the problem still exists by then I will search for the exact template.
Thanks for your help, much appreciated.