Re: [Django] #11240: Compilemessages fails if a % character is at certain places in the .po file
Django
-----------------------------------------+------------------------------
Reporter: tback | Owner: garcia_marc
Type: Uncategorized | Status: reopened
Milestone: | Component: Translations
Version: 1.3 | Severity: Normal
Resolution: | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
-----------------------------------------+------------------------------
Changes (by benjaoming):
* status: closed => reopened
* severity: => Normal
* resolution: worksforme =>
* component: Documentation => Translations
* version: 1.0 => 1.3
* easy: => 0
* type: => Uncategorized
Comment:
@ramiro:
> Note the reporter claims he extracted the translatable literal from a
template but it is prefixed by a #, python-format flag in the PO file, and
that's not consistent.
That's very consistent: It is compilemessages that generated this flag,
because it saw a '%' in the string!
> because I don't see any change committed to the compilemessages
management command related to this.
Sorry, but that's not enough. You're right that the error message will go
away. Nonetheless, two percentage characters are outputted when using {%
trans "findme 10%%" %}: "findme 10%%"
Reopening. I have reproduced it on Django 1.3 and 1.2. As of now, the fix
is to use blocktrans instead, in which case you write:
{{{
{% blocktrans %}findme 10%{% endblocktrans %}
}}}
...producing correct .po gramma:
{{{
#, python-format
msgid "findme 10%%"
msgstr "findme 10%%"
}}}
...and it will correctly output "findme 10%".
--
Ticket URL: <http://code.djangoproject.com/ticket/11240#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-----------------------------------------+------------------------------
Reporter: tback | Owner: garcia_marc
Type: Uncategorized | Status: reopened
Milestone: | Component: Translations
Version: 1.3 | Severity: Normal
Resolution: | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
-----------------------------------------+------------------------------
Comment (by benjaoming):
Oh god no sorry. The last line should be:
...and it will wrongly output an untranslated "findme 10%" -- it doesn't
translate!
--
Ticket URL: <http://code.djangoproject.com/ticket/11240#comment:8>
Django
------------------------------------+------------------------------
Reporter: tback | Owner: garcia_marc
Version: 1.3 | Severity: Normal
Resolution: | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
Changes (by kikko):
* cc: kikko (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:10>
Django
Reporter: tback | Owner: garcia_marc
Type: Bug | Status: closed
Version: 1.3 | Severity: Normal
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
Changes (by ramiro):
* status: reopened => closed
* resolution: => needsinfo
Comment:
Replying to [comment:2 ramiro]:
> Yes, for the translatable literals extraction process templates are
(internally) converted to python code and then fed as such to the gettext
tools.
>
> This means that Python string formatting specifiers rules should be
followed when using the i18n template tags with string literals containing
extrapolated variables or special meaning chars like `%`. In this case it
means you literal would need to be `{% trans "findme 10%% " %}`
I've verified this is behaving correctly with trunk as of now and with
1.3. Ignore that advice I gave back then. I suspect r11459 fixed this in
django so it works transparently for the app developer without the need to
use `%%` with the `trans` i18n template tag.
To test thing I created a template like this:
{{{
{% load i18n %}
{% trans "a literal with a percent symbol at the end %" %}</br>
{% trans "a literal with a percent symbol at the end 10%" %}</br>
{% trans "a literal with a percent % symbol in the middle" %}</br>
{% trans "a literal with a percent 20% symbol in the middle" %}</br>
}}}
`makemessages -l de` created a `.po` file like this from it (note I've
already added dummy translations):
{{{
#: t11240/templates/a.html:2
msgid "a literal with a percent symbol at the end %"
msgstr "translation to German of a literal with a percent symbol at the
end %"
#: t11240/templates/a.html:3
msgid "a literal with a percent symbol at the end 10%"
msgstr ""
"translation to German of a literal with a percent symbol at the end 10%"
#: t11240/templates/a.html:4
#, python-format
msgid "a literal with a percent % symbol in the middle"
msgstr ""
"translation to German of a literal with a percent % symbol in the middle"
#: t11240/templates/a.html:5
#, python-format
msgid "a literal with a percent 20% symbol in the middle"
msgstr ""
"translation to German a literal with a percent 20% symbol in the middle"
}}}
and `compilemessages` creates a corresponding `.mo` file without problems.
Setting `LANGUAGE="de"` and using the template in a view correctly shows:
{{{
translation to German of a literal with a percent symbol at the end %
translation to German of a literal with a percent symbol at the end 10%
translation to German of a literal with a percent % symbol in the middle
translation to German a literal with a percent 20% symbol in the middle
}}}
I'm closing this ticket and setting the reason ''needsinfo''. If you
reopen it please provide exact details of what you are seeing.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:11>
Django
Type: Bug | Status: closed
Milestone: | Component:
Resolution: needsinfo | Severity: Normal
Triage Stage: Accepted | Keywords:
Needs documentation: 0 | Has patch: 0
Patch needs improvement: 0 | Needs tests: 0
| Easy pickings: 0
-------------------------------------+-------------------------------------
Changes (by ramiro):
* component: Translations => Internationalization
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:12>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Component: | Status: reopened
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by harm):
* status: closed => reopened
* ui_ux: => 0
* resolution: needsinfo =>
Comment:
Reopened.
I see this issue too with django 1.3
template
{{{
{% trans '% of test' %}
}}}
po file
{{{
msgid "% of test"
msgstr "% van testtrans"
}}}
{{{ python manage.py compilemessages}}} gives the following error
{{{
locale/nl/LC_MESSAGES/django.po:950: 'msgstr' is not a valid Python format
string, unlike 'msgid'. Reason: In the directive number 1, the character
'v' is not a valid conversion specifier.
msgfmt: found 1 fatal error
}}}
Escaping with a double {{{ %% }}} in the template doesn'th help. (that
renders as a double percentage)
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:13>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: reopened
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by harm):
* cc: harm.verhagen+django@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: reopened
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by ramiro):
I think I understand now what this ticket has always been about.
The issue is with the `'% o'` fragment of the examples in the msgid's
provided by both users that experienced problems.
`'% o'` is a valid interpolation specification, it mean an unsigned octal
preceded by a space for positive values (or the `'-'` sign for negative
ones) this is what the space ''conversion flag'' means.
I suspect this isn't the only case when an unintended formatting specifier
can skean in translatable literals involving percent characters.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: reopened
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
The patch attached implements a fix for this in the 'makemessages' step by
implementing escaping of '%' symbols in literals passed to the `{% trans
%}` template tag (it replaces them with `'%%'`).
This means that starting with this change simple
[http://docs.python.org/library/stdtypes.html#string-formatting-operations
Python string interpolation] isn't supported in literals passed to
`'trans'` anymore.
This also means that the `'msgid'`'s extracted from such literals will
have now `'%%'` and that translators should also use the same sequence in
the respective `'msgstr'`'s. With this:
* Unfortunately GNU gettext's xgettext still marks the msgid/msgstr entry
with the `#, python-format` [http://www.gnu.org/s/hello/manual/gettext/PO-
Files.html flag]. Even when it contains no Python string formatting
specification (e.g. "A string with two %%" still gets marked so). There is
no way we can avoid addition of the flag or to remove it afterward because
the execution of xgettext is an opaque step.
* Even when the entry is marked with the `#, python-format` flag GNU
gettext's msgfmt command additional checks triggered by such mark doesn't
reject it with the error message reported in this ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:16>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: reopened
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by jezdez):
LGTM
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:17>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: reopened
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by claudep):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:18>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: reopened
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by claudep):
* cc: claude@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:19>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Internationalization | Version: 1.3
Severity: Normal | Resolution: fixed
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
* status: reopened => closed
* resolution: => fixed
Comment:
In [17190]:
{{{
#!CommitTicketReference repository="" revision="17190"
Fixed #11240 -- Made makemessages i18n command escape % symbols in
literals passed to the trans tag.
This avoids problems with unintended automatic detection, marking and
validation of Python string formatting specifiers performed by
xgettext(1)/msgfmt(1) that stem from the fact that under the hood
makemessages
converts templates to Python code before passing them to xgettext.
This also makes it consistent with its behavior on literals passed to the
blocktrans tag.
Thanks Jannis and claude for reviewing and feedback.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:20>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: closed
Internationalization | Version: 1.3
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by anonymous):
I still see this with 1.4.3? All msgstr's with % fail as original bug
explains.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:21>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: closed
Internationalization | Version: 1.3
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by anonymous):
Sorry ignore that, I messed up with versions.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:22>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
* status: closed => new
* resolution: fixed =>
Comment:
I have issues with trans tags containing % Symbols using django 1.5.1
The trans tag doesn't handle a % the same way makemessages does. A `{%
trans "value in %"}` still looks for a `msgid "value in %"` but
makemessages created a `msgid "value in %%"` for it.
The trans tag should implement the same string conversion as makemessage
does.
The workaround to already escape the % in the trans tag - `{% trans "value
in %%"}` - returns the right translation.
But as the translation will most likely have a "%%" in it the rendered
conent will have a "%%" too.
A workaround could be to let the TranslateNode return `value % ()`.
Currently the only way to avoid issues with % Symbols in translations is
using blocktrans tags.
It might be worth monitoring
http://savannah.gnu.org/bugs/?func=detailitem&item_id=30854 for changes on
gettext handling python format strings.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:23>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by claudep):
Unfortunately yes, I also think there is still an issue with `trans` and
percents. Just attached the test case as it should be (and which currently
fails).
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:24>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by anonymous):
You could just use a custom wrapper for pgettext (for example)
wrapper.py:
-----------------
from django.utils.translation import pgettext as pgt
def pgettext(context, msg):
return pgt(context, msg).replace("%%", "%")
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:25>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by skyjur):
Another workaround is to use alternative syntax for translation:
{{{
{{ _('Percent sign here % is not escaped.') }}
}}}
I guess that gettext might be looking for python syntaxed string-
formatting in template files and is mislead because text string is
followed directly by percent sign '''%}'''.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:26>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by timo):
* needs_better_patch: 0 => 1
* stage: Ready for checkin => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:27>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by ramiro):
It would have been much better if you had opened a new ticket instead of
reopening this one.
I suspect the issue has to do with the fact that the xgettext tool
unconditionally adds a `"#, python-format"` flag (see comment:16) but we
(by design) don't support Python string format specifiers on 'trans' tag
literals.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:28>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by bouke):
Trying to replicate the bug, I've found that `#, python-format` compiles
just fine with invalid string format specifiers.
{{{
#, python-format
msgid "My project has 20% success"
msgstr "No such thing as 20% success"
}}}
{{{
$ xgettext -V
xgettext (GNU gettext-tools) 0.18.2
}}}
Why would this be the case? Is this expected behaviour? If so, replacing
`%` by `%%` makes no sense and that changeset should be reverted.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:29>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by ramiro):
Replying to [comment:29 bouke]:
> Trying to replicate the bug, I've found that `#, python-format` compiles
just fine with invalid string format specifiers.
> {{{
> #, python-format
> msgid "My project has 20% success"
> msgstr "No such thing as 20% success"
> }}}
These aren't invalid Python format specifiers. "% s" is a valid one. Can
you re-test with one %-prefixed sequence that is actually invalid?
>
> {{{
> $ xgettext -V
> xgettext (GNU gettext-tools) 0.18.2
> }}}
>
> Why would this be the case? Is this expected behaviour? If so, replacing
`%` by `%%` makes no sense and that changeset should be reverted.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:30>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: new
Internationalization | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by ramiro):
I'm re-closing this ticket because the original issue was fixed two years
ago.
If you intend to follow-up please open a newer one with a clean slate.
Make sure you take in account comments comment:28 and comment:30 in you
analysis.
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:31>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Internationalization | Version: 1.3
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:32>
Django
-------------------------------------+-------------------------------------
Reporter: tback | Owner:
Type: Bug | garcia_marc
Component: | Status: closed
Internationalization | Version: 1.3
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by kingsley):
Or you could replace instances of "%" with ascii "%"
Works for me
--
Ticket URL: <https://code.djangoproject.com/ticket/11240#comment:33>