Backends for i18n in a contrib app - #14974
Marinho Brandao
I did open the ticket #14974 with a suggestion to make a backend
system for Internationalization component, in a contrib app.
Gettext is a good tool but limited to static files, and it sucks when
you work with long texts, you know: it gets a little unstable when the
message IDs are long.
There are other reasons to (sometimes) prefer an alternative to gettext, like:
- Translators hate work on potfiles, and tools like Rosetta and Pootle
Server just make it soft but only that
- It's (almost) impossible to work with dynamic translations (you must
be a ninja)
- It's impossible to translate database object (so there is a lot of
applications, like django-multilingual, polyglot, etc.)
- The current system is locked to "django.po" files, they get long and
starts the headache
The last 3 projects I'm working are a bit complicated in the topic of
internationalization and we used Pootle Server in one, used our own
"polyglot" [1] in the second and we are thinking on what use in the
third.
The project with Pootle Server forced me to make an alternate
Translation class, to use my own "project.(po|mo)", collecting
messages strings from database, and a Fabric script to update from/to
Pootle Server. I mean: it works but sucks.
Well, of course I'm not here to just criticize the current sollution,
that is better than any other sollution I have seen before (Rails had
a i18n system years after Django).
But I'd like to suggest that we could make a backend system, like we
have for many other components of the framework. So the developers
could choose between gettext and database or create their own
backends, with their dynamic sollutions.
So, summing up, my suggestion is, at least:
- Create a new contrib "i18n" with a backend system
- Move template tags and management commands to the new contrib (and
change them to work with the backends)
- Change the current ugettext, ugettext_lazy, etc. to work on the backend system
- Keep the gettext backend as default
Desirable plus(es):
- Create a backend for database storing and model class to store
- Create a model class for translation of model fields (for database
objects, of course)
Of course, gettext is not only defects and database is not only
qualities, but a backends system with both choices would help
developers to choose that more appropriate to each project.
This is a initial insight. If the idea get approved, I can work on that.
PS: I didn't say anything about Localization because I still haven't
thought about it.
[1] http://reddes.bvsalud.org/projects/clinical-trials/browser/trunk/opentrials/polyglot
--
Marinho Brandão (José Mário)
http://marinhobrandao.com
http://raminel.com.br
Suporte profissional para Django, Python, HTML5, TDD e desenvolvimento
web é no Alatazan.com
Jannis Leidel
> But I'd like to suggest that we could make a backend system, like we
> have for many other components of the framework. So the developers
> could choose between gettext and database or create their own
> backends, with their dynamic sollutions.
Repeating what I said on the ticket, I don't quite understand what you mean by backend system.
The big issue I see here (at least from what I understand) is that content translation and translation of UI elements are completely different stories due to the way those content types differ in structure and storage. Adding a backend system implies having a common API, but I don't see any. Can you elaborate?
Jannis
Marinho Brandao
> Repeating what I said on the ticket, I don't quite understand what you mean by backend system.
A "backend system" is almost the same of design pattern "Adapter" [2]
that is used in many of functions of Django:
- Cache backends
- Emails backends
- Messages backends
- Authentication backends
- Session backends
- Database wrappers (I know it's not a backend, but it's the same idea)
- etc.
All of them are "interfaces" that Django uses to allow swap a package
of rules to another that has the same methods and respond to calls
receiving the same parameters, etc.
The important part here is: to let the developer write a new class to
support I18N functions and choose it instead of gettext using a
setting (like CACHE_BACKEND, AUTH_BACKENDS, etc).
> The big issue I see here (at least from what I understand) is that content translation and translation of UI elements
> are completely different stories due to the way those content types differ in structure and storage. Adding a
> backend system implies having a common API, but I don't see any.
Ok, just be a little patient with me, because sometimes my english so
good as yours :)
I'm NOT saying anything about UI elements (yes, I'm aware about them,
I wrote some of form/fields/widgets [1] to support them, but this is
not the point of my ticket).
My point is about refactory the i18n functions to be extendable, so we
can choose gettext when we can or database when we need, or anything
else when we want to customize, whatever.
I would be very happy to just remove gettext from my life as a Django
developer, replacing the potfiles for database objects, and I guess
much would like too. How to do that? Well, I guess the only way would
be Django support backends with i18n functions.
Just to clear about what I mean with "database objects":
class MessageString(models.Model):
language = CharField()
hash = CharField()
message_id = TextField()
message_str = TextField()
Look? This would be enough to store message strings in database
instead of potfiles.
Now, the "plus": in addition to that, some of us need also to
translate database object (for example, a FlatPage "about" in many
languages), and some ones need also a way to translate the message
strings using a user friendly tool (like django-rosetta does).
> Can you elaborate?
I have the all idea in my mind, so, I can. But my question is: are you
interested on the idea or just disagree?
I think this is a real demand (all of us that worked for years with
i18n in Django know that it's an easy job). Everything on Django is
based on adapters, so, if the idea is approved, so we can write
patches, discuss and find the best way to do that.
I can assume the job, fork Django in GitHub, change the files, write
the tests and docs, etc. but of course, I can't do that without the
approving because I will be working on my own version of Django, that
only makes sense if some day it will be merged to the trunk.
Otherwise, I'm going to start on it as an external application, but
you know, it's not enough, because "django/utils/translation" must be
changed.
And, of course, all help is welcome.
[1] http://goo.gl/siNcR
[2] http://en.wikipedia.org/wiki/Adapter_pattern
Marinho Brandao
I tried to be more clear with a long comment in the ticket [1].
Please, I wrote that from my mind without revise anything. It's just a
trial of make the idea more clear.
[1] http://code.djangoproject.com/ticket/14974#comment:4
Jannis Leidel
>> are completely different stories due to the way those content types differ in structure and storage. Adding a
>> backend system implies having a common API, but I don't see any.
>
> I wrote some of form/fields/widgets [1] to support them, but this is
> not the point of my ticket).
Well, I actually didn't mean forms and widgets with "translation of UI elements", but the "UI" of your website/project compared to the "data" that is displayed on it (e.g. from models). The first is covered by Django fairly well by using the industry standard gettext. The latter depends on the app and type of content, IMO.
> My point is about refactory the i18n functions to be extendable, so we
> can choose gettext when we can or database when we need, or anything
> else when we want to customize, whatever.
But what are the advantages of supporting different backends than gettext?
> I would be very happy to just remove gettext from my life as a Django
> developer, replacing the potfiles for database objects, and I guess
> much would like too. How to do that? Well, I guess the only way would
> be Django support backends with i18n functions.
>
> Just to clear about what I mean with "database objects":
>
> class MessageString(models.Model):
> language = CharField()
> hash = CharField()
> message_id = TextField()
> message_str = TextField()
>
> Look? This would be enough to store message strings in database
> instead of potfiles.
That really isn't enough to be an alternative to gettext.
> Now, the "plus": in addition to that, some of us need also to
> translate database object (for example, a FlatPage "about" in many
> languages), and some ones need also a way to translate the message
> strings using a user friendly tool (like django-rosetta does).
>
>> Can you elaborate?
>
> I have the all idea in my mind, so, I can. But my question is: are you
> interested on the idea or just disagree?
Oh yeah, I am very much interested in the general idea of making it easier to translate the projects that I develop with Django. In fact during DjangoCon Europe this year we brainstormed about the problem and added it to the wiki [2].
But your proposal hasn't sufficently shown to me yet why you want to add a backend system.
> I can assume the job, fork Django in GitHub, change the files, write
> the tests and docs, etc. but of course, I can't do that without the
> approving because I will be working on my own version of Django, that
> only makes sense if some day it will be merged to the trunk.
Please read the contributing docs [1] for further details on how to make a feature request and work on code.
> Otherwise, I'm going to start on it as an external application, but
> you know, it's not enough, because "django/utils/translation" must be
> changed.
Speaking of 3rd party app: There are quite a few apps out there that try to solve the problem of translating content in the database. One of which I found rather intriguing was django-dbgettext.
Jannis
1: http://docs.djangoproject.com/en/dev/internals/contributing/
2: http://code.djangoproject.com/wiki/ModelInterNationalization
3: https://bitbucket.org/drmeers/django-dbgettext
Marinho Brandao
> Well, I actually didn't mean forms and widgets with "translation of UI elements", but the "UI" of your website/project compared to the "data" that is displayed on it (e.g. from models). The first is covered by Django fairly well by using the industry standard gettext. The latter depends on the app and type of content, IMO.
I see... but there is nothing that is a standard that couldn't be
improved, and gettext sucks, anyway. I know it's the only way, and I'm
thankful for that, but IMO it's something that should be improved a
lot to be enough - to support dynamic, for example.
About the latter, yes, this is why I said it is a "plus". But, I still
believe most of cases could be solved with the same sollution (the
same that multilingual and others do).
> But what are the advantages of supporting different backends than gettext?
Well, don't you see advantages after everything I said? For example
let the translators use a friendly tool, or dynamic tranlations, to
avoid necessity of reload .mo files to memory, etc.?
Are you happy translating very long "django.po" without can just split
it in many files or just database records stored in cache?
> That really isn't enough to be an alternative to gettext.
Of course
> Oh yeah, I am very much interested in the general idea of making it easier to translate the projects that I develop with Django. In fact during DjangoCon Europe this year we brainstormed about the problem and added it to the wiki [2].
The [2] is about the "plus" I said.
> But your proposal hasn't sufficently shown to me yet why you want to add a backend system.
Being a little more clear: to remove gettext when developer wants.
> Please read the contributing docs [1] for further details on how to make a feature request and work on code.
I know it
> Speaking of 3rd party app: There are quite a few apps out there that try to solve the problem of translating content in the database. One of which I found rather intriguing was django-dbgettext.
Yes, this is one more reason to notice that we have a problem that
must be solved.
Jannis Leidel
> Are you happy translating very long "django.po" without can just split
> it in many files or just database records stored in cache?
It gets the job done, yeah. For complex translations I've used Pootle and Transifex, both of which can import/export PO files.
>> But your proposal hasn't sufficently shown to me yet why you want to add a backend system.
>
> Being a little more clear: to remove gettext when developer wants.
I haven't heard from any developer wanting to remove gettext support yet.
> I tried to be more clear with a long comment in the ticket [1].
>
> [1] http://code.djangoproject.com/ticket/14974#comment:4
Interesting API indeed. Just a few question (not in order of importance):
- What kind of backends can you additionally think of?
- In the database backend, how would plural forms of strings work in languages with multiple plurals?
- How would editing of the database catalogue work?
- How would translation catalogues of 3rd party apps be handled? And how would translations be distributed/deployed?
Jannis
Marinho Brandao
>> Are you happy translating very long "django.po" without can just split
>> it in many files or just database records stored in cache?
>
> It gets the job done, yeah. For complex translations I've used Pootle and Transifex, both of which can import/export PO files.
yes, it's what we do too. As I said before, in the project that
Pootle/potfiles were choosed (not my decision), we have more/less 200
thousands of long terms to translate, including database objects (my
"plus"), so you can imagine how hard it is to maintain.
Other situation: potfiles aren't perfect. Some terms just are moved to
fuzzy or commented and sometimes we just can't figure out why. It's
easy to suppose what's the cause, but in a bad day, when something
just "hides" in production box, if the translations were in database,
a simple change in an Admin URL would be enough to save the day.
> I haven't heard from any developer wanting to remove gettext support yet.
I know some ones. Actually I don't know anybody **very happy** with
gettext. We all just use it because it is the only way we have.
Actually most of Django developers don't use i18n functions as deep as
we are discussing. Most of developers don't use or only work with
short terms.
But, anyway, the human kind was happy without airplanes, and now we
can't live without them, so...
Of course gettext is a mature and universal tool, but IMO it is more
appropriate for desktop/compiled software, and we are talking about
web/dynamic applications in the cloud.
> Interesting API indeed. Just a few question (not in order of importance):
>
> - What kind of backends can you additionally think of?
I could say "Java translation strings", "xml files", etc... but
actually don't believe somebody will work with them (but it's
possible).
But I believe that we can extend our own versions of both backends
(gettext and db) to make specific dynamic changes depending on the
project (as smart translator maybe?). Maybe rare, but possible.
> - In the database backend, how would plural forms of strings work in languages with multiple plurals?
I didn't think about everything (this is why I open a ticket to get a
discussion and approvation after that).
But we can:
- implement our own sollution
- use - if possible - the sollution that gettext lib does
We don't need to hate gettext. That's not the point. The point of my
opinion is that Django's wrapper for gettext doesn't support neither
dynamic translations nor database stored strings. We can use part of
gettext library when we need, why not? :)
> - How would editing of the database catalogue work?
I like the interface of Pootle, we could make something like that, but
under Admin.
> - How would translation catalogues of 3rd party apps be handled? And how would translations be distributed/deployed?
Fixtures.
:)
Marinho Brandao
>> - In the database backend, how would plural forms of strings work in languages with multiple plurals?
The database backend will not be the best in all features. In the same
way that "localmem://" it's not better than "memcached://" but
sometimes it's necessary, database can be slower than gettext and
probably less smart.
I bet most of application would keep in gettext without change anything. :)
The backend system just give us the choices.
Jonas Obrist
Marinho Brandao
> Okay I would like to jump in here quickly and give my two cents on this
> topic.
great :)
> A very important distinction that should be made is the difference between
> 'dynamic' and 'static' translations.
> I call translations 'static' when they're strings that are fix in the
> templates (eg the text on the login button) as opposed to what I call
> dynamic, which are fields in the database.
Very good thing you take this to clear what we mean for "static" and "dynamic".
So, let me say what I mean for "static" and "dynamic":
"Static" for me is something that you can't change easily. I mean: a
static text file "django.po" that have to compiled to "django.mo" and
loaded to memory when virtual machine is initialized. If you want to
change some string, you must: 1) update the .po file with current
strings, 2) set the translation for strings in msgstr, 3) compile
those files to django.mo and 4) reload the VM.
"Dynamic" for me is something that you can change easily when you
want. This means you shouldn't edit files (instead, you use Admin to
do that) and you shouldn't reload the VM. Why? Just because we are
talking about a dynamic language and because the Web is dynamic.
What's the difference between static HTML and a CMS? Well, a CMS
offers the possibility for you change the content dynamically, using
web tools so you can update a website faster and smarter than just
edit HTML files and send them using an FTP, right?
That's the same reason why we need dynamic translations.
> For the static translations we have a very good tool with gettext. This is a
> solid, mature and working tool that's used in a lot of places, not just
> Django.
I more/less disagree. I work with i18n apps with Django since 4 or 5
years ago and it always depend of tricks to work nicely.
If you have a {% blocktrans %} with 4 lines and you just indent those
lines, you lose the translation, so you have to make the files again
and update their translations (at least copying and pasting from
commented to new msgstr). You could fix that if you had a way to
change the behavior of i18n in Django - using backends, for example -
so you could just to replace multiple white spaces for this one and
this would be fixed.
If you have a line that ends with \n and your translation doesn't, you
get an error. So, a backend could help too.
The two situations above would be worst if you have long texts.
If "django.po" gets very long, this increases its maintaining. You can
say that Pootle solves that but you can just try to search for a short
term in Pootle when you have thousands of strings and you will notice
that this is a hell.
If you show messages to end user using i18n strings and your customer
wants to change them (this happens all the time), this has to be done
by a developer, that must do those 4 steps to have it working in
production.
If you are the developer in vacation and your client wants to just fix
a spell mistake, he must choose among 1) call you, 2) wait you be back
or 3) go ahead into hard code (believe me, this happens). If you had
translations in database, he could do that for himself.
So, there are lots of situations that static potfiles make you work
around. You can tell me what is the work around for each of those
situations, but the truth is that they are just work around.
> For dynamic translations we indeed have a huge problem, a problem I think
> needs to be solved. However this should not be done using 'backends' but
> rather by extending the ORM in a way that it allows applications like
> django-multilingual and the like to be written easier (without their own
> SQLCompiler, Query-class Queryset-class etc). Ideally Django would also
> provide a contrib app that actually gives you a full implementation of
> translatable database contents, but that's optional.
Ok, now we are talking about that thing I called "plus". Just because
this is not the point of my ticket.
I wouldn't say this is the "dynamic" in counterpart to "static"
potfiles because they are really different things.
The translated strings that Django's i18n support are about the
software, and this is about database content.
To illustrate: if you talk about a "dynamic car", you could mean
something like a smarter or customizable car, but you never would mean
that a "dynamic car" is "the people that drives the car".
Another illustration: if you have an e-commerce website, you can
translate dinamically the message strings and the page title - this is
one thing - or you could translate the product descriptions - this is
another thing. The first is part of the box, and the second is part of
its content.
I'm not sure if my limited english was enough to be clear in the
paragraph above, if it is confuse, please let me know and I try to be
clear, but what I mean is that they are different layers of business.
I agree to you that this is not related to backends.
But I guess this is already being discussed in the Wiki and another
thread, so, this is not my focus here.
I just said this could be a "plus" because it is in the same topic -
the same reason because I said we could move template tags, management
commands and generic views to a contrib application "i18n", together
the backends classes.
> Now the issue with this is, that it's not yet clear what needs to be done
> how in the ORM to make these kind of things easier. I have a couple of ideas
> and I'm sure others do too, but nothing mature has come up yet as far as I
> know.
Well, I'd like to try something with proxy classes (not model proxy
classes) instead of change anything in ORM component, but as I said,
this is not the point of this thread.
thanks for your thoughts :)
Jonas Obrist
Marinho Brandao
> Let's assume for a moment we use your idea and provide a couple of different
> backends to store your translations. How would third party applications ship
> default translations then? At the moment they use gettext pofiles and it
> works like a charm. Let's assume now for my project I use your database
> backend. I would be unable to use any of the default translations.
I'm not sure I got what you mean, but as far as I got, that's my
question for you: what is the difference you see from a {% trans
"Default string here" %} from gettext to database? And why fixtures
would be worse than potfiles to store translations as part of the
application?
> Also what stops you today from using a database backend for translations?
> Nothing! Write custom template tags (you can even override the existing
> ones) and get going.
> The whole 'if someone changes a po file we need to manually reload the
> server' is not true, you can very well automate that and allow clients to
> translate their strings using rosetta or a similar project. Then when the
> files change, restart the server (which is very trivial with mod_wsgi for
> example, since you just need to 'touch' the wsgi files).
yes, we will ever find a work around to solve things that should be
designed correct. We are good developers, to hack is in our blood.
> To be perfectly honest I do not see any point in storing my strings that
> hardly ever change in the database, it would be terribly slow and try to
> solve a problem that is already solve, in my opinion very well, by gettext.
This is not about to decide what is the best and just paste the
framework to work ONLY with that. This is about to make the framework
flexible enough to let people decide what is the best for their
projects.
memcached is the best cache backend that Django offers, so why do we
have a cache backend for database if it is "worse"? The answer is the
same reason that I'm saying we need backends for i18n.
The all Django is flexible for many things, but i18n is different to
the rest, why? How many developers you think that extend Email
backends? Why would be Email backends a good idea and i18n backends
not?
You don't need for database stored translations but >>> I and others
need <<< so, Django is not about to attend the requirements of some
ones but everybody as far as possible.
Even if 99% of people will not use another backend but gettext, the
framework should support a way for the 1% do what they need. To create
a backend will not make Django worse than it is so, why don't?
Shai Berger
I think improving the translation facilities is, indeed, a noble goal.
Reading this thread raised the following questions for me:
1) Do you intend to have more than one backend supported in one project?
If not (as implied by your I18N_BACKEND setting), this means that translations
will have to be made available for all backends -- do you expect pluggable
applications to provide both .po files and fixtures, or do you have in mind
some kind of mechanism for making one out of the other?
If you do, how do you specify which one to use for a specific string?
2) Your suggestion for the database backend seems to be very minimalist
(doesn't even support ngettext); if you're going to improve upon gettext, my
first wish would be to go in the exactly opposite direction: For my language
(Hebrew), in many cases, I'd like to have richer parameters for inflections --
not just single/plural, but also male/female and in some cases even
identified/unidentified.
Would you be interested in any of that? That is, would you like an API which
allows for richer syntactic tagging and agreement options?
Thanks,
Shai.
Marinho Brandao
> I think improving the translation facilities is, indeed, a noble goal.
:)
> Reading this thread raised the following questions for me:
>
> 1) Do you intend to have more than one backend supported in one project?
>
> If not (as implied by your I18N_BACKEND setting), this means that translations
> will have to be made available for all backends -- do you expect pluggable
> applications to provide both .po files and fixtures, or do you have in mind
> some kind of mechanism for making one out of the other?
>
> If you do, how do you specify which one to use for a specific string?
You make sense, I made some mistakes on my example (I was really hasty).
I was thinking on a setting like AUTHENTICATION_BACKENDS, so, if a
backend fails, the next backend is called and going on.
It's probably that for most of cases, database backend will be before
gettext, so the strings changed in database will be returned instead
of original strings from gettext, when they're in both.
When running the command "makemessages" and "compilemessages" we
should inform the path of backend.
> 2) Your suggestion for the database backend seems to be very minimalist
> (doesn't even support ngettext); if you're going to improve upon gettext, my
My example was just to show the flow (to clarify for who doesn't get
what I mean for "backends system", or just Adapters).
I'm sure we should make backends to support everything that Django's
I18N currently supports, that's include ngettext and everything else,
so we could effectively customize it when we want.
> first wish would be to go in the exactly opposite direction: For my language
> (Hebrew), in many cases, I'd like to have richer parameters for inflections --
> not just single/plural, but also male/female and in some cases even
> identified/unidentified.
That's why I insist on "dynamic" support, just to support that kind of
feature of each language and/or project.
But I have no idea how would we support genre, quantities (like
"between" versus "among"), directly/indirectly, etc.
Because plural is based on a number that is informed, but to support
genre it should be aware about the noun before/after the adjective.
> Would you be interested in any of that? That is, would you like an API which
> allows for richer syntactic tagging and agreement options?
Anyway, I guess we should support backends, so many developers would
extend gettext/database/base backends to make their smarter versions
and publish as snippets.
Anyway (2), once the core team approve the idea, we can go next step
to write a Wiki page, and decide some details like this, and work on a
branch/fork until get it enough to merge.
Some things I have no idea what's the best way to do:
- trans_null (and USE_I18N=False) should be a "dummy" backend or just
an "if" in the functions from "django.utils.translation"?
- in DB backend, makemessages should write to database or to fixture files?
- what command would copy from potfiles to database and vice-versa?
- fixtures with translations should be in directory "locale" or "fixtures"?
- should we include Localization support also? I guess we should, but...