Marc Garcia created a new translatable fields project: django-transmeta [1] [2]. It works by add new <field>_<lang> fields to a model automatically, instead of adding a new table.
Because the model syntax is so similar to DM, I feel we should work together to come up with a shared 'API', so that the implementation of translated fields becomes, well, an implementation detail.
This stuff should be in Django proper, and that will happen sooner if there is consensus.
Joost Cassee <jo...@cassee.net> wrote: > Marc Garcia created a new translatable fields project: > django-transmeta [1] [2]. It works by add new <field>_<lang> fields > to a model automatically, instead of adding a new table.
> Because the model syntax is so similar to DM, I feel we should work > together to come up with a shared 'API', so that the implementation of > translated fields becomes, well, an implementation detail.
> This stuff should be in Django proper, and that will happen sooner if > there is consensus.
The "stuff" that should be in Django is the API, you mean? Not django-transmeta? Because I dislike django-transmeta. In my opinion, it violates the 1st normal form of the database; it may look simpler at first but it will create a huge number of problems later when compared to a design such as DM (this is why I chose DM; it's the only one that is correctly designed from a theoretical point of view). This, however, concerns the implementation. The DM interface, I agree, sucks. Transmeta is nicer. I agree that a shared API would be great. Maybe it would be easy for DM to follow the Transmeta interface? I mean that
class MyModel(models.Model): name = models.CharField(max_length=100) description = models.TextField() class Meta: translate = ('name', 'description')
just looks very much more elegant than
class MyModel(models.Model): class Translate: name = models.CharField(max_length=100) description = models.TextField()
> On Sat, 07 Mar 2009 09:01:05 +0100 > Joost Cassee <jo...@cassee.net> wrote: >> Marc Garcia created a new translatable fields project: >> django-transmeta [1] [2]. It works by add new <field>_<lang> fields >> to a model automatically, instead of adding a new table.
>> Because the model syntax is so similar to DM, I feel we should work >> together to come up with a shared 'API', so that the implementation of >> translated fields becomes, well, an implementation detail.
>> This stuff should be in Django proper, and that will happen sooner if >> there is consensus.
> The "stuff" that should be in Django is the API, you mean?
Yes, I mean the API.
> Not > django-transmeta? Because I dislike django-transmeta. In my opinion, it > violates the 1st normal form of the database; it may look simpler at > first but it will create a huge number of problems later when compared > to a design such as DM (this is why I chose DM; it's the only one that > is correctly designed from a theoretical point of view).
I agree, I also think the DM implementation is better. On the other hand, denormalizing the database can be trade-off for performance.
> This, however, > concerns the implementation. The DM interface, I agree, sucks. > Transmeta is nicer. I agree that a shared API would be great.
It's not (necessarily) that the DM sucks, it's the fact that although various projects are developing different implementations, they all share the same goal. Having a shared API would make it easier for developers to 'try out' some of these projects without having to rewrite their applications.
There has been talk of different APIs (for example in [1]), but the current one is just good enough.
On Mar 9, 12:08 pm, Joost Cassee <jo...@cassee.net> wrote:
> > Not
> > django-transmeta? Because I dislike django-transmeta. In my opinion, it
> > violates the 1st normal form of the database; it may look simpler at
> > first but it will create a huge number of problems later when compared
> > to a design such as DM (this is why I chose DM; it's the only one that
> > is correctly designed from a theoretical point of view).
> I agree, I also think the DM implementation is better. On the other
> hand, denormalizing the database can be trade-off for performance.
Right.
Django-transmeta is, in many ways, something that I originally
envisioned as a second mode for operation for DM, but the amount of
effort of implementing both of them in a single library turned out to
be too high.
> It's not (necessarily) that the DM sucks, it's the fact that although
> various projects are developing different implementations, they all
> share the same goal. Having a shared API would make it easier for
> developers to 'try out' some of these projects without having to rewrite
> their applications.
A shared API would be great, but there is a place where the Law of
Leaky Abstractions kicks in: overriding save() on multilingual models.
To use our Article example again:
class Article(models.Model):
class Translation(multilingual.Translation):
title = ....
content = ...
Right now Article.Translation is a normal Django model, which lets DM
reuse large part of Django machinery. This saved us a lot of time,
but it also means there is no easy way to handle translations in
Article.save -- if you need to define Article.Translation.save to do
any modify the translation data.
Using an API similar to transmeta would mean that we either:
1. need a different way to provide at least Article.Translation.save
(and perhaps other stuff you might want to do to a model)
2. need to reimplement DM so that it does not use Django ORM. We could
find a way to modify DM data directly in Article.save, but the rewrite
would be a huge PITA.
Any suggestions of ways to solve it cleanly would be very appreciated.
first of all I want to point out that I wanted to join all projects
for model i18n since I started writing transdb. I think that sooner or
later this feature must go into django itself, and I already discussed
it with Malcolm, the core developer most involved in i18n stuff.
My idea creating, first transdb, and then django-transmeta is first
using them on some projects where I need to translate fields, but then
testing different approaches, as well as getting feedback on the goals
and handicaps of every solution. I absolutely agree that DM database
approach is the only one to consider when developing the "definitive"
solution. But not sure how the latest DM looks like, but this
definitive solution should avoid being as django multilingual in the
UI, and in the developer syntax.
I wrote all those comments in a presentation some months ago [1] that
I discussed with Marcin, Malcolm and few others, with the idea of not
reinventing the wheel, and being as a roadmap to join all projects and
integrate them into django. I should update the presentation, but most
of the staff is still valid I think.
So, I encourage you to test django-transmeta (forget transdb), to see
how it works, specially from the developer point of view; and
afterwards may be you could check how difficult could be making DM
simpler (in a transmeta way, or a better one).
May be you could create a branch for it, and I'll try to spend some
time to join both projects (and any other with good ideas).
> > On Sat, 07 Mar 2009 09:01:05 +0100
> > Joost Cassee <jo...@cassee.net> wrote:
> >> Marc Garcia created a new translatable fields project:
> >> django-transmeta [1] [2]. It works by add new <field>_<lang> fields
> >> to a model automatically, instead of adding a new table.
> >> Because the model syntax is so similar to DM, I feel we should work
> >> together to come up with a shared 'API', so that the implementation of
> >> translated fields becomes, well, an implementation detail.
> >> This stuff should be in Django proper, and that will happen sooner if
> >> there is consensus.
> > The "stuff" that should be in Django is the API, you mean?
> Yes, I mean the API.
> > Not
> > django-transmeta? Because I dislike django-transmeta. In my opinion, it
> > violates the 1st normal form of the database; it may look simpler at
> > first but it will create a huge number of problems later when compared
> > to a design such as DM (this is why I chose DM; it's the only one that
> > is correctly designed from a theoretical point of view).
> I agree, I also think the DM implementation is better. On the other
> hand, denormalizing the database can be trade-off for performance.
> > This, however,
> > concerns the implementation. The DM interface, I agree, sucks.
> > Transmeta is nicer. I agree that a shared API would be great.
> It's not (necessarily) that the DM sucks, it's the fact that although
> various projects are developing different implementations, they all
> share the same goal. Having a shared API would make it easier for
> developers to 'try out' some of these projects without having to rewrite
> their applications.
> There has been talk of different APIs (for example in [1]), but the
> current one is just good enough.
> I absolutely agree that DM database
> approach is the only one to consider when developing the "definitive"
> solution. But not sure how the latest DM looks like, but this
> definitive solution should avoid being as django multilingual in the
> UI, and in the developer syntax.
Yeah, the UI could definitely use a facelift. Philippe Lafaye shared
a patch for great-looking tabbed interface:
But it would mean adding a dependency on jquery and several of its
components, which is something I hesitate to do -- just like Django, I
don't want to tie DM to any particular JS framework. Even tie in the
sense that "using this framework is not mandatory, but you lose
important/nice parts of the interface".
> I wrote all those comments in a presentation some months ago [1] that
> I discussed with Marcin, Malcolm and few others, with the idea of not
> reinventing the wheel, and being as a roadmap to join all projects and
> integrate them into django.
I can not open the presentation right now, but I do remember us
discussing it and talking about the API about a year ago.
To make it clear -- we are talking about model syntax, right? There
is more to the API than that: configuration, choosing the "default"
language, translation fallbacks, queries, being able to tell if the
"en" translation contains empty fields or is completely absent,
probably more.
My impression, year ago, was that both DM database layout and model
syntax were the right choice; what changed in the meantime is that
more people told me the syntax was complex and I've seen one use case
where it forced a developer to repeat information. Also, I know I
might be biased.
So, a poll might be a good idea :)
There are some technical problems with adopting a different syntax
(like save() in my post above), but it's doable, it just might take
better ideas and more time.
> May be you could create a branch for it, and I'll try to spend some
> time to join both projects (and any other with good ideas).
> > I absolutely agree that DM database
> > approach is the only one to consider when developing the "definitive"
> > solution. But not sure how the latest DM looks like, but this
> > definitive solution should avoid being as django multilingual in the
> > UI, and in the developer syntax.
> Yeah, the UI could definitely use a facelift. Philippe Lafaye shared
> a patch for great-looking tabbed interface:
> But it would mean adding a dependency on jquery and several of its
> components, which is something I hesitate to do -- just like Django, I
> don't want to tie DM to any particular JS framework. Even tie in the
> sense that "using this framework is not mandatory, but you lose
> important/nice parts of the interface".
> > I wrote all those comments in a presentation some months ago [1] that
> > I discussed with Marcin, Malcolm and few others, with the idea of not
> > reinventing the wheel, and being as a roadmap to join all projects and
> > integrate them into django.
> I can not open the presentation right now, but I do remember us
> discussing it and talking about the API about a year ago.
> To make it clear -- we are talking about model syntax, right? There
> is more to the API than that: configuration, choosing the "default"
> language, translation fallbacks, queries, being able to tell if the
> "en" translation contains empty fields or is completely absent,
> probably more.
> My impression, year ago, was that both DM database layout and model
> syntax were the right choice; what changed in the meantime is that
> more people told me the syntax was complex and I've seen one use case
> where it forced a developer to repeat information. Also, I know I
> might be biased.
> So, a poll might be a good idea :)
> There are some technical problems with adopting a different syntax
> (like save() in my post above), but it's doable, it just might take
> better ideas and more time.
> > May be you could create a branch for it, and I'll try to spend some
> > time to join both projects (and any other with good ideas).
Hello,
I have read the poll on your blog and feel like I have an opinion on
the subject. I have used multilingual in the past on a project of mine
and I think it shine where you have a complete control on all the
django app you are using. And nowadays I find myself reusing much more
django application that I was doing one year ago, that are of course
not built to support the multilingual scenario. On one hand I do not
have any issue to fork such application to add/remove features and
then submit a patch to the maintainer and see if it could be
integrated to trunk. On the other hand adding directly into the app
the multilingual machinery will make all these bi directional
exchanges much more difficult. I will end up to be back 2 years ago
where I had to write every piece of the application.
So to put it in a nutshell I thing that all these reusable apps are
the killer feature of django based project so I would prefer to have
yet another reusable app to add the multilingual capability to my
project. This is why my preferred solution to the one available now
is:
"""
separate model
class MyModel(model.Model):
my_field = CharField()
my_i18n_field = CharField()
Class MyModelTranslation(TranslationOptions):
fields = ('my_i18n_field',)
"""
This the approach taken by django-modeltranslation now this being said
at this stage I think that Malcom has an excelent point about where
this translation string should be stored. I think that idealy the
solution should be able to offer several backend db, file .po, ...
So my main requirement is don't require me to fork every reusable apps
I am using.
Thank you to get this discussion started.
Regards,
--yml
On Mar 9, 7:43 pm, Marc Garcia <garcia.m...@gmail.com> wrote:
> On Mar 9, 5:09 pm, Marcin Kaszynski <marcin.kaszyn...@gmail.com>
> wrote:
> > Hi Marc,
> > thanks a lot for joining the discussion :)
> > > I absolutely agree that DM database
> > > approach is the only one to consider when developing the "definitive"
> > > solution. But not sure how the latest DM looks like, but this
> > > definitive solution should avoid being as django multilingual in the
> > > UI, and in the developer syntax.
> > Yeah, the UI could definitely use a facelift. Philippe Lafaye shared
> > a patch for great-looking tabbed interface:
> > But it would mean adding a dependency on jquery and several of its
> > components, which is something I hesitate to do -- just like Django, I
> > don't want to tie DM to any particular JS framework. Even tie in the
> > sense that "using this framework is not mandatory, but you lose
> > important/nice parts of the interface".
> > > I wrote all those comments in a presentation some months ago [1] that
> > > I discussed with Marcin, Malcolm and few others, with the idea of not
> > > reinventing the wheel, and being as a roadmap to join all projects and
> > > integrate them into django.
> > I can not open the presentation right now, but I do remember us
> > discussing it and talking about the API about a year ago.
> > To make it clear -- we are talking about model syntax, right? There
> > is more to the API than that: configuration, choosing the "default"
> > language, translation fallbacks, queries, being able to tell if the
> > "en" translation contains empty fields or is completely absent,
> > probably more.
> > My impression, year ago, was that both DM database layout and model
> > syntax were the right choice; what changed in the meantime is that
> > more people told me the syntax was complex and I've seen one use case
> > where it forced a developer to repeat information. Also, I know I
> > might be biased.
> > So, a poll might be a good idea :)
> > There are some technical problems with adopting a different syntax
> > (like save() in my post above), but it's doable, it just might take
> > better ideas and more time.
> > > May be you could create a branch for it, and I'll try to spend some
> > > time to join both projects (and any other with good ideas).
> But it would mean adding a dependency on jquery and several of its > components, which is something I hesitate to do -- just like Django, I > don't want to tie DM to any particular JS framework. Even tie in the > sense that "using this framework is not mandatory, but you lose > important/nice parts of the interface".
In case you are interested and if jQuery would be tolerated, I could create a patch for django-multilingual of a custom inline admin class I use in one of my projects. It would require a bit of cleanup and update to the current dm version, but should be reusable.
> So to put it in a nutshell I thing that all these reusable apps are > the killer feature of django based project so I would prefer to have > yet another reusable app to add the multilingual capability to my > project.
Agreed, that's a good point given most available Django apps don't support multilingual content out of the box. Not having a complete overview of the technical difficulties, I think it would be great for developers to be able to retrofit external apps with translatability depending on their use case, just like we are able to do it with the admin app, for example. App authors could of course also provide sensible defaults.
At some point in the future, Pinax needs to be able to handle translated content. Although I've used django-multilingual in a Pinax- based project recently, I wondered about the inner class declaration. Mostly because it reminded me of the inner Admin class drama and its limitated reusability. Having said that I'd be happy to contribute.
Your reply to Malcolms comment on your blog, and Yann's reply to this e-mail, highlight the (in my opinion) most important goal of a shared API: to allow a model to indicate which fields can be translated, like strings in static Python text or templates.
There is a division of labour here between application developers and project developers. I think we can safely leave the decision of which fields are translatable to the app developers. They make this decision already for static strings. Decisions such as which languages to provide, fallback to default language, where to store translations, etc. should be left to project developers.
I believe this separation of labour evades the problem Malcolm sees with a one-size-fits-few solution.
> On Mar 9, 12:08 pm, Joost Cassee <jo...@cassee.net> wrote: >> It's not (necessarily) that the DM sucks, it's the fact that although >> various projects are developing different implementations, they all >> share the same goal. Having a shared API would make it easier for >> developers to 'try out' some of these projects without having to rewrite >> their applications.
> A shared API would be great, but there is a place where the Law of > Leaky Abstractions kicks in: overriding save() on multilingual models.
> [...]
> Using an API similar to transmeta would mean that we either:
> 1. need a different way to provide at least Article.Translation.save > (and perhaps other stuff you might want to do to a model)
Wouldn't that be easy if you override the __metaclass__ property on models (like transmeta does)? Probably I didn't understand the problem.
Anyway, what would happen if Django would accept and store the 'i18n=True' argument to Field constructors. Apps like DM could just listen to the 'class_prepared' signal and do their magic there.
> On Mar 9, 2:25 pm, Marc Garcia <garcia.m...@gmail.com> wrote:
>> May be you could create a branch for it, and I'll try to spend some
>> time to join both projects (and any other with good ideas).
> done. I just created the unified_api branch and added you to the
> project. Welcome on board :)
On Mar 10, 1:04 pm, Joost Cassee <jo...@cassee.net> wrote:
> Wouldn't that be easy if you override the __metaclass__ property on
> models (like transmeta does)? Probably I didn't understand the problem.
Yeah, we already use metaclasses in DM. A lot.
The problem is the sequence of events in some situations. For
example, when you save an Article with related Article.Translations
edited inline, it looks like this:
1. Article is saved
2. Article.Translations are handled (added, modified, deleted)
To make the API consistent (and to get rid of the internatl
Translation class) we should be able to modify translations in
Article.save. But at this point the translations might not even exist
yet. And even if they do, the second part will overwrite any
modifications you do to them.
This might be easier with Django 1.0, but still be very tricky.
> Anyway, what would happen if Django would accept and store the
> 'i18n=True' argument to Field constructors.
This would require monkey patching the constructors of all fields you
might want to use in models or creating special fields. This is bad,
as we want to be able to handle translations of _any_ fields, no
matter if we know about them or not.
Which is why I don't treat options 2 and 5 in the poll as serious
alternatives to other methods. They are simply too limiting to what
you can accomplish.
Reading your reply let me think tha tI was not clear enough on the
point I was trying to make. I think it is not reasonable to believe
that reusable app developer **will do any effort** to enable
multilingual support for the information stored in db. And this for
the foreseeable future. I have been recently playing with supertagging
[1] and solango [2]. Both project share a similar approach to register
the model and the fileds you manipulate in the machinery that will add
a particular behavior.
The net win of this approach is that you can add a new behavior
without touching a single line of any of the reusable application.
> Your reply to Malcolms comment on your blog, and Yann's reply to this
> e-mail, highlight the (in my opinion) most important goal of a shared
> API: to allow a model to indicate which fields can be translated, like
> strings in static Python text or templates.
> There is a division of labour here between application developers and
> project developers. I think we can safely leave the decision of which
> fields are translatable to the app developers. They make this decision
> already for static strings. Decisions such as which languages to
> provide, fallback to default language, where to store translations, etc.
> should be left to project developers.
> I believe this separation of labour evades the problem Malcolm sees with
> a one-size-fits-few solution.
> Reading your reply let me think tha tI was not clear enough on the > point I was trying to make. I think it is not reasonable to believe > that reusable app developer **will do any effort** to enable > multilingual support for the information stored in db. And this for > the foreseeable future.
Aha! :-) Well, my experience is that all application projects already accept patches to internationalize static strings in Python or templates, so I'm very optimistic on this point.
This depends, of course, on the availability of a way to specify translatable fields without depending on some external i18n project.
so, if we don't override this in our project's `settings.py` file it's false by default.
In the `models.py` this is defined like so::
if settings.IS_MULTILINGUAL: import multilingual ...
class Category(models.Model): if settings.IS_MULTILINGUAL: class Translation(multilingual.Translation): name = models.CharField(_('name'), max_length=50) desc = models.TextField(_('description'), blank=True) else: name = models.CharField(_('name'), max_length=50) desc = models.TextField(_('description'), blank=True)
On Tue, Mar 10, 2009 at 4:53 PM, Joost Cassee <jo...@cassee.net> wrote: > On 10-03-09 15:45, yml wrote:
>> Reading your reply let me think tha tI was not clear enough on the >> point I was trying to make. I think it is not reasonable to believe >> that reusable app developer **will do any effort** to enable >> multilingual support for the information stored in db. And this for >> the foreseeable future.
> Aha! :-) Well, my experience is that all application projects already > accept patches to internationalize static strings in Python or > templates, so I'm very optimistic on this point.
> This depends, of course, on the availability of a way to specify > translatable fields without depending on some external i18n project.
On Mar 10, 10:45 am, yml <yann.ma...@gmail.com> wrote:
> The net win of this approach is that you can add a new behavior
> without touching a single line of any of the reusable application.
Big +1 from me on this point. A reusable apps' models should not be
cluttered up with translation stuff. I don't _want_ a separate
"multilingual flatpages app," I want a translation app that lets me
use contrib.flatpages and add translations. This is simply how Django
reusable apps should work, IMO.
The admin application already went through this whole syntax struggle
once, starting out with a highly-coupled inner class (AFAICS, all of
the poll options except modeltranslation's are just as highly
coupled), and eventually settling on a very nice decoupled
registration syntax. Can't we learn from that experience without
going through it again?
On Mar 10, 4:41 pm, Carl Meyer <carl.j.me...@gmail.com> wrote:
> Can't we learn from that experience without
> going through it again?
Too late, we already went through this :)
There are some differences between DM and Admin. For starters, Admin
does not have to interfere at all when you use models in your views
while the multilingual layer does. There's a lot of stuff happening
when Python creates your model class and even more stuff when that
model is supposed to be multilingual.
Adding the DM stuff from the outside, after the class is already
built, is asking for more problems.
Another point in the changes to admin interface was the idea of having
multiple admin interfaces for the same model. This is something we
won't need with multilingual layers.
> The admin application already went through this whole syntax struggle > once, starting out with a highly-coupled inner class (AFAICS, all of > the poll options except modeltranslation's are just as highly > coupled), and eventually settling on a very nice decoupled > registration syntax. Can't we learn from that experience without > going through it again?
The difference here is that the admin app lives in contrib, whereas if model-level support for i18n enters Django, it's going to live in the core code. The admin gets added to a project via INSTALLED_APPS, but once model-level i18n gets added it's going to just be baked in. I don't think it makes sense to think about these solutions as something optional that may or may not be there. Ultimately, some sort of solution /will/ be there. That's what we're all working towards, right?
After looking at the poll results, and reading all comments on this
subject, what I think is...
Model syntax should be:
Class MyModel(models.Model)
my_field = CharField()
my_i18n_field = CharField()
[...]
class Meta:
[...]
translate = ('my_i18n_field', ...)
because it's what most people think is the most comfortable way to
specify it.
Then, we've to consider all comments regarding "unobstructive"
translation of reusable applications. While I agree with Marcin that
translating fields has nothing to do with Admin, it would be great
being able to do that. The perfect case for me would be just doing the
same as in file system based translations. The developer of the
reusable application is who marks all string for translation, even if
he/she is not going to translate them. Then the user of the
application is who decides if he will translate already marked
strings. Same way, where the author mark fields for translation
without any change for them would be great. But as far as I've checked
a good syntax that works for us, and don't break applications doesn't
exist. May be we could try to make django be patched to not break if a
translate attribute is found in Meta.
But anyway, I don't see any reason for not providing an additional way
to register translations with no change on the original application.
So what do you think about implementing both syntax and let the
developer to choose which one is better for each case?
Regards,
Marc
On Mar 10, 4:53 pm, Marcin Kaszynski <marcin.kaszyn...@gmail.com>
wrote:
> On Mar 10, 4:41 pm, Carl Meyer <carl.j.me...@gmail.com> wrote:
> > Can't we learn from that experience without
> > going through it again?
> Too late, we already went through this :)
> There are some differences between DM and Admin. For starters, Admin
> does not have to interfere at all when you use models in your views
> while the multilingual layer does. There's a lot of stuff happening
> when Python creates your model class and even more stuff when that
> model is supposed to be multilingual.
> Adding the DM stuff from the outside, after the class is already
> built, is asking for more problems.
> Another point in the changes to admin interface was the idea of having
> multiple admin interfaces for the same model. This is something we
> won't need with multilingual layers.
