Model translation and meta options

[Claude Paroz]

Dear developers,

I'd like to suggest an addition to the Django Model definitions to ease the implementation of translatable model fields. There are currently several third-party applications for model translation, and even if I think we are not ready yet to bless one of them to be included in core Django, I would suggest the following features which could then be used by several different applications (or custom one).

1. Add a `translatable` Meta option with a list/tuple/set of translatable field names.
2. Add a `translatable_fields` cached property to meta Options classes to get a list of a model translatable field instances.

Each app could then use those elements to implement its own version of the model translation functionality. The problem with the current situation is that custom or third-party apps have to find not-always-pretty workarounds to mark which field of a model can be translated. It's a bit unusual to introduce data structures in Django which have no direct use by Django itself, but I think this could help the Django ecosystem in general.

I'm looking forward to your feedback.

Claude

[Raphael Michel]

Hi Claude,

I spent some time looking at the implementations out there and in one
of my projects I'm running a custom one, that uses yet another approach
(that I plan to release as a library some day).

It is not only that we are not yet ready to bless one of them, I have a
feeling that we never will, as they are incredibly different from each
other.

django-parler, django-hvad and django-model-i18n use seperate tables to
store the data, which is cleaner from a database point of view but
requires more JOINs. The former two use custom model base classes, the
latter uses a registration-based approach.

django-model-translation uses seperate columns for the different
languages, thus working with a "static" set of languages and
requiring a migration if you add translation capabilities or
change the set of languages. On the other hand, it saves you a lot of
JOIN queries. It uses a registration-based approach to register models
with the system.

My project uses only a custom field type that stores the translated
contents in a JSONField-type thing. This has some downsides with
regards to indexing and searching (at least on non-Postgres) and is
less clean from the DB side but requires neither migrations for new
langages nor any additional JOINs. (And I believe it is very clean on
the Django-side as it just requires you to replace a CharField with an
I18nCharField, no custom base classes, no registration, etc.).

Each of those approaches has severe advantages and disadvantages and
more importantly, they are fundamentally different. I don't believe a
new meta option would help you in any way, as for some of them you'd
probably *still* need a custom model base class.

It also would not allow you to easily switch between them or have a
reusable app to be compatible with multiple of them because it would
still be very undefined how you get to the translated versions in the
different systems, how you store the data into the field, etc.

Cheers
Raphael

[Claude Paroz]

Hi Raphael,

Le 14. 02. 17 à 14:53, Raphael Michel a écrit :

> Hi Claude,
>
> I spent some time looking at the implementations out there and in one
> of my projects I'm running a custom one, that uses yet another approach
> (that I plan to release as a library some day).
>
> It is not only that we are not yet ready to bless one of them, I have a
> feeling that we never will, as they are incredibly different from each
> other.

That could be totally true, at least that what history showed us until
now. I have not totally abandoned the idea of the "ideal" app yet, but
we are still far from that, and as you suggest we might never reach that
point due to various use cases.

(...)

> Each of those approaches has severe advantages and disadvantages and
> more importantly, they are fundamentally different. I don't believe a
> new meta option would help you in any way, as for some of them you'd
> probably *still* need a custom model base class.

Sure, custom model classes would still be needed much of the time to
exploit the new meta fields. Avoiding that was not my idea.

> It also would not allow you to easily switch between them or have a
> reusable app to be compatible with multiple of them because it would
> still be very undefined how you get to the translated versions in the
> different systems, how you store the data into the field, etc.

Absolutely, and neither was that my objective. My proposal is only aimed
to offer a cleaner way for those apps to declare translatable fields on
models, nothing more.

Claude
--
www.2xlibre.net

[jp...@yourlabs.org]

Django itself includes excellent i18n features for everything, except for model data: but this can be improved, even in a multi-milestone fashion starting with Claude's feature idea which is why I support it and I will restrain myself from talking about internal implementation details which continue their lives in the wild for now.

However, there's a third alternative to changing Meta that may be worth suggesting: adding a new Field attribute ie. translatable=True, which could be default for CharField and TextField perhaps. It might even turn to be an advantage that no existing translation app supports it that way.

[Patryk Zawadzki]

I think the proposed fields are mostly useful for translation utilities that modify your schema on the fly. I consider these to be the least clean solutions as they mean none of the translated fields can be marked as required. For other solutions (depending on separate tables and registration) there's no reason for the field list to live inside the parent model.

For anyone curious we're implementing translations as a regular Django model with a ForeignKey to the main model. The parent has a descriptor attribute that implements the language selection logic, the rest is a matter of doing proper joins or prefetches. Example:

from django.utils.translation import get_language


class TranslationWrapper(object):
    def __init__(self, instance, locale):
        self.instance = instance
        self.translation = next(t for t in instance.translations.all() if t.locale == locale, None)

    def __getattr__(self, item):
        if self.translation is not None and hasattr(self.translation, item):
            return getattr(self.translation, item)
        return getattr(self.instance, item)


class TranslationProxy(object):
    def __get__(self, instance, owner):
        locale = get_language()
        return TranslationWrapper(instance, locale)


class Foo(models.Model):
    # ...
    translated = TranslationProxy()

[Claude Paroz]

Le 14. 02. 17 à 16:16, jp...@yourlabs.org a écrit :
(...)

> However, there's a third alternative to changing Meta that may be worth
> suggesting: adding a new Field attribute ie. translatable=True, which
> could be default for CharField and TextField perhaps. It might even turn
> to be an advantage that no existing translation app supports it that way.

Yes, thanks for bringing this up. Defining the translatability of fields
at the field or model level should be discussed. I don't feel very
strongly about that.
Putting it at the model level can be nice because it centralizes it in
one point, but it might technically more appropriate in the field
definition (read also Patryk post).

Claude
--
www.2xlibre.net

[Claude Paroz]

Le 14. 02. 17 à 16:31, Patryk Zawadzki a écrit :

> I think the proposed fields are mostly useful for translation utilities
> that modify your schema on the fly. I consider these to be the least
> clean solutions as they mean none of the translated fields can be marked
> as required. For other solutions (depending on separate tables and
> registration) there's no reason for the field list to live inside the
> parent model.

As proposed by jpic, I could live with a translatable property at the
field level. We could also leave out the translatable_fields meta
property because it could be done rather easily with
[field for field in meta.get_fields() if field.translatable]

At this stage, I'd like to be as implementation-independent as possible.

> For anyone curious we're implementing translations as a regular Django
> model with a ForeignKey to the main model. The parent has a descriptor
> attribute that implements the language selection logic, the rest is a
> matter of doing proper joins or prefetches. Example:

Thanks for your example, very instructive.

Claude
--
www.2xlibre.net

[Patryk Zawadzki]

W dniu wtorek, 14 lutego 2017 16:19:48 UTC+1 użytkownik jp...@yourlabs.org napisał:

However, there's a third alternative to changing Meta that may be worth suggesting: adding a new Field attribute ie. translatable=True, which could be default for CharField and TextField perhaps. It might even turn to be an advantage that no existing translation app supports it that way.

I'd vote against adding non-database-related attributes to model fields. I actually hope one day Django will drop some of the existing ones that are only used by admin (like `editable`).

[Adam Johnson]

Sorry for the terrible pun here, but I'd like to suggest the meta-feature... allowing 3rd party apps to add their own options to Meta classes. If there was a sensible API for this (or if Django just copied all attributes defined in Meta onto _meta blindly), Django wouldn't have to add the translatable option without defining its behaviour.  Instead the third party apps could all define their own and use that.

Another use case I've thought about for custom Meta options is allowing models to define their MySQL storage engines or other MySQL specific table options. Again something where you want to attach data per model but without polluting the core Options class.

Currently third party apps working with custom model attributes can either rely on class-level attributes (not very Django-ey), or their own meta-style class (increasing noise in model definitions). Definitely Meta is the least surprising place to put these extra attributes, it would just be nice to have a way of doing this.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f58a8e66-e792-4468-91e6-7ca6e1b641c4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Tim Graham]

Adam, here's a ticket about allowing custom Meta attributes: https://code.djangoproject.com/ticket/5793. I guess it would be better to have a separate thread for that discussion.

On Tuesday, February 14, 2017 at 6:22:08 PM UTC-5, Adam Johnson wrote:

Sorry for the terrible pun here, but I'd like to suggest the meta-feature... allowing 3rd party apps to add their own options to Meta classes. If there was a sensible API for this (or if Django just copied all attributes defined in Meta onto _meta blindly), Django wouldn't have to add the translatable option without defining its behaviour.  Instead the third party apps could all define their own and use that.

Another use case I've thought about for custom Meta options is allowing models to define their MySQL storage engines or other MySQL specific table options. Again something where you want to attach data per model but without polluting the core Options class.

Currently third party apps working with custom model attributes can either rely on class-level attributes (not very Django-ey), or their own meta-style class (increasing noise in model definitions). Definitely Meta is the least surprising place to put these extra attributes, it would just be nice to have a way of doing this.

On 14 February 2017 at 15:58, Patryk Zawadzki <pat...@gmail.com> wrote:

W dniu wtorek, 14 lutego 2017 16:19:48 UTC+1 użytkownik jp...@yourlabs.org napisał:

However, there's a third alternative to changing Meta that may be worth suggesting: adding a new Field attribute ie. translatable=True, which could be default for CharField and TextField perhaps. It might even turn to be an advantage that no existing translation app supports it that way.

I'd vote against adding non-database-related attributes to model fields. I actually hope one day Django will drop some of the existing ones that are only used by admin (like `editable`).

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f58a8e66-e792-4468-91e6-7ca6e1b641c4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[James Pic]

On Wed, Feb 15, 2017 at 12:21 AM, Adam Johnson <m...@adamj.eu> wrote:

Sorry for the terrible pun here, but I'd like to suggest the meta-feature... allowing 3rd party apps to add their own options to Meta classes. If there was a sensible API for this (or if Django just copied all attributes defined in Meta onto _meta blindly), Django wouldn't have to add the translatable option without defining its behaviour.  Instead the third party apps could all define their own and use that.

I recon that was my initial thought too (this is also something we could use for ModelForms BTW). While that would also work, I still think that trying to make model data intl support as great as intl support everywhere else in Django is a good idea - but I'm a non English native so I guess I have strong feelings about this: without model intl I just can't use django to communicate with my people. I still /believe/ "Django admin should at least support translation of data, somehow, at some point". If we can have a consensus on the requirement itself then it will be easier I think to decide which first step to take to get there.