Call to arms: suggestions needed

[Marcin Kaszynski]

Hi everyone,

I need your help with a design decision.

One of the most requested, yet small features is support for
translation fallbacks. Take, for example, the following structure:

class Article(models.Model):
class Translation(multilingual.Translation):
title = models.CharField(verbose_name=_("The title"),
blank=True, null=False,
max_length=250)
contents = models.TextField(verbose_name=_("The contents"),
blank=True, null=False)

With three languages (pl, en, de) the following data:

Article:
title_pl: "artykul 1"
content_pl: "tekst 1"
title_en: "article 1"
content_en: ""
(no "de" version)

The goal is to be able to request "give me title and content in
English, but if it is not available, then any other version will do".

So, for example:

>>> a = Article.objects.get(id=1)

# this is current behaviour
>>> (a.title_pl, a.title_en, a.title_de)
("artykul 1", "article 1", None)

# this is what we want
>>> (a.title_pl_any, a.title_en_any, a.title_de_any)
("artykul 1", "article 1", "artykul 1")
>>> (a.content_pl_any, a.content_en_any, a.content_de_any)
("artykul 1", "article 1", "artykul 1")

# same for the default getter:
>>> Article.objects.for_language("de").get(id=1).title_any
"artykul 1"

So, there are two points that need decisions and suggestions:

1. syntax for the fallback

a) article.title, article.title_pl - adding fallbacks to current
getters

Pro: short, simple
Cons: blurs the distinction between hard data and
fallbacks. I feel that title and title_pl should
return real data, not something derived from it.

Also, this might suggest that these names behave
like this also in filtering:

Article.objects.filter(title__contains="artikel")

Which would not be true.

b) article.title_any, article.title_pl_any

Pro: still short and readable
Cons: a bit less readable for title_pl_any, but
combinations like this should be rare in
real-world source code.

It also means a possible name conflict with
a field called title_any, but we already have
something similar with title_en and so on, so
it's not really a problem.

c) article.title_fb, article.title_pl_fb ("FallBack")

Same as above. Perhaps less readable and obvious.

d) article.any.title, article.any.title_en

Pro: short, very readable for any.title
Also, it encloses all the new names in a single
attribute (any) which lowers the chance of name
collisions.
Cons: slightly less readable and obvious for any.title_en

Any suggestions? Votes? Other, more readable possibilities?

2. Fallback granularity

Should the fallback be per-instance or per-field? In the
above example, how should this behave:

>>> article = Article.objects.for_language("en").get(id=1)

# variant a)
>>> (article.title_en_any, article.content_en_any)
("article 1", "")

# variant b)
>>> (article.title_en_any, article.content_en_any)
("article 1", "tekst 1")

I think that a) makes more sense and so that one gets my vote
right now, but I know on some occasions the second variant
might be useful. So, the question is what is more useful
(saves more time) in the sites you are building with DM.

Cheers,
-mk

[x0nix]

Hi,

I would vote for 1b, 2b.

If I understand it right, this is behaviour of 2a?

>>> (article.title_en_any, article.content_en_any)
("article 1", "")

but just changing order would fundamentally change the output:
>>> (article.content_en_any, article.title_en_any)
("tekst 1", "artykul 1")
If it is so, I don't like it much.

Also I would like fallback languages to be configurable in settings
(Sometimes instead of fallback to some exotic language I'd prefer
empty string).

Just one other suggestion for 1), but not sure about it myself :-)
article.title.any()
article.title_pl.any(fallback=('en', 'de')) # is this useful? Maybe
not.

Regards

Jakub

[Marcin Kaszynski]

Hi,

thanks a lot for ideas.

On Mar 5, 2:19 pm, x0nix <x00...@gmail.com> wrote:
> If I understand it right, this is behaviour of 2a?>>> (article.title_en_any, article.content_en_any)
>
> ("article 1", "")
>
> but just changing order would fundamentally change the output:>>> (article.content_en_any, article.title_en_any)
>
> ("tekst 1", "artykul 1")
> If it is so, I don't like it much.

Neither would I. Which is why the order does not matter, in the
second case you will get the same data, just reversed.

> Also I would like fallback languages to be configurable in settings
> (Sometimes instead of fallback to some exotic language I'd prefer
> empty string).

Good point. I just thought about setting the sequence of languages,
but perhaps it should allow using a subset of languages. Something
like:

LANGUAGES = ['en', 'pl', 'de', 'ru']
TRANSLATION_FALLBACK_LANGUAGES = ['en', 'pl']

So that when asked for title_de_any DM would fall back to title_en or
title_pl, in this sequence, but not to title_ru.

> Just one other suggestion for 1), but not sure about it myself :-)
> article.title.any()
> article.title_pl.any(fallback=('en', 'de')) # is this useful? Maybe
> not.

Nah, I really don't want to mess with any data and code inside of the
object that is available as .title or .title_pl :)

-mk

[x0nix]

Ok, so "2a" would work this way?
article.content_en_any would return empty string, because there is
english translation object, but it's content field is empty
article.content_de_any would return "Tekst 1" (or possibly empty
string from content_en - depending on priority of pl/en)

Than it is important to make clear distinction (in admin interface)
between translation with empty fields and not existing translation.
(Part of "Issue 12: Create a better admin interface" ?)

Anyway now I agree that "2a" makes more sense.

TRANSLATION_FALLBACK_LANGUAGES is exactly what I had in mind. It
should be optional ... like this?
# languages.py
try:
TRANSLATION_FALLBACK_LANGUAGES =
settings.TRANSLATION_FALLBACK_LANGUAGES
except AttributeError:
TRANSLATION_FALLBACK_LANGUAGES = [language[0] for language in
settings.LANGUAGES]

Thanks

Jakub

[Marcin Kaszynski]

On Mar 5, 5:17 pm, x0nix <x00...@gmail.com> wrote:
> Ok, so "2a" would work this way?
> article.content_en_any would return empty string, because there is
> english translation object, but it's content field is empty
> article.content_de_any would return "Tekst 1" (or possibly empty
> string from content_en - depending on priority of pl/en)

Yes, precisely so.

> Than it is important to make clear distinction (in admin interface)
> between translation with empty fields and not existing translation.

Right. The interface was changed about two weeks ago to do that -- it
now allows you to save translations for a subset of languages and
distinguishes between empty and non-existant ones -- should it be even
more visible?

-mk

[x0nix]

Oh I am sorry, didn't notice that change. After update it's quite
visible ;-)

[obioma]

> I need your help with a design decision.
>

I vote for 1b), 2b)

[Joost Cassee]

On 04-03-09 18:25, Marcin Kaszynski wrote:

> I need your help with a design decision.

First of all, thanks for spending some more time on DM, Marcin.

My vote is for:

1b: we should not try to be too magical, explicit is better.
1a: I feel fallback on fields would make instances 'inconsistent'.

By the way, I agree with Jakub that fallback languages should be
configurable. Even more, I think the fallback language is
language-specific. For example, the fallback for 'fr-be' should be
'nl-be', but for 'fr-ca' it should be 'en-ca'.

Regards,

Joost

--
Joost Cassee
http://joost.cassee.net

[Marcin Kaszynski]

On Mar 7, 8:54 am, Joost Cassee <jo...@cassee.net> wrote:
> First of all, thanks for spending some more time on DM, Marcin.

You're welcome. I'm lucky to have most of that time paid for by
http://phaethon-designs.gr/ -- thanks a lot, Panos :)

> 1b: we should not try to be too magical, explicit is better.

> 2a: I feel fallback on fields would make instances 'inconsistent'.

Yeah, that is what I thought, but wanted to make sure it makes sense.
1b, 2a it will be.

> By the way, I agree with Jakub that fallback languages should be
> configurable. Even more, I think the fallback language is
> language-specific. For example, the fallback for 'fr-be' should be
> 'nl-be', but for 'fr-ca' it should be 'en-ca'.

It makes sense, but the configuration is getting complicated.

TRANSLATION_FALLBACK_LANGUAGES = {
'fr-be': ['nl-be', 'en'],
'fr-ca': ['en-ca', 'en'],
'': ['en', 'pl'], # for all the other languages
}

Any ideas for simpler solutions?

-mk

[Joost Cassee]

On 07-03-09 18:16, Marcin Kaszynski wrote:

>> By the way, I agree with Jakub that fallback languages should be
>> configurable. Even more, I think the fallback language is
>> language-specific. For example, the fallback for 'fr-be' should be
>> 'nl-be', but for 'fr-ca' it should be 'en-ca'.
>
> It makes sense, but the configuration is getting complicated.
>
> TRANSLATION_FALLBACK_LANGUAGES = {
> 'fr-be': ['nl-be', 'en'],
> 'fr-ca': ['en-ca', 'en'],
> '': ['en', 'pl'], # for all the other languages
> }
>
> Any ideas for simpler solutions?

Ha! I'd begun typing a similar example, but gave up and hoped you would
come up with something simpler. :-) This is complicated, I agree. So
what about this idea: automatically fallback to the default language
(LANGUAGE_CODE), but allow the developer to override the fallback by
providing a function. For example:

settings.py:
TRANSLATION_FALLBACK_LANGUAGES = 'somemodule.get_fallback_translation'

somemodule.py:
FALLBACK = [[l[0] for l in settings.LANGUAGES]
def get_fallback_translation(lang):
return FALLBACK

or

somemodule.py:
FALLBACK = {...}
def get_fallback_translation(lang):
return FALLBACK[lang]

On the other hand, this might be too much trouble. Maybe we can get away
with Jakub's proposal until anyone complains? (One could, of course,
test the TRANSLATION_FALLBACK_LANGUAGES variable type for a list or a
dict...)

By the way, we might want to start every DM setting with a common
prefix, for example MULTILINGUAL_.

[Marcin Kaszynski]

On Mar 7, 7:23 pm, Joost Cassee <jo...@cassee.net> wrote:
> On the other hand, this might be too much trouble. Maybe we can get away
> with Jakub's proposal until anyone complains?

Yeah. Lazy programming FTW :)

> (One could, of course,
> test the TRANSLATION_FALLBACK_LANGUAGES variable type for a list or a
> dict...)

Yeah, but the dict looks overly complex in this place. And it's
definitely too much work if noone actually needs it.

I'll go for the list right now and wait with more complex stuff until
someone proposes a nicer syntax. With a patch, hopefully :)

> By the way, we might want to start every DM setting with a common
> prefix, for example MULTILINGUAL_.

Agreed.

-mk

[Marcin Kaszynski]

Okay, it's done and in the repository.

The new setting is MULTILINGUAL_FALLBACK_LANGUAGES and should contain
a list of language codes. The syntax is title_any, title_de_any etc.
See testproject.fallback for samples.

These fallbacks are NOT available in database queries -- they are only
meant to get values from model instances, ie. in the templates.

Cheers,
-mk

[Fabio Corneti]

I'm experiencing some weird errors with the latest fallback stuff; the
issues seems to be caused
by the initialization of the FALLBACK_LANGUAGE_IDS variable in
languages.py, which is populated
using positional language indexes (get_language_idx) instead of
get_language_from_id_or_code.

I put a patch to the test in the group files which should demonstrate
the issue; let me know if this is correct or if
I simply misunderstood the new code.

Best,
Fabio Corneti
in...@corneti.com

[Fabio Corneti]

I have a website in which only part of the content is translated in
all the six languages
that the site features; in order to make fallbacks fully usable I'd
have to change every default
getter to its _any counterpart (for example .title should
become .title_any everywhere).

Wouldn't it be nice to add a setting to enable fallback by default on
default getters?
From what I can see it would affect only one line of code in
translation.create_translation_attrs;

217: setattr(main_cls, fname, TranslatedFieldProxy(fname, fname,
field))

would become something like:

217: setattr(main_cls, fname, TranslatedFieldProxy(fname, fname,
field, fallback=ENABLE_FALLBACK_ON_DEFAULT_GETTER))

Best,
Fabio Corneti
in...@corneti.com

On 9 Mar, 16:32, Marcin Kaszynski <marcin.kaszyn...@gmail.com> wrote:

[Marcin Kaszynski]

Hi,

On Mar 13, 11:43 am, Fabio Corneti <i...@corneti.com> wrote:
> using positional language indexes (get_language_idx) instead of
> get_language_from_id_or_code.

you are right, this is a bug in my code: I should use language IDs,
not indexes. Good catch!

Could you commit the fix?

-mk

[Marcin Kaszynski]

On Mar 13, 12:21 pm, Fabio Corneti <i...@corneti.com> wrote:
> Wouldn't it be nice to add a setting to enable fallback by default on
> default getters?

This goes against the "do no magic" philosophy, but I understand that
it would be useful in old code.

I think we could add a switch for that, as long as it's off by
default.
What do you guys think?

-mk

[Fabio Corneti]

Committed, I fixed another small issue in the meantime related to a
problem in the fallback
implementation when there are no translation for the instance (look at
the fallback tests
for the details).

Best,
Fabio Corneti
in...@corneti.com

[Fabio Corneti]

I have implemented the switch
(MULTILINGUAL_ENABLE_FALLBACK_ON_DEFAULT_GETTER) for my own use, it's
off by default.
You can have a look in my Github repo at:
http://github.com/fabiocorneti/django-multilingual/commits/fabiocorneti

Best,
Fabio Corneti
in...@corneti.com

[Semmel]

Hi all,

damn - i joined the discussion very late and most of it is already
implemented but it still have a word or two.

I currently use the fallback mechanism 1a for about 6 months now but i
would go with 1b since it's close to what we already have and is more
obvious then some _fb or .any stuff.
The fallback granularity is a bit tricky imho. I would say that a
makes more sense too (rather then b) but i think we should consider a
third alternative like so:

# variant c)
>>> (article.title_en_any, article.content_en_any)

("artykul 1", "tekst 1")

I some case it doesn't make sense to translate only some part of it or
leave it empty. In my case i use multilingual flatpages for many
descriptions on the website and it doesn't make sense to translate the
title but not the content (or vice versa).

So - to make things short i would vote for 1b and 2c/2a.
But for 2 - maybe this should be configurable?

One more thing about the fallback language(s) - it think they should
be configurable but it should default to just the default language
(DEFAULT_LANGUAGE setting).

> By the way, we might want to start every DM setting with a common
> prefix, for example MULTILINGUAL_.

Agreed. When we're already on the way changing this i would recommend
that we use MULTILINGUAL_LANGUAGES for our languages and let it
default to settings.LANGUAGES (that's what i do for about a year now).

> Wouldn't it be nice to add a setting to enable fallback by default on
> default getters?

In general i'm against the "magic stuff" but it definately makes sense
so an option (as long as we're not yet 1.0) would be great that - of
course - defaults to off.

And last but not least some comment to the fallback stuff. I agree
that language specific fallbacks do make some sense but i don't think
that you need such a complex solution. It's okay for me if you could
do that but the default setting should be very simple like a fallback
to the default language (as stated above).

Regards
Flo

[Semmel]

> The new setting is MULTILINGUAL_FALLBACK_LANGUAGES and should contain
> a list of language codes.  The syntax is title_any, title_de_any etc.

I just wondered why MULTILINGUAL_FALLBACK_LANGUAGES defaults to
settings.LANGUAGES? That doesn't make sense for me. One of the biggest
problems of dm is that it is using indexes everywhere and since i
can't always change the database tables to use different indexes i
ordered my languages so that change is unlikely to occur. So - long
story short - why not default it to just the DEFAULT_LANGUAGE? That
makes sense because the default is what you'll likely fall back to
(because it's your default - we already have that info) and it doesn't
have such a complex default so that it's dependant on the ordering of
your languages.

Suggestions anyone?

Regards
Flo