Translatable Architecture

[Michael Gall]

I don't want to sound like I'm beating up on Translatable here, cause it's a really tricky thing to get right, IMHO the approach of using a Decorator is the right way to do it. I've gotten to the stage with my exploration of wondering about the database architecture. Was there consideration into modifying the parent table and adding a Lang and an OriginalID column to the original table. Then, by overwriting the queries, DataObject::get can retrieve only the current language objects. The reason I say this, is due to the lack of support for 2 major parts of the SS data model. Many-many relationships and the DataObject extendability.

Let me focus on the extending of DataObjects. As far as I can tell the only way to provide support for it will be to create a DataObject_lang table for each and every subclass (with a table), not only that, but the translatable fields for all subclasses will always need to be defined when the extension is applied. The approach of adding a Lang column to the base table, would mean that support for all subclasses would effectively be supported straight away, as the added criteria to the DataObject::get will always to be the base table.

The next functionality that would need to be supported is the selecting of Translatable fields and subsequently the fallback to the original fields. This could be supported with arguments to the constructor of the Decorator as it is now, but I would suggest a blacklist of fields rather than a whitelist. Looking at the current implementation with SiteTree not translating a field appears to be the exception rather than the rule.


What are everybody's thoughts? I'm also really interested in hearing about the reasoning behind the current implementation, I don't think that writing a migration script would be that difficult either.

Cheers,

Michael


--
Checkout my new website: http://myachinghead.net
http://wakeless.net

[Sam Minnee]

I wasn't involved in the initial development of the translatable
system, Ingo had much more to do with it. Unfortunately, he is on
leave for 4 weeks! :-S

However, I think we can probably still have a fruitful discussion
about this. The model for Translatable was Versioned. I think that
the data model for Versioned has generally been pretty successful, and
I think that the use-cases of Translatable are sufficiently similar
for that to be a good justification for the current Translatable data-
model.

I think that my general thinking is that Translatable is definitely a
little rough around the edges and in need of some love, but I don't
think that changing the data model is going to fix the most important
problems.

Your approach could have merit; that is the way that we attacked the
subsites module. However, the difference between subsite module and
the translatable module is that translatable has a much stronger tie
between versions of the same page in different languages. We have
struggled with that a bit in subsites - one can do it but we keep
running into bugs.

One thing that irks me a little is the fact that Translatable is so
heavily baked into the core. I would like to see Translatable spun
out as an add-on module. If we did that, then there would be more
room to experiment with alternative Translatable implementations.
Although I think that would have to be "experiment" in the sense of
"try a couple of alternatives and then determine a winner" rather than
"support 2 translatable implementations in the long term". ;-)

I would, however, prefer to look at debugging the Translatable model
with its existing data model before radically changing it. It would
also be good to increase the test coverage of the current Translatable
system; this will make it much less risky to muck with the internals
in the way that you describe.

> Let me focus on the extending of DataObjects. As far as I can tell the only
> way to provide support for it will be to create a DataObject_lang table for
> each and every subclass (with a table),

Adding tables is cheap; so I don't see the fact that there are
additional tables as a significant issue.

> not only that, but the translatable
> fields for all subclasses will always need to be defined when the extension
> is applied.

This is a bigger issue. In general, we need to avoid situations where
the order that extensions are applied matters. The way to address
this issue would be to call loadExtraDBFields() on every extension and
then call augmentDatabase() on every extension. It looks like the
current implementation is structured in this way, so I'm not sure what
the problems are there.

I agree with what you say about using a blacklist rather than a
whitelist.

[Sam Minnee]

Hi Michael,

I got your comments on the IRC channel but didn't get a chance to
reply until after you had left.

You raised the point that the .yml fixtures made use of Data Objects,
rather than SQL insertions. This is correct, and I think that in 90%
of the test cases it's beneficial to do things that way. However,
sometimes you need a little more fine-grained control.

One way of doing this would be to allow for fixture files in
either .yml or .sql format. .yml fixtures would support the creation
of fixtures from DataObjects, and .sql fixtures would execute raw SQL
and therefore allow for anything.

What do you think?

[Michael Gall]

I modified sapphireTest to detect if the table exists and if it didn't, do it with insertions. You just run into problems with the many-many and one-many relations.

I'll post my patches on Monday or Tuesday.

[Ingo Schommer]

Hey Michael,

first of all, thanks for you elaborate posts and contributions :)

i18n+Translatable+translate.silverstripe.com was a 3-month/single-
person effort during GSOC07,
I think Bernat did a fantastic foundation there. More on the evolution
and original motivations
are documented on the GSOC wiki:
http://trac.silverstripe.com/gsoc/wiki/GSoc07i18n
http://trac.silverstripe.com/gsoc/wiki/i18nConcepts
http://trac.silverstripe.com/gsoc/wiki/i18nImplementation
http://trac.silverstripe.com/gsoc/wiki/i18nMultilingualSupport

The last link also contains a pro/con list of different storage
solutions that you adressed:
http://trac.silverstripe.com/gsoc/wiki/i18nMultilingualSupport#Storagealternatives.Prosandcons

A "Lang" column on the base table means you have multiple rows for the
same record in different languages, right?
Even if this could be handled opaquely by DataObject::get(), this
means that *any* manual querying will get more complicated, and heaps
of existing implementations will break (because you now have to add a
GROUP BY or WHERE lang=... statement).

Correct me if I'm wrong, but isn't many-many possible with the current
translatable datamodel? You relate the two records on the base-table,
and for those two records the appropriate translation is selected (if
existing).
The only thing missing is translation of many_many_extrafields. I've
created a ticket: http://open.silverstripe.com/ticket/2793

Mostly agreed on the blacklist, makes specification a lot easier. It
gets a bit tricky when selecting which fields should be automatically
translated though. Everything text-based is an obvious choice, but how
about Int fields? Foreign keys might sometimes be language-specific,
but mostly not? Whats the default then, and how can we override with
just a blacklist?

> One thing that irks me a little is the fact that Translatable is so
heavily baked into the core. I would like to see Translatable spun
out as an add-on module.

Good point, that would tie in nicely with the intermediary objects
between
DataObject::get(), DataObjectDecorators and plaintext modifications of
SQLQuery right?

Michael, what have your experiences been with making
DataObjectDecorators translatable?
Its been a while since I played around with Translatable
unfortunately...

Greets
Ingo

[Michael Gall]

Hey guys,

I've put an initial patch of changing the Database schema at 2778, I'm currently working on adding hooks and stuff so it's far more removed from the core.

I can't access those pages on trac.silverstripe.com, I don't have a username/password

A "Lang" column on the base table means you have multiple rows for the
same record in different languages, right?
Even if this could be handled opaquely by DataObject::get(), this
means that *any* manual querying will get more complicated, and heaps
of existing implementations will break (because you now have to add a
GROUP BY or WHERE lang=... statement).

Yes, but you can't really do this manual querying on the translated versions of stuff at all at the moment. I've just done a quick audit and I don't believe any queries currently in Silverstripe would break.

Correct me if I'm wrong, but isn't many-many possible with the current
translatable datamodel? You relate the two records on the base-table,
and for those two records the appropriate translation is selected (if
existing).
The only thing missing is translation of many_many_extrafields. I've
created a ticket: http://open.silverstripe.com/ticket/2793

Mostly agreed on the blacklist, makes specification a lot easier. It
gets a bit tricky when selecting which fields should be automatically
translated though. Everything text-based is an obvious choice, but how
about Int fields? Foreign keys might sometimes be language-specific,
but mostly not? Whats the default then, and how can we override with
just a blacklist?

I don't really know what you mean by automatically translated. 

[Ingo Schommer]

On Sep 15, 2:31 am, "Michael Gall" <mic...@wakeless.net> wrote:
>
> I've put an initial patch of changing the Database schema at 2778, I'm
> currently working on adding hooks and stuff so it's far more removed from
> the core.

Is that revision 2778 in your repository? Whats the URL?

>
> I can't access those pages on trac.silverstripe.com, I don't have a
> username/password

Ah sorry, forgot that its basicauth protected - i've sent you the
documents through email.

>
> A "Lang" column on the base table means you have multiple rows for the
>
> > same record in different languages, right?
> > Even if this could be handled opaquely by DataObject::get(), this
> > means that *any* manual querying will get more complicated, and heaps
> > of existing implementations will break (because you now have to add a
> > GROUP BY or WHERE lang=... statement).
>
> Yes, but you can't really do this manual querying on the translated versions
> of stuff at all at the moment. I've just done a quick audit and I don't
> believe any queries currently in Silverstripe would break.

The difference is that you can still do normal i18n-unaware queries
when storing the additional language information in separate tables.
So basically every DB::query('SELECT... will break with your approach
when i18n is enabled as far as i see.

>
>
> > Mostly agreed on the blacklist, makes specification a lot easier. It
> > gets a bit tricky when selecting which fields should be automatically
> > translated though. Everything text-based is an obvious choice, but how
> > about Int fields? Foreign keys might sometimes be language-specific,
> > but mostly not? Whats the default then, and how can we override with
> > just a blacklist?
>
> I don't really know what you mean by automatically translated.

Like, a blacklist implies that you just specify fields you explicitly
*don't* want to have translated right?
In the reverse this means the system has to find out which fields *do*
need to be translated -
hence my question about this process.

[Michael Gall]

> I've put an initial patch of changing the Database schema at 2778, I'm
> currently working on adding hooks and stuff so it's far more removed from
> the core.

Is that revision 2778 in your repository? Whats the URL?

Ticket 2778. There is a patch attached. 

> A "Lang" column on the base table means you have multiple rows for the
>
> > same record in different languages, right?
> > Even if this could be handled opaquely by DataObject::get(), this
> > means that *any* manual querying will get more complicated, and heaps
> > of existing implementations will break (because you now have to add a
> > GROUP BY or WHERE lang=... statement).
>
> Yes, but you can't really do this manual querying on the translated versions
> of stuff at all at the moment. I've just done a quick audit and I don't
> believe any queries currently in Silverstripe would break.

The difference is that you can still do normal i18n-unaware queries
when storing the additional language information in separate tables.
So basically every DB::query('SELECT... will break with your approach
when i18n is enabled as far as i see.

Not really, looking at how Heirarchy.php works, it calls $this->owner->extend('augmentSQL') which adds the necessary Where statement.

> > Mostly agreed on the blacklist, makes specification a lot easier. It
> > gets a bit tricky when selecting which fields should be automatically
> > translated though. Everything text-based is an obvious choice, but how
> > about Int fields? Foreign keys might sometimes be language-specific,
> > but mostly not? Whats the default then, and how can we override with
> > just a blacklist?
>
> I don't really know what you mean by automatically translated.

Like, a blacklist implies that you just specify fields you explicitly
*don't* want to have translated right?
In the reverse this means the system has to find out which fields *do*
need to be translated -
hence my question about this process.

Currently the implementation I'm working on translates every DB field on the table, except the blacklist. It's working quite well as far as I can see.
 

[Ingo Schommer]

I've created a feature branch for this development effort, and am
coordinating with Michael by email. If you want to keep track of the
progress:
http://open.silverstripe.com/browser/modules/sapphire/branches/translatable
http://open.silverstripe.com/browser/modules/cms/branches/translatable
http://open.silverstripe.com/browser/modules/translatable/trunk