XBlocks and i18n

[Eugene Medvedev]

I already asked this before and got no clues, but maybe this time around... :)

So I'm writing an XBlock. I wish to i18n every single string in it, and if I want to distribute it for others to use, I will have to, because few people will want an XBlock hardcoded in Russian. The extant documentation mentions the use of i18n service, and there are some examples of this being done with ORA2, however, there are more than a few problems with doing it:

1. It is not at all clear where do the translated strings go.
2. How does one translate, say, display names of top level class members, like scoped variables? Because even if I define self._, I can't use it there.
3. It is even less clear if the translated strings ever get loaded without loading the XBlock as a Django application, which requires modifying the envs, which involves forking edX, which kind of defies the point of a plugin architecture.

What if the host is not a Django application, for that matter? Because if I'm reading the documentation right, XBlocks are meant to be a standard, and are meant to be Django-independent.

[Ned Batchelder]

Eugene,

You are right that the i18n infrastructure for XBlocks is not complete. We don't have a way for you to deliver translations (.po and .mo files), and we don't have a way for translations to be found at run time.  We'd be glad to talk about how that might happen and work with you on a contribution to make it work.  Is that something you'd be interested in?

--Ned.

--
You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/145d7495-c306-4805-afbb-046bcb8e22e6%40googlegroups.com.

[Eugene Medvedev]

On Monday, 23 March 2015 14:07:36 UTC+3, Ned Batchelder wrote:

You are right that the i18n infrastructure for XBlocks is not complete. We don't have a way for you to deliver translations (.po and .mo files), and we don't have a way for translations to be found at run time.  We'd be glad to talk about how that might happen and work with you on a contribution to make it work.  Is that something you'd be interested in?

If I'm even qualified to make one when I still can't wrap my head around writing tests properly, that would indeed be something I'd be interested in. :)

That said, here's a thought:

From looking at the Django sources, the only way I can envision this working is by altering settings.LOCALE_PATHS at the moment @XBlock.needs("i18n") to make sure that XBlock's locale files are included before the XBlock acquires the translation function. (Whether the XBlock supplies the locale files directly to the service in some way, or contains them in a predefined location is irrelevant, it would work the same.) Without that, replacing the entire translation framework (including templates!) would be required, which doesn't sound fun. If the host platform can be assumed to be Django, it sounds feasible.

Just how far the XBlocks must be platform-agnostic? Because if they really must be totally agnostic python-only, it's going to require a reimplemented translation framework or monkeypatching all over Django to implement which would effectively be the same.

[Ned Batchelder]

On Mon, Mar 23, 2015 at 11:14 AM, Eugene Medvedev <rn3a...@gmail.com> wrote:

On Monday, 23 March 2015 14:07:36 UTC+3, Ned Batchelder wrote:

You are right that the i18n infrastructure for XBlocks is not complete. We don't have a way for you to deliver translations (.po and .mo files), and we don't have a way for translations to be found at run time.  We'd be glad to talk about how that might happen and work with you on a contribution to make it work.  Is that something you'd be interested in?

If I'm even qualified to make one when I still can't wrap my head around writing tests properly, that would indeed be something I'd be interested in. :)

​Just having someone to discuss it with is a help! 

​

 

That said, here's a thought:

From looking at the Django sources, the only way I can envision this working is by altering settings.LOCALE_PATHS at the moment @XBlock.needs("i18n") to make sure that XBlock's locale files are included before the XBlock acquires the translation function. (Whether the XBlock supplies the locale files directly to the service in some way, or contains them in a predefined location is irrelevant, it would work the same.) Without that, replacing the entire translation framework (including templates!) would be required, which doesn't sound fun. If the host platform can be assumed to be Django, it sounds feasible.

​The thing to remember about XBlocks is that the runtime is free to implement the services any way it likes.  The edx-platform runtime can use whatever Django machinery it wants, since it is built on Django.

​

 

Just how far the XBlocks must be platform-agnostic? Because if they really must be totally agnostic python-only, it's going to require a reimplemented translation framework or monkeypatching all over Django to implement which would effectively be the same.

​The XBlocks API is platform-agnostic.  The edx-platform implementation of the API doesn't have to be.  But the instructions for how to package the translations in the XBlock should be platform-agnostic.

It might still be complicated to work the XBlock translations into the Django environment, but we don't have to reimplement the whole pipeline from scratch.  We should also make the translations work in the XBlock Workbench.

--Ned.

​

 

--
You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/a6271d0f-4558-4981-b6be-74c1cc4ca3c3%40googlegroups.com.

[Eugene Medvedev]

On Tuesday, 24 March 2015 00:38:55 UTC+3, Ned Batchelder wrote:

​The thing to remember about XBlocks is that the runtime is free to implement the services any way it likes.  The edx-platform runtime can use whatever Django machinery it wants, since it is built on Django.

That basically means any runtime except edX will need to reimplement a translation pipeline from scratch. :) Probably a templating service also, because every XBlock I've seen so far makes liberal use of Django templates, can't work without them, and therefore can't truly be platform-agnostic. Exposing mako as a service (or better yet, defining a self-contained templating service just for XBlocks to use) might be a good idea, the problem here is that templates contain the grand majority of strings requiring translation, so this would require some i18n service implementation too...

I'm pretty sure Django's own translation framework can be cheated by altering settings.LOCALE_PATHS at runtime, as long as django.utils.translations is imported after that happens, (knowing Django and it's magic, though, I'd have to experiment to be completely sure, but I'll try to set something up soon) and the actual location of the files can be anything, as long as they follow the pattern "<locale id>/LC_MESSAGES/django.mo" -- deviating from this pattern would involve either monkeypatching or rewriting. (The 'django' name is hardcoded, everything else is how GNU gettext likes it, and everyone on Earth uses gettext anyway.) Such an implementation seems easy enough to make, and I think it should also affect javascript translations, as long as settings.LOCALE_PATHS is altered globally. (django.views.i18n.javascript_catalog view checks it on every call, and the request would happen when javascript first runs. The translation service would need to supply the url to the XBlock for it to get used. Maybe this is better dealt with somewhere in javascript runtime? All XBlocks would want to use that, too...) Keeping the 'django' and 'djangojs' filenames hardcoded sounds a bit silly, but as long it's acceptable, it can be left as a problem for other implementations to circumvent with whatever translation machinery they have available. (Pylons has no hardcoded names, Flask-Babel actually consumes every .mo it finds and ignores the names -- so I guess the other Python frameworks will mostly be ok with it, Django is just the odd one out.)

The smoothest way would be to run this somewhere in the @XBlock.needs decorator, so that it fires early, giving a parameter to the service like so: @XBlock.needs("i18n",["path","path","path"]) -- i.e. explicitly tell the service where our locale files are, which would put the least restrictions on the internal structure of the XBlock. Which implies that the decorator needs to try to call something to initialise the service for a particular XBlock, which it doesn't currently do, and looking at the code in the XBlock repository, I'm not sure it should. I.e. it should probably work like this:

class WhateverBlock(XBlock):

    _ = None

    def __init__(self):

        i18n_service = self.runtime.service(self,"i18n")

        i18n_service.add_locale_files(["path","path","path"])

        self._ = i18n_service.ugettext

        # I'm not sure if def _(self) will work, because then there's no guarantee it doesn't race the LOCALE_PATHS change.

It might still be complicated to work the XBlock translations into the Django environment, but we don't have to reimplement the whole pipeline from scratch.  We should also make the translations work in the XBlock Workbench.

Workbench is also a Django application, so just about anything that works for edX will work for Workbench. :) In fact, it's probably easiest to experiment with all this hackery in Workbench and later apply the working solution to edX rather than the other way round.

...Except templating, that is. Workbench does not include mako. Come to think of it, that's why every XBlock uses django templates, isn't it -- after all, that's what the XBlock SDK examples do.

Looks like I've opened a can of worms. :)

[Calen Pennington]

To chime in on template rendering: My intention when designing XBlocks was that each block could make it's own choice on how to render templates (and that the default choice would be something like mako). The style of using `runtime.render_template` was a hold-over from edx-platform and XModules, and is something about the XModule design that I really didn't like. The workbench code that provides render_template even has a TODO marking it as something we were going to get rid of (https://github.com/edx/xblock-sdk/blob/master/workbench/runtime.py#L260-L265).

Workbench doesn't include mako, but it shouldn't have to. Any individual XBlock should just be able to put Mako into it's setup.py `requires` list, and then use the mako library to do its own rendering (and that makes it runtime independent, since any python runtime can easily have mako installed as a library, but that's not true for using django templating).

-Cale

--
You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/e0b9ba03-4f69-45cc-8d65-fc1ad2b81621%40googlegroups.com.

[Eugene Medvedev]

On Tuesday, 24 March 2015 20:32:56 UTC+3, Calen Pennington wrote:

Workbench doesn't include mako, but it shouldn't have to. Any individual XBlock should just be able to put Mako into it's setup.py `requires` list, and then use the mako library to do its own rendering (and that makes it runtime independent, since any python runtime can easily have mako installed as a library, but that's not true for using django templating).

The need to i18n it kind of makes it considerably more complicated, though. Sure, an XBlock can require Mako, but Mako by itself does not appear to have any provision for getting translations as far as I can see, you're supposed to define your own translation methods and implement them using whatever machinery the rest of your platform provides.

Either there is no i18n service and XBlock has to do everything by itself, including picking a templating engine and requiring it, (...what if two different XBlocks decide they want different specific versions of Mako though?...) or there is an i18n service, and then every XBlock needs to somehow feed it's translation functions to whatever templating engine it ends up using. That's quite a ways off from "batteries included". Templating is kind of one of the biggest batteries that one would need.

[Calen Pennington]

I'll admit, I haven't tackled i18n in mako before. If it's trickier than just passing in the i18n service provided by the runtime and calling it in the templates, then I think it would be great to have a standard XBlock that uses mako and provides internationalization support, so that XBlock authors don't need to roll their own.

In fact, even if it's just a matter of getting the i18n service and putting it into the mako context, I'd love to see a base XBlock that would provide that as a convenience.

-Cale

--

You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/0129cb71-5882-45ce-9259-e2f38b38884c%40googlegroups.com.

[Eugene Medvedev]

On Tuesday, 24 March 2015 21:16:19 UTC+3, Calen Pennington wrote:

I'll admit, I haven't tackled i18n in mako before. If it's trickier than just passing in the i18n service provided by the runtime and calling it in the templates, then I think it would be great to have a standard XBlock that uses mako and provides internationalization support, so that XBlock authors don't need to roll their own.

In fact, even if it's just a matter of getting the i18n service and putting it into the mako context, I'd love to see a base XBlock that would provide that as a convenience.

edX's own templates typically do it like this:

<%! from django.utils.translation import ugettext as _ %>

I'm not sure how to pass a service to a mako template in the first place, actually -- I don't really know how mako works, but from what I can see, it actually compiles the template to python code and then runs it, so passing it a function as an object could turn out to be rather complicated.

As a side note, an XBlock could implement it's own i18n machinery, (gettext is part of the standard library, and Babel should take care of the other needs) -- but that brings it's own questions. (How would it know which language is currently active, and would the platform's language setting affect whatever the XBlock does?) 

[Eugene Medvedev]

On Wednesday, 25 March 2015 12:54:38 UTC+3, Eugene Medvedev wrote:

Suddenly, a consideration that strongly suggests a templating service is the way to go, regardless of i18n considerations, was prompted by a ticket I just got.

Theming.

As it is, the XBlock is so self-contained, there is no way to impose any consistent style, but a central templating service could inject style overrides at least.

[Calen Pennington]

Most of the edX templates are for XModules (the predecessor to XBlock), and are thus a bad example to base XBlocks on (we're working to migrate all of the XModules to the XBlock apis, but it's slow going).

Passing `_` to a mako template could be as simple as `template.render(_=self.runtime.service('i18n').ugettext)`. The compilation is just from mako syntax to a python function, so passing in arguments to that is as easy as passing them in to any other function.

-Cale

--

You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/030d72fa-d2d9-40c3-9260-c345e47ae491%40googlegroups.com.

[Calen Pennington]

When I've talked with our in-house UX folks, the conclusion that I think we'd come to was that one way to do theming would be to publish an XBlock css guide, so that XBlocks could use a well known set of semantic styles, and then expect the surrounding runtime to supply css definitions that would theme the block. That said, we're still working on simply making edx-platform themable, so we haven't dug in to what it will take to really make XBlocks follow the same set of themes.

I don't think that a templating service gives you more flexibility to override styles than having the runtime modify the returned Fragment does. How would you envision the actual style overrides/injections working, if we did have a templating service?

-Cale

--

You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/e5eff58f-5bb2-4434-91f8-8d9282b34d42%40googlegroups.com.

[Eugene Medvedev]

On Wednesday, 25 March 2015 14:52:38 UTC+3, Calen Pennington wrote:

I don't think that a templating service gives you more flexibility to override styles than having the runtime modify the returned Fragment does. How would you envision the actual style overrides/injections working, if we did have a templating service?

A templating service would be the natural place to apply any modifications, possibly even before a template is rendered, rather than after, which I'd think would be more flexible. You could actually override the template files themselves, for one.

Though to be honest, I'm not sure, "theming" tends to mean very different things in different contexts, and while in some environments a theme encompasses most of the frontend, including the entire template, in others it's just replacing styles and in yet others it's only injecting colors. Where does, in fact, edX want to be on this scale, considering how much of it is javascript and won't work without? While the existing theming options lean towards the front of that scale, I can tell you that for me theming is my primary method of hacking the way edX actually works (Many of my tickets consist of "disable this feature immediately!", for some, the quickest way to do it is hacking it out of the template, and theming saves me from having to touch the platform itself.) so I naturally apply this thinking to XBlocks as well, when it's probably not the way to go for everyone else.

And for that matter, just how many batteries should be included in the XBlock runtime? Considering that it has to share it's virtualenv with everything else in edX, I'd say it's better to include more rather than less, before some XBlock decides to require version A of package B while another one requires version C, but I'm probably not aware of the entire logic behind XBlocks.

[Calen Pennington]

That's an interesting point around overriding the templates than an XBlock uses. By way of analogy, Django has templates included in apps, but allows the project to override those templates. I'll have to think about that. It may be worth forcing XBlock authors to use a single templating system to allow that sort of override.

There's a longer arc around XBlocks where we'll be pulling them out and running them out-of-process (and thus, out-of-virtualenv), to allow us to be more flexible with allowing course authors to upload/deploy their own XBlocks with less concern about them clashing with each other. But how close that is... that's a bit harder to say.

-Cale

--
You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/a2a028e6-fda5-43e9-a435-0f2b9d388eea%40googlegroups.com.

[new user]

Hi,

Is there a solution to this issue now, how can we make the xblocks template files check its own translation files rather than looking into edx-platform translation files?