humanize middleware
jws
commas and such. I was about to add filters for dates when it occured
to me that it should reflect I18N to to show proper US/Euro dates, etc.
Before I spend time on it, can we discuss how this should work in
Django? I currently have no knowledge of I18N.
Radek Svarz
I18N is nicely described here: http://www.djangoproject.com/documentation/i18n/
Basically you need to mark any strings as _(original string), eg.:
return _('%.1f million') % (value / 1000000.0)
However there is one BIG issue with humanize - it pretty fits English,
but not other languages, that are more complex than English.
Eg. function ordinal - in English you get 1st, or 2nd, etc.
In Czech I can use either word representations: 1 = > first, 3=> third
etc. - which is not the meaning of this function, I believe. Or I can
use easy representation, such as:
1=> 1.
3 => 3.
In the latter case the ordinal function for Czech would look like:
def ordinal(value):
return '%d.' % (value)
For Czech, function intcomma could be used, we just use dot exchanged
with comma. This can be accomplished by the mentioned _(). And other
languages might be using even different separator (eg. space).
Therefore this function should be renamed. I propose to "intseparator"
or more exactly "thousandseparator".
intword is easy to be "translated" using _()
appnumber is not that easy though. In English you just use "two" for
anything that counts to 2. In Czech we differentiate by gender. For
example we say "dva muzi ~ two men" and "dve zeny ~ two women". Word
"dva" representing 2 changes.
Radek
Nebojsa Djordjevic
Why middleware?
Current I18N only uses middleware to get locale preference from the browser. Rest of the functionality are in template
tags and gettext family of functions.
IMHO, if we already used gettext convention for i18n we can use the same for l10n (localization) (use LC_MONETARY and
other to store number, datetime and other locale dependent formats).
We should start by compiling list of all necessary functionality that must be included and the start on the
implementation (somebody more knowledgeable than me in english can put this on the wiki).
Basic functionality:
1. Number formating - format numbers according to current locale (decimal and thousands separator)
* change validators accept numbers which are in current locale format
(i.e. to accept 1 000 000,658 same as 1000000.658)
this will require changes through SQL generation functions and validators
(or we can wait to validators go away and work inside field itself)
2. Money formating - based on number format but with currency display
3. Locale aware date and time formatting (currently we have some of that functionality but it's far from complete)
Any other?
--
Nebojša Đorđević - nesh
Studio Quattro - Niš - SCG
http://studioquattro.biz/
http://djnesh.blogspot.com/ | http://djnesh-django.blogspot.com/ | http://trac.studioquattro.biz/djangoutils/
Registered Linux User 282159 [http://counter.li.org]
GinTon
Nebojsa Djordjevic wrote:
>
> We should start by compiling list of all necessary functionality that must be included and > the start on the implementation (somebody more knowledgeable than me in english can
> put this on the wiki).
Hi, I'm working in a full localization:
http://groups.google.com/group/django-ecommerce/browse_thread/thread/ed90b6f909964621/bc9a8aee68eea058#bc9a8aee68eea058
I have finished tables about countries, subdivisions, and time zones:
http://satchmo.python-hosting.com/browser/trunk/satchmo/data/
Now I'm working with the languages table, and the next tables will be:
* date format
* Phone
* Postal code (RE for each country)
* Currency
* City line format (for correct mail address)
GinTon
http://satchmo.python-hosting.com/browser/trunk/satchmo/localization/
and the program for prepopulate all those data is:
http://satchmo.python-hosting.com/file/trunk/satchmo/load_localization.py
Nebojsa Djordjevic
> Now I'm working with the languages table, and the next tables will be:
>
> * date format
Well, date and currency (and IIRC number) formats are defined in LC_MONETARY, LC_NUMERIC, LC_DATE and LC_TIME on most of
the *nix systems anyway. So why not using that? We already agreed to use gettext for text storage.
I think that (like i18n implementation) we can use locale.py from standard python distribution as a base and work from
there (current python locale implementation is not usable because is not thread safe).
With that (at least for numbers, date and time) we can just copy LC_* to locale/<lang>/ dir.
For start let's compile some requirements for l10n, then chose storage format and implementation.
p.s. I'll post link to this thread in django-devel/user just to attract more opinions.
GinTon
Nebojsa Djordjevic wrote:
> GinTon wrote:
> > Now I'm working with the languages table, and the next tables will be:
> >
> > * date format
> > * Currency
>
> Well, date and currency (and IIRC number) formats are defined in LC_MONETARY,
> LC_NUMERIC, LC_DATE and LC_TIME on most of
> the *nix systems anyway. So why not using that? We already agreed to use gettext for
> text storage.
>
1) Django could be running into a system different from *unix as
Windows, and I don't know if MacOs have those definitions (I'm supposed
that yes, because is *unix).
2) The user could have not defined those variables.
GinTon
Nebojsa Djordjevic
> With that, there are 2 problems:
>
> 1) Django could be running into a system different from *unix as
> Windows, and I don't know if MacOs have those definitions (I'm supposed
> that yes, because is *unix).
It's like that we use now with gettext. We *will* replicate LC_* files in the django/conf/locale/<lang>/ dir for
supported locales so windows users will not be left out.
> 2) The user could have not defined those variables.
These are files just like django.po or djangojs.po not variables and it will be treated like django.po (loaded when
thread is started).
GinTon
Nebojsa Djordjevic wrote:
> GinTon wrote:
> > With that, there are 2 problems:
> >
> > 1) Django could be running into a system different from *unix as
> > Windows, and I don't know if MacOs have those definitions (I'm supposed
> > that yes, because is *unix).
>
> It's like that we use now with gettext. We *will* replicate LC_* files in the django/conf/locale/<lang>/ dir for
> supported locales so windows users will not be left out.
>
Uff! I see this very dirty. I think that is best use a data base
related a countries table that have been already created.
So when a user chooses where is from, the application can show him data
about its local date, time, etc and whatever data related to its
country.
Nebojsa Djordjevic
>
> Nebojsa Djordjevic wrote:
>> GinTon wrote:
>>> With that, there are 2 problems:
>>>
>>> 1) Django could be running into a system different from *unix as
>>> Windows, and I don't know if MacOs have those definitions (I'm supposed
>>> that yes, because is *unix).
>> It's like that we use now with gettext. We *will* replicate LC_* files in the django/conf/locale/<lang>/ dir for
>> supported locales so windows users will not be left out.
>>
> replicate LC_* files?
>
> Uff! I see this very dirty. I think that is best use a data base
> related a countries table that have been already created.
>
Why database? LC_* replication is only for users which will not have LC_* stuff and to allow translators to work on
them. Also, we can reuse most of standard locale.py if we choose LC approach.
For me it even dirtier that I'll have <locale tables> * <django project> installed instead of replicating (approx 30)
LC_* files inside django/conf/locale.
Localization data will not be changed often so let's make it accessible to all installed django projects without need
for per project installation.
IMHO static data (like base translations, l10n stuff, countries, ...) should be stored into files not in the database.
> So when a user chooses where is from, the application can show him data
> about its local date, time, etc and whatever data related to its
> country.
Just add call to (IIRC) django.utils.set_locale(locale) after selection (if user wants that - for example I often select
my country and I'll still wish to see content in the English (ie. translation sucks ;) ) ).
Nebojsa Djordjevic
GinTon
Nebojsa Djordjevic wrote:
> Why database? LC_* replication is only for users which will not have LC_* stuff and to allow translators to work on
> them. Also, we can reuse most of standard locale.py if we choose LC approach.
>
> For me it even dirtier that I'll have <locale tables> * <django project> installed instead of replicating (approx 30)
> LC_* files inside django/conf/locale.
>
> Localization data will not be changed often so let's make it accessible to all installed django projects without need
> for per project installation.
>
> IMHO static data (like base translations, l10n stuff, countries, ...) should be stored into files not in the database.
>
drop list for that he chooses its country. In addition is possible
filter those data by geographical region or inclusive by languages when
I have finished those table. You can not make this using files.
So an administrator could show in its Django project only the countries
related to North America or countries with germany language only. There
is a field called 'display' for this proposal.
Also is possible show to the user in the first the geographical regions
and then he choose its country form that region. It's very heavy choose
your country from a log list (250 countries more or less)
And as the main table goes in the data base, the rest of data also
because are data related to that table. In addition there are some data
that could need a Many-to-many relationship as languages and countries
and this is easy with the data base.
jws
>
> Current I18N only uses middleware to get locale preference from the browser. Rest of the functionality are in template
> tags and gettext family of functions.
contrib.humanize is Arian's code. I just wanted this functionally for
my current project, so I made a crude patch in my local code. As I
said, I know little about L10N or I18N, so it needs to discussed on
this list to find the correct solution.
Nebojša Đorđević
> If you have the countries in a data base is possible show to the
> user a
> drop list for that he chooses its country. In addition is possible
> filter those data by geographical region or inclusive by languages
> when
> I have finished those table. You can not make this using files.
> So an administrator could show in its Django project only the
> countries
> related to North America or countries with germany language only.
> There
> is a field called 'display' for this proposal.
>
> Also is possible show to the user in the first the geographical
> regions
> and then he choose its country form that region. It's very heavy
> choose
> your country from a log list (250 countries more or less)
>
> And as the main table goes in the data base, the rest of data also
> because are data related to that table. In addition there are some
> data
> that could need a Many-to-many relationship as languages and countries
> and this is easy with the data base.
Well, we are mixing scopes here. L10N is primary based on *locale*
settings which don't have to be related to the country. If user is
already selected language why don't we use that and present all the
relevant data (dates, numbers, ...) in the *locale* format. I
deliberately left out currency, tracking and updating exchange rates
will be overkill for most of the django sites.
This proposal is for *core* L10N functionality (just like I18N) - so
let's add number and date/time formatting for start (any other
ideas?) and build other I18N and L10N functionality on top of that.
Also, for date/time django already does a great job. I only wish to
move DATE_FORMAT and similar from django.po into LC_* files.
I will soon start to work on something like I described for my
current project (and yes, I will also need the currency support). I
don't want to patch django core if I don't have to so I'll try to
make this a nice reusable application if it's possible.
--
Nebojša Đorđević - nesh
Studio Quattro - Niš - Serbia
GinTon
Nebojša Ðordevic wrote:
> On 24 Jul 2006, at 14:44, GinTon wrote:
> Well, we are mixing scopes here. L10N is primary based on *locale*
> settings which don't have to be related to the country. If user is
> already selected language why don't we use that and present all the
> relevant data (dates, numbers, ...) in the *locale* format. I
> deliberately left out currency, tracking and updating exchange rates
> will be overkill for most of the django sites.
We have to be flexible, and not thinking that a user have to have
selected those variables, or that if is using a system different from
*Unix we could have to copy those *LC files. As I said this is very
dirty, and in addition those *LC files have been created thinking into
a operating system and not into a framework that is USING A DATA BASE,
so why don't use that data base for create a full localization with all
advantages of database?
> This proposal is for *core* L10N functionality (just like I18N) - so
> let's add number and date/time formatting for start (any other
> ideas?) and build other I18N and L10N functionality on top of that.
>
> Also, for date/time django already does a great job. I only wish to
> move DATE_FORMAT and similar from django.po into LC_* files.
>
> I will soon start to work on something like I described for my
> current project (and yes, I will also need the currency support). I
> don't want to patch django core if I don't have to so I'll try to
> make this a nice reusable application if it's possible.
>
I already have created tables about countries, languages, countries &
languages, subdivisions, and time zones. With all those data there are
many possibilities in order to make the easiest life to the
administrators and the own users.
You can load those those data of easily way using:
http://satchmo.python-hosting.com/file/trunk/satchmo/load_localization.py
but you need these directories whith its files:
http://satchmo.python-hosting.com/browser/trunk/satchmo/localization/
http://satchmo.python-hosting.com/browser/trunk/satchmo/data/
I think that in 2 weeks more I'll have almost finished the full
localization with phones, currencies, date-time formats, postal codes,
and address format
I'll speak with some of main Django developers when is finished the
next table that is phone.
Nebojša Đorđević
> I'm going to repeating some things:
>
> We have to be flexible, and not thinking that a user have to have
> selected those variables, or that if is using a system different from
> *Unix we could have to copy those *LC files. As I said this is very
> dirty, and in addition those *LC files have been created thinking into
> a operating system and not into a framework that is USING A DATA BASE,
> so why don't use that data base for create a full localization with
> all
> advantages of database?
Correction: LC_* are *directories* not files - just like LC_MESSAGES
directory.
With LC_* (or maybe some of L10N XML based formats <ducks>) approach
*all* of the django projects will be using the *same* files - no need
to update all of the django projects when new locale is added. And
LC_* approach *is* standard (at least on POSIX systems) for storing
this type of information (number, currency, date and time), see
http://www.opengroup.org/onlinepubs/007908799/xbd/locale.html.
For the same reason django uses .po files instead database based
translations.
> I already have created tables about countries, languages, countries &
> languages, subdivisions, and time zones. With all those data there are
> many possibilities in order to make the easiest life to the
> administrators and the own users.
And I don't need this. I just need the following functionality (IMHO
bare-bones L10N):
1. |number filter which will format number according to the current
locale
- something like |floatformat but locale aware.
2. |money filter or maybe a tag if exchange rate is to be supported.
3. {% now %} tag (for now we can use something like {% now _
('DATETIME_FORMAT') %}).
4. |date filter (for now we can use something like |date:_
('DATE_FORMAT')).
5. |time filter (for now we can use something like |time:_
('TIME_FORMAT')).
6. Collation order - in django this responsibility is delegated to
database layer.
And for above, what exactly are advantages for database based
approach (except for caching exchange rates for the money filter)? As
I already said, I'm talking about a *core* L10N functionality.
IMHO we have two options to accomplish that:
1. Use technical message IDs in django(js).po which is the current
solution (and only change now, date and time to use appropriate
technical ID as default) or,
2. Move all relevant L10N data inside LC_* hierarchy where (by POSIX
standards) they belong and change above filters and tags to use
current locale formats as the default (and keep backwards
compatibility).
GinTon
> On 28 Jul 2006, at 13:48, GinTon wrote:
>
> With LC_* (or maybe some of L10N XML based formats <ducks>) approach
> *all* of the django projects will be using the *same* files - no need
> to update all of the django projects when new locale is added. And
not necessary update the Django projects. Anyway this is not any
problem because until version 1.0 could suffer changes significatives.
> LC_* approach *is* standard (at least on POSIX systems) for storing
> this type of information (number, currency, date and time), see
> http://www.opengroup.org/onlinepubs/007908799/xbd/locale.html.
> For the same reason django uses .po files instead database based
> translations.
LC_* approach is a standard *for operating systems based on Unix*.
It is not a stardard for a framework based on a data base.
> > I already have created tables about countries, languages, countries &
> > languages, subdivisions, and time zones. With all those data there are
> > many possibilities in order to make the easiest life to the
> > administrators and the own users.
>
> And I don't need this.
I say you what I already made it until now.
> And for above, what exactly are advantages for database based
> approach (except for caching exchange rates for the money filter)? As
> I already said, I'm talking about a *core* L10N functionality.
I say you the disadvantages about use LC*:
1) You need all locales installed in your system to let the correct
localization to users of whatever country.
2) This system is more slow that use directly the data base:
You have to use a tool to get data from a directories and files, and it
will be more slow when you have s system with a lot of users with
different locales.
If you think that using directories and files is more fast that using a
data base then I don't know why you are using a framework based on a
data base.
Steven Armstrong
<snip>
> I say you the disadvantages about use LC*:
>
> 1) You need all locales installed in your system to let the correct
> localization to users of whatever country.
>
I would actually prefer the LC_* approach, but GinTon has a point with this.
It's bitten me before in a PHP project.
I'm from Switzerland, where we have 4 languages/locales, but the site
was hosted on a server in the US. The sysadmin refused to install the
missing locales so the only solution was a ugly "emulate missing system
locales" hack.
Nebojša Đorđević
>> With LC_* (or maybe some of L10N XML based formats <ducks>) approach
>> *all* of the django projects will be using the *same* files - no need
>> to update all of the django projects when new locale is added. And
> When the tables are finished it is not necessary add any more so it is
> not necessary update the Django projects. Anyway this is not any
> problem because until version 1.0 could suffer changes significatives.
And you'll still end up with x tables * django projects with
*identical* data.
> I say you what I already made it until now.
>
>
>> And for above, what exactly are advantages for database based
>> approach (except for caching exchange rates for the money filter)? As
>> I already said, I'm talking about a *core* L10N functionality.
>>
> I say you the disadvantages about use LC*:
>
> 1) You need all locales installed in your system to let the correct
> localization to users of whatever country.
>
Correction, *not on system* only in django.
Country to locale is not 1-1 mapping lot's of countries uses more
than one locale and also some locales are shared between more than
one country.
Note that we here tying i18n with l10n, so when user selects
*existing* locale it will see translated texts with localized
numbers, dates,... It's no use (and also confusing) to show localized
numbers when rest of interface is in i.e. english.
> 2) This system is more slow that use directly the data base:
> You have to use a tool to get data from a directories and files,
> and it
> will be more slow when you have s system with a lot of users with
> different locales.
Hmmmmmm, so you saying that current i18n approach is slower than
database based? Let's see:
FILE BASED:
When server thread/process is started all locale files (for the
default locale) are loaded and parsed and stored in the memory. When
a new locale is accessed it is loaded and parsed in the same way.
So we have 1 default locale load/parsing + (number of requested
locales * load/parsing) *per process loading*.
No duplicated data.
DATABASE BASED:
You can emulate above with database result caching but you'll be
always use more resources or you can load data as requested so you'll
end up with database access *per page view* at least.
All of django applications will have duplicated localization tables
with the same data.
> If you think that using directories and files is more fast that
> using a
> data base then I don't know why you are using a framework based on a
> data base.
Database is meant to store data which will be searched, CRUDed ....
L10N (and I18N) data is load *only once* per process/thread so with
avoiding database overhead we come with faster solution. You can
newer beat filesystem in any *read-only* operations.
Huh, this conversation getting longer and longer :). I'll will soon
have my solution ready for publishing an then we can compare results.
Nebojša Đorđević
Because that I propose to copy all of the needed LC_* data into
django locale directory so it will be system independent.
If you need another locale just add it yourself - no need to sysadmin
intervention.
It's the same system that we using now for translations.
So let me emphasize that again:
Yes, we will be using LC_* based l10n data but all of the required
files will be *inside* django distribution like LC_MESSAGES we are
using now. It will work even on windows which don't have any ability
to use this type of data and all of the locale settings and data will
be *local* to the django installation and independent of current
server setup.
GinTon
Nebojša Ðordevic wrote:
> On 28 Jul 2006, at 18:09, GinTon wrote:
> Country to locale is not 1-1 mapping lot's of countries uses more
> than one locale and also some locales are shared between more than
> one country.
GinTon
Nebojša Ðordevic wrote:
> On 28 Jul 2006, at 18:35, Steven Armstrong wrote:
> Because that I propose to copy all of the needed LC_* data into
> django locale directory so it will be system independent.
> If you need another locale just add it yourself - no need to sysadmin
> intervention.
>
> It's the same system that we using now for translations.
>
> So let me emphasize that again:
> Yes, we will be using LC_* based l10n data but all of the required
> files will be *inside* django distribution like LC_MESSAGES we are
> using now. It will work even on windows which don't have any ability
> to use this type of data and all of the locale settings and data will
> be *local* to the django installation and independent of current
> server setup.
But I have a dude. If an administrator install all locales in the
system, then does each Django project can use directly those locales?
GinTon
GinTon wrote:
> But I have a dude.
GinTon
to something as country or countries.
Nebojša Đorđević
> But I have a dude. If an administrator install all locales in the
> system, then does each Django project can use directly those locales?
I'm been thinking along this line:
First search for l10n files inside current django i18n path
(see http://www.djangoproject.com/documentation/i18n/#using-
translations-in-your-own-projects)
If all this fails try to use system locale path (i.e. /usr/share/
locale) -> no windows support - is there any way to get locale data
from windows?
So, answer is yes, it can use system wide locales (if django can't
find local ones).
Nebojša Đorđević
>
> And I'll have to change the name of 'localization' used in my project
> to something as country or countries.
Well if I can make this worth of inclusion in django core naming will
not be a problem because no new namespace will be introduced, just
django.utils.localization along with django.utils.translation.
Until then it will probably be under nesh.* namespace like other
stuff I published.
I'm also thinking to separate currency handling into another app
because it will need some sort of exchange rate database and currency
conversion handling functionality which will not be needed for most
of the projects.
Hmmm, we definitely need some sort of naming guidelines for django
add-on applications.
I hope that some of the core developers (and Hugo) are watching this
thread. I'll definitely like to hear their opinion on this.
GinTon
Nebojša Ðordevic wrote:
> On 29 Jul 2006, at 13:07, GinTon wrote:
>
> > But I have a dude. If an administrator install all locales in the
> > system, then does each Django project can use directly those locales?
>
> I'm been thinking along this line:
>
> First search for l10n files inside current django i18n path
> (see http://www.djangoproject.com/documentation/i18n/#using-
> translations-in-your-own-projects)
>
> If all this fails try to use system locale path (i.e. /usr/share/
> locale) -> no windows support - is there any way to get locale data
> from windows?
>
that the administrator should install all locales in order to
facilitate the full location to all the projects, we could to check at
install or run Django that the system has all locales installed, and
indicating it in the Django config file:
IF System == Unix:
check_all_locales
If has_all_locales:
write_config_file
So when Django has to use a locale, it knows in the first where search
it
Nebojša Đorđević
> Until then it will probably be under nesh.* namespace like other
> stuff I published.
I started a wiki at http://trac.studioquattro.biz/djangoutils/wiki/
Localization, feel free to comment.