Datefield i18n sorrow

296 views
Skip to first unread message

Martimiz

unread,
Sep 15, 2012, 12:10:56 PM9/15/12
to silverst...@googlegroups.com
Hi guys

Lately I've been wrestling with the DateField and the jQuery-UI DatePicker in an attempt tu use month shortnames in a different locale (dutch in my case). Numeric dates are not really a problem as long as we stick to the formats in Zend/Date.php (although I haven't tried them all), but character type dates like month shortnames do pose a problem:

First there are 3 separate units that have to work together:
- SilverStripe DateField
- Zend Date (and Locale)
- jQuery UI

Issue 1:
The jQuery-UI datepicker doesn't translate out of the box: DateField wants there to be a framework/thirdparty/jquery-UI/minified/i18n/jquery.ui.datepicker-nl.min.js file (path and filename hardcoded). At the moment they do not exist in SilverStripe but are easily downloaded here: http://jquery-ui.googlecode.com/svn/trunk/ui/i18n/ (or whereelse??).

Issue 2:
The Zend localization files can be found in thirdparty/Zend/Locale/Data/nl.XML Unfortunately, for Dutch at least, the shortnames are different from the ones in jquire-ui, so they'll never validate

Issue 3:
Zend Format uses iconv_strpos($number, $name, 0, 'UTF-8')) !== false) to check if the monthname is part of the Zend list of valid monthnames. Unfortunately this comparison seems to be case sensitive. And unfortunately again, DateField.php wants the first char of every word to be always uppercase (#203):  if(is_string($val)) $val = ucwords(strtolower($val)); probably to make sure en_US dates always validate, but this again breaks Dutch validation

Last but not least: if you set the admin profile to dutch, Datefield will send the Dutch locale to Zend. Bottom line: now you cannot use shortdates or longdates, unless you make sure there is a jquery localization file, it is equal to the Zend version, and you disable the DatField ucword setting.

Maybe a small issue for most en_US people, still it took me quite a couple of hours, and I'd like to streamline this for SilverStripe. So how can we best do this, any advice?

Martine

Ingo Schommer

unread,
Sep 17, 2012, 8:10:48 PM9/17/12
to silverst...@googlegroups.com
Hello there,

On 15/09/2012, at 6:10 PM, Martimiz <mart...@gmail.com> wrote:

Hi guys

Lately I've been wrestling with the DateField and the jQuery-UI DatePicker in an attempt tu use month shortnames in a different locale (dutch in my case). Numeric dates are not really a problem as long as we stick to the formats in Zend/Date.php (although I haven't tried them all), but character type dates like month shortnames do pose a problem:

First there are 3 separate units that have to work together:
- SilverStripe DateField
- Zend Date (and Locale)
- jQuery UI
Yes, localization is a pain :) To add another issue in the mix:
Zend_Date deals with ISO date formats, but the underlying
data is in CLDR date format. They're 99% the same, but not fully. Yay.

Issue 1:
The jQuery-UI datepicker doesn't translate out of the box: DateField wants there to be a framework/thirdparty/jquery-UI/minified/i18n/jquery.ui.datepicker-nl.min.js file (path and filename hardcoded). At the moment they do not exist in SilverStripe but are easily downloaded here: http://jquery-ui.googlecode.com/svn/trunk/ui/i18n/ (or whereelse??).
Not all available languages for e.g. the CMS UI will be localized for jQuery UI.
In this case, I opted to silently fail if the file is not available, and fall back to english.
Might just be a documentation issue on DateField.php?


Issue 2:
The Zend localization files can be found in thirdparty/Zend/Locale/Data/nl.XML Unfortunately, for Dutch at least, the shortnames are different from the ones in jquire-ui, so they'll never validate
Argh! Its probably easiest to change their configuration in the jQuery UI datepicker() call, 
since we don't want to start mucking with the XML CLDR files.
I don't think we can fix all possible format variations across our 50+ supported languages,
but we still want the CMS to be useable in those cases (= not fail validation).
The easiest way I see to achieve this is to globally use Zend's "short" format,
which should be all numeric in most cases. We can then add a tutorial
describing how to correctly configure longer date formats in the CMS.
Users can set their own date formats (incl. month name abbreviations)
in their preferences - in the light of those changes, we might need
to put a warning label on the field (or remove it altogether).

Issue 3:
Zend Format uses iconv_strpos($number, $name, 0, 'UTF-8')) !== false) to check if the monthname is part of the Zend list of valid monthnames. Unfortunately this comparison seems to be case sensitive. And unfortunately again, DateField.php wants the first char of every word to be always uppercase (#203):  if(is_string($val)) $val = ucwords(strtolower($val)); probably to make sure en_US dates always validate, but this again breaks Dutch validation
Yes, this is highly annoying of Zend_Date, and we needed the fix the other way around (for en_US).
Maybe we just need to add our own $.datepicker.regional overrides in DateField.js,
and make sure they match the Zend ones exactly? (and ensure they're loaded after the jQUI defaults).
That still leaves manual entry susceptible to case-sensitive validation, but its a start.

Last but not least: if you set the admin profile to dutch, Datefield will send the Dutch locale to Zend. Bottom line: now you cannot use shortdates or longdates, unless you make sure there is a jquery localization file, it is equal to the Zend version, and you disable the DatField ucword setting.

Maybe a small issue for most en_US people, still it took me quite a couple of hours, and I'd like to streamline this for SilverStripe. So how can we best do this, any advice?
Martine, I'm unlikely to have much time for this in the next weeks.
Given that you're already so deep in the matter,
do you want to make a start on that tutorial (or improving the existing i18n.md docs),
and/or the datepicker overrides?

Thanks
Ingo

Martimiz

unread,
Sep 18, 2012, 5:11:10 AM9/18/12
to silverst...@googlegroups.com
Hi Ingo,
 
Zend_Date deals with ISO date formats, but the underlying
data is in CLDR date format. They're 99% the same, but not fully. Yay.
 
Yes, I saw that one. In fact there seems to be only a limited range of Date constants that work with Zend and jQuery and can be persuaded to validate.

In this case, I opted to silently fail if the file is not available, and fall back to english.
Might just be a documentation issue on DateField.php?

I don't think so: Zends' isDate() function will then check the jQuery en version against its own current locale locale values, so only numeric dates will ever validate. 

The easiest way I see to achieve this is to globally use Zend's "short" format,
which should be all numeric in most cases. We can then add a tutorial
describing how to correctly configure longer date formats in the CMS.
Users can set their own date formats (incl. month name abbreviations)
in their preferences - in the light of those changes, we might need
to put a warning label on the field (or remove it altogether).

Wow! Don't remove it - I still want it, and seeing the amount of questions on the forums, others do too :)  
By Zend shortdate, do you mean DATE_SHORT  = 'FF'? Because that format isn't recognised by the DatePicker, nor are (most of?) the other full or ISO-xxx dates. Using a format like 'MM dd yyyy' works fine and people can change it at will. 

Maybe we just need to add our own $.datepicker.regional overrides in DateField.js,
and make sure they match the Zend ones exactly? (and ensure they're loaded after the jQUI defaults).
That still leaves manual entry susceptible to case-sensitive validation, but its a start.

That's what I was thinking of. It isn't very hard to set up some localization files, at least for some of the most common languages for the DatePicker and have them match Zend. Then we could add some locale dependent settings to tackle the ucword issue and maybe set the locale to en_US if a localization file cannot be found.  Next a small tutorial could explain how people can create/contribute their own?

Martine, I'm unlikely to have much time for this in the next weeks.
Given that you're already so deep in the matter,
do you want to make a start on that tutorial (or improving the existing i18n.md docs),
and/or the datepicker overrides?

I've a couple of hours available and I'd like to try and come up with a working solution like the one above first. For the time being I've written up an overview on my blog (http://www.balbuss.com/jquery-ui-datepicker-in-silverstripe-3/). If I get this working, I'll do the tutorial afterwards. 

Martine

Ingo Schommer

unread,
Sep 18, 2012, 5:47:17 AM9/18/12
to silverst...@googlegroups.com
On 18/09/2012, at 11:11 AM, Martimiz <mart...@gmail.com> wrote:

Hi Ingo,
 
Zend_Date deals with ISO date formats, but the underlying
data is in CLDR date format. They're 99% the same, but not fully. Yay.
 
Yes, I saw that one. In fact there seems to be only a limited range of Date constants that work with Zend and jQuery and can be persuaded to validate.

In this case, I opted to silently fail if the file is not available, and fall back to english.
Might just be a documentation issue on DateField.php?

I don't think so: Zends' isDate() function will then check the jQuery en version against its own current locale locale values, so only numeric dates will ever validate. 
That's true for stuff like month names, which I've suggested a solution for below.
For UI labels, jQuery UI will fall back to english. And for date format, we're
passing our own one into the jQuery UI config anyway, regardless of
whether the <lang>.js file exists.


The easiest way I see to achieve this is to globally use Zend's "short" format,
which should be all numeric in most cases. We can then add a tutorial
describing how to correctly configure longer date formats in the CMS.
Users can set their own date formats (incl. month name abbreviations)
in their preferences - in the light of those changes, we might need
to put a warning label on the field (or remove it altogether).

Wow! Don't remove it - I still want it, and seeing the amount of questions on the forums, others do too :)  
By Zend shortdate, do you mean DATE_SHORT  = 'FF'? Because that format isn't recognised by the DatePicker, nor are (most of?) the other full or ISO-xxx dates. Using a format like 'MM dd yyyy' works fine and people can change it at will. 
In nl.xml the short date format is defined as:
<dateFormatLength type="short">
<dateFormat>
<pattern>dd-MM-yy</pattern>
</dateFormat>
</dateFormatLength>
I don't think there's a unified definition of what a "short date" is across locales,
but from my western perspective I'd assume that it doesn't involve spelled out month names?
Either way, its not a great solution as it shortens the year to two digits, which is confusing.

Maybe we just need to add our own $.datepicker.regional overrides in DateField.js,
and make sure they match the Zend ones exactly? (and ensure they're loaded after the jQUI defaults).
That still leaves manual entry susceptible to case-sensitive validation, but its a start.

That's what I was thinking of. It isn't very hard to set up some localization files, at least for some of the most common languages for the DatePicker and have them match Zend. Then we could add some locale dependent settings to tackle the ucword issue and maybe set the locale to en_US if a localization file cannot be found.  Next a small tutorial could explain how people can create/contribute their own?
I'd be in favour of removing the ucwords() invocation once we have the localization overrides in place,
its just too much complexity. In short, if you're expecting dates to be typed in manually
rather than selected from a calendar, don't use month abbreviations.
Good idea about tutorial to contrib your own localization overrides,
should just be a section in the DateField docs, linked from the i18n docs.

Martine, I'm unlikely to have much time for this in the next weeks.
Given that you're already so deep in the matter,
do you want to make a start on that tutorial (or improving the existing i18n.md docs),
and/or the datepicker overrides?

I've a couple of hours available and I'd like to try and come up with a working solution like the one above first. For the time being I've written up an overview on my blog (http://www.balbuss.com/jquery-ui-datepicker-in-silverstripe-3/). If I get this working, I'll do the tutorial afterwards. 
Great, thank you!

Ingo

Martimiz

unread,
Sep 18, 2012, 9:10:44 AM9/18/12
to silverst...@googlegroups.com

What I have so far is this:

1 - I've adapted a couple of datepicker language files (en, nl, de, fr, es, nb, da and sw) so that they now match with Zend.
2 - I've disabled the ucwords statement
3 - Ive added a method HasLocaleFile() to DateField_View_JQuery, that checks if the corresponding file exists.
4 - If no such file exists, and the dateformat has long/short monthnames, I replace the dateformat with 'yyyy-MM-dd': this makes datepicker add the date in numeric format, but after saving the set format using Zend is displayed.
5. I think I might add a 'preferred numeric fallback' to config that you can but need not set...

We then can at least safely use the following constants (and all kinds of litterals like  -  _  ;  : , . and more ):

M, MM,
d, dd
y, yy, yyyy
MMM and MMMM  (monthnames)
EEE, EEEE  (daynames, as they need no validation it doesn't matter that the datepicker doesn't use them...)
SS (seemingly untranslatable day prefix, as EEE, EEEE)

For me, that's what I initially wanted. Maybe there's more to be gained by tweaking convert_iso_to_jquery_format(), like linking other jQuery formats, but that will have to be tested.
I don't think so: Zends' isDate() function will then check the jQuery en version against its own current locale locale values, so only numeric dates will ever validate. 
That's true for stuff like month names, which I've suggested a solution for below.

Yes - and for shortdates, I did test... :) They are different for different locales, so jQuery should know that, and it wont because it has no locale file, there goes the default (snake biting)

For UI labels, jQuery UI will fall back to english. And for date format, we're
passing our own one into the jQuery UI config anyway, regardless of
whether the <lang>.js file exists.

True, labels don't matter.

The easiest way I see to achieve this is to globally use Zend's "short" format,
which should be all numeric in most cases. We can then add a tutorial
describing how to correctly configure longer date formats in the CMS.
Users can set their own date formats (incl. month name abbreviations)
in their preferences - in the light of those changes, we might need
to put a warning label on the field (or remove it altogether).

Wow! Don't remove it - I still want it, and seeing the amount of questions on the forums, others do too :)  
By Zend shortdate, do you mean DATE_SHORT  = 'FF'? Because that format isn't recognised by the DatePicker, nor are (most of?) the other full or ISO-xxx dates. Using a format like 'MM dd yyyy' works fine and people can change it at will. 
 
In nl.xml the short date format is defined as:
<dateFormatLength type="short">
<dateFormat>
<pattern>dd-MM-yy</pattern>
</dateFormat>
</dateFormatLength>
I don't think there's a unified definition of what a "short date" is across locales,
but from my western perspective I'd assume that it doesn't involve spelled out month names?
Either way, its not a great solution as it shortens the year to two digits, which is confusing.

Yes, and jQuery doesn't understand FF or FFF or FFFF so it treats it as a litteral. Another job for convert_iso_to_jquery_format() ???
 
I'd be in favour of removing the ucwords() invocation once we have the localization overrides in place,
its just too much complexity. In short, if you're expecting dates to be typed in manually
rather than selected from a calendar, don't use month abbreviations.

That's a good tip for the docs, Ill keep that in mind. Still people are allowed to type in the Datefield, even if the calendar is enabled...
 
Good idea about tutorial to contrib your own localization overrides,
should just be a section in the DateField docs, linked from the i18n docs.

Where in the docs should the datefield live? in Reference I suppose?

Martine



Uncle Cheese

unread,
Sep 18, 2012, 11:45:25 AM9/18/12
to silverst...@googlegroups.com
Awesome blog, Martine. Thanks for that.

For future reference, this thread answers a lot of questions in my previous post: https://groups.google.com/forum/?fromgroups=#!searchin/silverstripe-dev/woes/silverstripe-dev/YQr73weatf4/j8872IZGmisJ

Martimiz

unread,
Sep 18, 2012, 2:22:30 PM9/18/12
to silverst...@googlegroups.com

@Uncle Cheese: you're very welcome, thanks :-)

I found one more player on the Date court: the native PHP date() function:

What more or less happens is that, when the DateField retrieves the ISO Date from the database for display (setValue), it creates a Zend_Date object that turns the retreived date into a UNIX timestamp. Next the DateFields requests the formatted date back for the  date format we selected. Zend will then feed the timestamp + format to, yes, the php date function.

@ingo: you said it: Zend is sort of ISO - obviously that doesn't always go for the locale files :( In nl.xml, for instance, the abbreviated name for friday in the Zend nl.xml language file is = 'vr', so I copied this to the  jQuery file as wel. But php date() will return 'vri' (which I don't like, but hey) and that will be displayed. AAAArgh! But both vr and vri validate, so what the heck.

At least I got the DatePicjker to recognize short and long days as well :-)

Martine


Martimiz

unread,
Sep 25, 2012, 12:23:32 PM9/25/12
to silverst...@googlegroups.com
Hi all,

@Ingo
Please take a look at the DateField fix, before I send in a pull request
https://github.com/Martimiz/sapphire/commit/51d87b9b301bd1b495d027a98d89181708c4f390

This includes:
- updated DateField.php
- some adapted jQuery DatePicker locale files + README
- somewhat extensive documentation for the DateField (including howto contribute locale files)

Basically now month and daynames validate for locales that have a valid locale file. If no such file exists, the datepicker will fallback to datavalueformat (yyyy-MM-dd). I've removed the forced ucwords, so locales that have lowercase values may validate. Solved an issue where EEE(E) formats were overwritten before being served to the datepicker.

I hope this is more or less what you were looking for, awaiting your comments for now...

Martine

Ingo Schommer

unread,
Oct 2, 2012, 8:50:49 AM10/2/12
to silverst...@googlegroups.com
Hello Martine,

Sorry, your email seemed to be pushed down the stack in my inbox :/
Commented inline on the commit, but mostly trivial stuff.
Thanks for documenting this so thoroughly!
Functionally, the fix makes sense. Tested a bit locally,
on de_DE (German), which has locale overrides, 
and mi_NZ (Te Reo Maori), which doesn't.
Both work fine for me. Usually I'd say throw an exception
if the 'showcalendar' setting is incompatible with the 'dateformat',
but given that users can set their own format, its better to silently
fall back to the standard US date format - it should still be intelligible 
enough for different cultures (less day/month confusion than m/d/y).

Happy to look over a pull request once you've done those smaller fixes.

Thanks
Ingo

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/silverstripe-dev/-/-25vcnEFv5MJ.
To post to this group, send email to silverst...@googlegroups.com.
To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/silverstripe-dev?hl=en.

Martimiz

unread,
Oct 3, 2012, 5:12:59 AM10/3/12
to silverst...@googlegroups.com
Hi Ingo,

No problem, I realise you must be very busy!! :-)

I'm happy it's only a few trival matters. As as things are getting a bit busy for me at the moment, I hope to complete this later this week and answer  your questions as well.

Thanks, Martine



Op dinsdag 2 oktober 2012 14:50:56 UTC+2 schreef Ingo Schommer het volgende:
Reply all
Reply to author
Forward
0 new messages