Sorting on the web -- UCA and CLDR for web?

[Tim Chien]

Hi all, quick question,

A mature platform accompany needs of it's global users. On the web, we
have no ICU [1] available in browser (or even as a Javascript shim),
which prevent web authors for achieving the following:

-- Making a Gaia Contacts app that would sort contacts by Chinese
pinyin, i.e. the sounding of the characters.
-- Implementing a search that could match "Céline Dion" against "Celine Dion"

As a result, every list we wrote on the client-side is always sorted
in Unicode codepoints, and special, but never complete filter must be
written every time in every application to match "é" against "e".

I wonder if there is already a discussion on Web APIs for providing
such native support to the web? Would it be possible to integrate that
into, say, Array.sort(), or IndexedDB range search?

Please provide some pointers if there is already a discussion. Thanks :)


[1] https://en.wikipedia.org/wiki/International_Components_for_Unicode
provides UCA and/or CLDR support in C/C++ & Java applications. UCA
stands for Unicode collation algorithm and CLDR stands for Common
Locale Data Repository.

--
Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
OS, Mozilla Corp. (Taiwan)

[Gervase Markham]

On 08/05/13 10:58, Tim Chien wrote:
> I wonder if there is already a discussion on Web APIs for providing
> such native support to the web? Would it be possible to integrate that
> into, say, Array.sort(), or IndexedDB range search?

http://norbertlindenberg.com/2012/12/ecmascript-internationalization-api/

Gerv

[Anne van Kesteren]

On Wed, May 8, 2013 at 2:58 AM, Tim Chien <timd...@mozilla.com> wrote:
> I wonder if there is already a discussion on Web APIs for providing
> such native support to the web? Would it be possible to integrate that
> into, say, Array.sort(), or IndexedDB range search?
>

> Please provide some pointers if there is already a discussion. Thanks :)

http://norbertlindenberg.com/2012/12/ecmascript-internationalization-api/index.html
is what you want I think.


--
http://annevankesteren.nl/

[Marcos Caceres]

On Wednesday, May 8, 2013 at 4:29 PM, Anne van Kesteren wrote:

> On Wed, May 8, 2013 at 2:58 AM, Tim Chien <timd...@mozilla.com (mailto:timd...@mozilla.com)> wrote:
> > I wonder if there is already a discussion on Web APIs for providing
> > such native support to the web? Would it be possible to integrate that
> > into, say, Array.sort(), or IndexedDB range search?
> >
> > Please provide some pointers if there is already a discussion. Thanks :)
>
> http://norbertlindenberg.com/2012/12/ecmascript-internationalization-api/index.html
> is what you want I think.

You might also enjoy:
http://marcoscaceres.github.io/jsi18n/

A more dev friendly introduction to the API… though I haven't updated it in a while.

[Andrew Sutherland]

On 05/08/2013 11:29 AM, Anne van Kesteren wrote:

> On Wed, May 8, 2013 at 2:58 AM, Tim Chien <timd...@mozilla.com> wrote:
>> I wonder if there is already a discussion on Web APIs for providing
>> such native support to the web? Would it be possible to integrate that
>> into, say, Array.sort(), or IndexedDB range search?
>>
>> Please provide some pointers if there is already a discussion. Thanks :)
> http://norbertlindenberg.com/2012/12/ecmascript-internationalization-api/index.html
> is what you want I think.

The proposal doesn't seem to explicitly address the IndexedDB aspect of
this. Specifically, in order to properly sort strings in IndexedDB,
either IndexedDB needs to support specifying collation rules or the JS
API needs to provide a way to perform case/accent/etc. folding so that a
normalized representation can be produced that will sort closer to the
collation rules.

For full-text search purposes, it would likely be desirable to have the
normalization/folding mechanism available for tokenization purposes.
Although maybe at that point if things are being done in client JS space
for experimentation, that code would just need a whole bunch of tables
itself. And after that's working okay standards could be proposed,
either including more JS API, IndexedDB, or other support...

Andrew

[Jonas Sicking]

On Wed, May 8, 2013 at 11:32 AM, Andrew Sutherland
<asuth...@asutherland.org> wrote:
> On 05/08/2013 11:29 AM, Anne van Kesteren wrote:
>>
>> On Wed, May 8, 2013 at 2:58 AM, Tim Chien <timd...@mozilla.com> wrote:
>>>
>>> I wonder if there is already a discussion on Web APIs for providing
>>> such native support to the web? Would it be possible to integrate that
>>> into, say, Array.sort(), or IndexedDB range search?
>>>
>>> Please provide some pointers if there is already a discussion. Thanks :)
>>
>>
>> http://norbertlindenberg.com/2012/12/ecmascript-internationalization-api/index.html
>> is what you want I think.
>
>
> The proposal doesn't seem to explicitly address the IndexedDB aspect of
> this. Specifically, in order to properly sort strings in IndexedDB, either
> IndexedDB needs to support specifying collation rules or the JS API needs to
> provide a way to perform case/accent/etc. folding so that a normalized
> representation can be produced that will sort closer to the collation rules.

When we developed IndexedDB initially we talked about this. But it was
punted until later versions because it's a quite complex problem space
and we wanted to get the basics right first.

This is actually the first time I've heard from developers actually
asking for it though, so this is great feedback.

What we should be able to do in a relatively short term is to add
support for using ArrayBuffers as keys. That way you can implement
sorting yourself in JS. Definitely quite painful though :(

> For full-text search purposes, it would likely be desirable to have the
> normalization/folding mechanism available for tokenization purposes.
> Although maybe at that point if things are being done in client JS space for
> experimentation, that code would just need a whole bunch of tables itself.
> And after that's working okay standards could be proposed, either including
> more JS API, IndexedDB, or other support...

Full-text search is a super complex area. We've recently talked about
adding a few low-level primitives to IndexedDB to allow you to
implement full-text search on top of it.

/ Jonas

[Tim Chien]

Thanks everyone. Now I know there is a shim to look into, and a Intl
API already being spec'd.
Is there already a bug filed on Gecko for implementation?

The IndexedDB part sounds ... unfortunate. Let's hope there is no FxOS
feature requests pop-up before we figure it out on that end.

> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia

[Axel Hecht]

https://bugzilla.mozilla.org/show_bug.cgi?id=866301 is the bug to enable
the i18n api on firefox os. fyi, desktop just bounced yesterday.

Axel