Jira (PUP-1800) Ruby downcase does not handle all unicode letters

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1800 Ruby downcase does not handle all unicode letters Change By: Henrik Lindberg Scrum Team: Language Add Comment

This message was sent by Atlassian JIRA (v6.3.10#6340-sha1:7ea293a)

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-1800 Ruby downcase does not handle all unicode letters

Change By: Ethan Brown Labels: i18n utf-8

Add Comment

This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-1800

Re: Ruby downcase does not handle all unicode letters Note that: normalization is required which is performance demanding in Ruby user locale is needed as rules for up/down case depends on locale locale is also needed for correct comparisons/ordering of strings These together makes this a much thornier problem than just adding a library and comparing a different way.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1800

Ruby downcase does not handle all unicode letters

Change By: Henrik Lindberg Team: Puppet Developer Support

Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Ethan Brown (JIRA)]

Ethan Brown commented on PUP-1800

Re: Ruby downcase does not handle all unicode letters Henrik Lindberg I'm removing the PUP-1031 epic in an effort to close that off. This seems like a ticket that belongs in a separate language (or similar) related epic like the ones that we've created at PUP-6718 / PUP-6719 / PUP-6720. I'd suggest creating that new epic and having it block the LOC-11 epic.

Add Comment

[John Duarte (JIRA)]

John Duarte updated an issue

Puppet / PUP-1800

Ruby downcase does not handle all unicode letters

Change By: John Duarte Labels: i18n  triaged  utf-8

Add Comment

[Moses Mendoza (JIRA)]

Moses Mendoza updated an issue

Puppet / PUP-1800 Ruby downcase does not handle all unicode letters

Change By: Moses Mendoza Labels: i18n  triaged  utf-8

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown commented on PUP-1800

Re: Ruby downcase does not handle all unicode letters FYI - Ruby 2.4 (which will ship with Puppet 5) is supposed to have fixed this.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-1800

Re: Ruby downcase does not handle all unicode letters

Here is an article about the Unicode support in Ruby 2.4 http://www.sw.it.aoyama.ac.jp/2016/pub/IUC40-Ruby2.4/ I found that it follows Unicode 9.0.0 recommendations for handling of special characters in German, Turkish, etc. by making compromises. That is better than having to rely on Locale and having different behavior depending on it. This means that up/down-case operations work - but that the operation in reverse (upcase.downcase or downcase.upcase) is not guaranteed to produce the same result. This is good enough since Puppet is not a typesetting system I have yet to try out sorting and comparisons (<=>, <, >, etc) and operations like casecmp. (Article did not talk much about that)

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-1800

Re: Ruby downcase does not handle all unicode letters

Oh, the implementation is naturally filled with gotchas - see this article: http://blog.honeybadger.io/ruby-s-unicode-support/ Read through the list of tests in that post and cry... We may have to normalize all strings everywhere and that will have a huge performance impact. (Every JSON and YAML read for example). If we do nothing we get the new behavior and we will probably be seeing bug after bug in relation to use of unicode characters and non normalized strings.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-1800

Ruby downcase does not handle all unicode letters

Change By: Josh Cooper Team: Puppet Developer Experience Night's Watch

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-1800 Ruby downcase does not handle all unicode letters Change By: Josh Cooper

In the puppet language, {{" A" == "a"}} is true and {{" Ä" == "ä"}} should be true, but it is false . : {noformat} bx puppet apply -e 'notice("A" == "a") notice("Ä" == "ä")' Notice: Scope(Class[main]): true Notice: Scope(Class[main]): false {noformat} Ruby doesn't properly handle changing from upper to lower case letters for all unicode glyphs in the letters category. Other interesting cases to look at are things like "ß" which has no upper/lower-case distinction.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-1800

Re: Ruby downcase does not handle all unicode letters This would be good to resolve but we don't have any plans on fixing it anytime soon. Please reopen if this is needed.

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)