                                                                                                                                                                                                                                                                
Received-SPF: pass (google.com: domain of erlang-questions-boun...@erlang.org designates 192.121.151.104 as permitted sender) client-ip=192.121.151.104;
Message-ID: <50179132.8070...@gmail.com>
Date: Tue, 31 Jul 2012 10:02:58 +0200
From: Richard Carlsson <carlsson.rich...@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:14.0) Gecko/20120714 Thunderbird/14.0
MIME-Version: 1.0
To: erlang-questi...@erlang.org
References: <CAANBt-pPC1Tscvekyf++-u7r43Nub7skO1C_N=+3Dn03wwT...@mail.gmail.com>
 <CAKrexV+D8WdibhyEMdzxXjXpois816-S2n9_eo4YA5LTu8m...@mail.gmail.com>
 <50168ABD.50...@gmail.com>
 <C48D77F1-4A12-4903-A90A-465B48CB4...@cs.otago.ac.nz>
 <50171BA4.3000...@gmail.com>
 <CAGPwZJ0=rNwN+=TydARW4c275P6_8oQdkzNmRGEJ6Udc9KR...@mail.gmail.com>
 <501784BA.6080...@gmail.com>
 <CAGPwZJ0B-f4m3912CeE5qgswJrm1iYa1w8mSL_0g2Zg7ozo...@mail.gmail.com>
In-Reply-To: <CAGPwZJ0B-f4m3912CeE5qgswJrm1iYa1w8mSL_0g2Zg7ozo...@mail.gmail.com>
Subject: Re: [erlang-questions] unicode in string literals
Precedence: list
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="windows-1252"; Format="flowed"
Errors-To: erlang-questions-boun...@erlang.org
Sender: erlang-questions-boun...@erlang.org

On 07/31/2012 09:32 AM, Michel Rijnders wrote:
> IMO this doesn't solve the problem, and only confuses the issue;
> consider the following:
>
> test() ->
>      io:format("~w~n", ["Just my =800.02"]),
>      io:format("~w~n", [lists:reverse("Just my =800.02")]).
>
>> test().
> [74,117,115,116,32,109,121,32,226,130,172,48,46,48,50]
> [50,48,46,48,172,130,226,32,121,109,32,116,115,117,74]

Yes, this is what happens today, because all involved parts (including =

the call to io:format with ~w) assumes Latin-1 and just passes all the =

bytes straight through. Basically, it's your editor and terminal that =

are lying by displaying a particular sequence of 3 bytes as =80 although =

the program is really using Latin-1. They conspire against you to make =

you think that things are working correctly.

> If the list data was kept as UTF-8 then the output of the second
> statement should be:
> [50,48,46,48,226,130,172,32,121,109,32,116,115,117,74]

That would only be the result if you used a single code point =

representation for the input to reverse, and then converted the result =

back to a byte encoding (e.g. by printing with ~ts).

> The above of course depends on whether you view strings as lists of
> bytes vs lists of characters.

Strings are lists of characters (code points), so when your example gets =

through tokenization, the encoding from the file would already be =

forgotten, and you'd have a single integer for the =80. (The same goes for =

atoms and variable names, by the way, the answer to so Vlad's question =

is that these will also get a greater range of available characters.) =

String manipulation functions should assume they are working on single =

code points, not on a byte encoding.

     /Richard

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions