Decoding UTF16 sequences to ASCII string

[Tong Sun]

Hi, 

I think my test program, https://github.com/suntong/lang/blob/master/lang/Go/src/text/utf16B.go, went very well, as I see the output, which is enclosed at the bottom of the program, is exactly what I'm expecting, until I redirect the output to a file, and view it with my `less`, when I found out the output is still in utf16, not ASCII string. 

I've put it on golang play

http://play.golang.org/p/p-nV7fk33w

but please don't let its current output deceives you, 

Please help me make the output to be true ASCII string, as you can see, I was trying with `utf16.Decode` but that failed. 

Thanks

[Manlio Perillo]

Il giorno martedì 3 novembre 2015 19:01:45 UTC+1, Tong Sun ha scritto:

Hi, 

I think my test program, https://github.com/suntong/lang/blob/master/lang/Go/src/text/utf16B.go, went very well, as I see the output, which is enclosed at the bottom of the program, is exactly what I'm expecting, until I redirect the output to a file, and view it with my `less`, when I found out the output is still in utf16, not ASCII string. 

> [...]

Your program is not really correct.

The correct code is this:

http://play.golang.org/p/2aeqEJ_vWK

The reason the output looks correct on a terminal (or on a browser) is because the \x00 byte is not printed and has a zero width.

However look at:

http://play.golang.org/p/X53vOBLPD2

Note that Go string is just a sequence of bytes assumed to be UTF-8 encoded.

Regards  Manlio 

[Jakob Borg]

You need to give utf16.Decode a slice of uint16s, each representing one character (or code point or whatever):

http://play.golang.org/p/cCufRfEQso

//jb

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tong Sun]

Oh Thank you Jakob, and Manlio!

On Tue, Nov 3, 2015 at 1:46 PM, Jakob Borg wrote:
> You need to give utf16.Decode a slice of uint16s, each representing one
> character (or code point or whatever):
>
> http://play.golang.org/p/cCufRfEQso
>
> //jb
>

[Manlio Perillo]

Il giorno martedì 3 novembre 2015 19:52:46 UTC+1, Tong Sun ha scritto:

Oh Thank you Jakob, and Manlio!

Note that the correct solution is to use an UTF16 decoder:

http://play.golang.org/p/8fwFyBInpW

The decoding is a bit more complex, in general, due to surrogate pairs:

https://github.com/golang/text/blob/master/encoding/unicode/unicode.go#L194

> [...]

Regards  Manlio