Encoding Problems

[spectr]

I am running into encoding problems:

db = SQLite3::Database.new "encoding.db"
=> #<SQLite3::Database:0x9b69cbc>

db.encoding
=> #<Encoding:UTF-16LE>

r1 = db.execute("select * from db_log_sink limit 2").first.last
=> "\u7953\u7473\u6D65\u5420\u6D69\u7A65\u6E6F
\u3A65\u4520\u5453\u2F20\u4520\u5444\x0A"

r2 = db.execute("select * from db_log_sink limit 2").last.last
=> "????????\u0A3A"

r1.encoding
=> #<Encoding:UTF-16LE>

r2.encoding
=> #<Encoding:UTF-8>

Working at the linux command line with the same file I get:
select * from db_log_sink limit 2;
1|2011-11-16T12:02:15|0|System Timezone: EST / EDT
2|2011-11-16T12:02:15|0|Server Hostnames:

In the browser r2 comes out as Chinese traditional han. r1 comes out
as normally formatted text.

About half the records in text columns come out garbled when using the
gem/ruby. Everything looks normal at the Linux command line and using
SQLiteSpy under Windows.

The database can be downloaded from here:
https://s3.amazonaws.com/ilf/encoding.db

Any help would be much appreciated.

[spectr]

Found workaround: r2.unpack('U*').pack('v*').force_encoding('utf-8')

[Luis Lavena]

On Wed, Nov 23, 2011 at 5:39 PM, spectr <tkan...@gmail.com> wrote:
> I am running into encoding problems:
>
> db = SQLite3::Database.new "encoding.db"
> => #<SQLite3::Database:0x9b69cbc>
>
> db.encoding
> => #<Encoding:UTF-16LE>
>
> r1 = db.execute("select * from db_log_sink limit 2").first.last
> => "\u7953\u7473\u6D65\u5420\u6D69\u7A65\u6E6F
> \u3A65\u4520\u5453\u2F20\u4520\u5444\x0A"
>
> r2 = db.execute("select * from db_log_sink limit 2").last.last
> => "????????\u0A3A"
>
> r1.encoding
> => #<Encoding:UTF-16LE>
>
> r2.encoding
> => #<Encoding:UTF-8>
>

How are you storing that data? can you show us the code you use to insert it?

Can you check the encoding when inserting?
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry