Ruby/Unicode library - ruby-talk-google

Message from discussion Ruby/Unicode library
Received-SPF: pass (googlegroups.com: best guess record for domain of ruby-talk-ad...@ruby-lang.org designates 210.163.138.100 as permitted sender)
Date: Mon, 19 Jun 2006 03:11:32 +0900
Posted: Sun, 18 Jun 2006 11:09:36 -0700
From: Rob Leslie <r...@mars.org>
Reply-To: ruby-t...@ruby-lang.org
Subject: Ruby/Unicode library
To: ruby-t...@ruby-lang.org (ruby-talk ML)
Message-Id: <3D218B7A-07C1-4A6C-AE72-747F1460F0BF@mars.org>
Mime-Version: 1.0 (Apple Message framework v750)
Content-Type: text/plain; charset=UTF-8; delsp=yes; format=flowed
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
Lines: 72

Since there's been a lot of talk about Unicode lately, I thought I'd =20
throw out a Ruby library I've been working on to support Unicode =20
characters and strings based on the 4.1.0 standard and key =20
specifications from the Unicode Consortium.

   ftp://ftp.mars.org/pub/ruby/Unicode.tar.bz2

The library adds an encoding property to native String objects, and =20
allows conversion to and from Unicode::String and Unicode::Character. =20=

A default encoding is chosen based on $KCODE, or the default can be =20
set/changed explicitly via String.default_encoding.

Unicode strings can be obtained by applying the + unary operator to =20
native strings, e.g. +"Hello" (where the native string is encoded in =20
the default encoding).

   % irb -I. -runicode -Ku
   irb(main):001:0> ustr =3D +"=CF=80 is pi"
   =3D> +"=CF=80 is pi"

Native strings are obtained from Unicode strings by calling to_s, =20
which accepts an optional argument to indicate the desired encoding.

   irb(main):002:0> str =3D ustr.to_s
   =3D> "=CF=80 is pi"
   irb(main):003:0> str.encoding
   =3D> Unicode::Encoding::UTF8

Individual characters can be indexed from Unicode strings, returning =20
a Unicode::Character object.

   irb(main):004:0> ustr[0]
   =3D> U+03C0 GREEK SMALL LETTER PI

Case conversion is handled as with native strings.

   irb(main):005:0> ustr.upcase
   =3D> +"=CE=A0 IS PI"

Normalization is accomplished with the ~ unary operator.

   irb(main):006:0> ustr =3D +"m=C3=AD"
   =3D> +"m=C3=AD"
   irb(main):007:0> ustr.to_a
   =3D> [U+006D LATIN SMALL LETTER M, U+00ED LATIN SMALL LETTER I WITH =20=

ACUTE]
   irb(main):008:0> (~ustr).each_char { |ch| p ch }
   U+006D LATIN SMALL LETTER M
   U+0069 LATIN SMALL LETTER I
   U+0301 COMBINING ACUTE ACCENT
   =3D> +"m=C3=AD"

There is much more -- character properties, text boundaries (grapheme =20=

clusters and words), Hangul decompositions, modular encodings (ASCII, =20=

Latin1, EUC, SJIS, UTF32, UTF16, UTF8) -- yet the project is =20
unfinished. If anyone is interested in helping develop it further, =20
let me know.

The library incorporates the entire Unicode 4.1.0 Character Database =20
(demand-loaded!) which is why the archive is rather large.

Cheers,

--=20
Rob Leslie
r...@mars.org