Unicode problem
Alan Siegrist
Hi All,
There was some discussion on the list a ways back on the problems of Unicode. I could not think of a specific example at the time, but I just ran across one, so let me show you a concrete example.
The character 奠 is used in both Japanese and Chinese, but its character form is significantly different in the fonts typically used for each language. In Chinese fonts, the top of the character looks like the character 酋, while in Japanese fonts, it looks like 八+西.
To illustrate these differences, here is an enlarged example in MS 明朝 (a typical Japanese font) and a Chinese font called SimSun (I hope these display correctly):
奠 (MS 明朝)
奠 (SimSun)
Despite the character forms being significantly different in modern Japanese and Chinese, the same Unicode code point is used for both. This is not usually a problem for monolingual readers, since the fonts for each language are simply designed to reflect the preferences and expectations of the expected readers.
It does become a problem when it is not possible to change the display font. For example, my system is set to use Japanese display fonts and I went to this web page:
http://www.hanzismatter.com/2009/11/csi-ny-it-happened-to-me.html
I was confused because the Japanese form 奠 appeared in the image caption text, while the Chinese form 奠 clearly appeared in the image, so I knew something was wrong, but I was not sure what.
It took a while to sort everything out and I finally realized the problem was due to Han Unification under Unicode. This is clearly illustrated on the Unicode page for this character:
http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=%E5%A5%A0&useutf8=true
However, if these sorts of problems are indeed widespread and do cause significant difficulties, they can be solved by simply adding extra Unicode code points to disambiguate the character forms. Unicode is extensible enough to permit this.
Regards,
Alan Siegrist
Carmel, CA, USA
Alan Siegrist
I wrote:
in Japanese fonts, it looks like 八+西.
Oops, it is more like 八+酉.
Adam Rice
> Despite the character forms being significantly different in modern
> Japanese and Chinese, the same Unicode code point is used for both.
> This is not usually a problem for monolingual readers, since the
> fonts for each language are simply designed to reflect the
> preferences and expectations of the expected readers.
>
> It does become a problem when it is not possible to change the
> display font. For example, my system is set to use Japanese display
> fonts and I went to this web page:
> http://www.hanzismatter.com/2009/11/csi-ny-it-happened-to-me.html
>
> I was confused because the Japanese form 奠 appeared in the
> image caption text, while the Chinese form 奠 clearly appeared
> in the image, so I knew something was wrong, but I was not sure what.
>
> It took a while to sort everything out and I finally realized the
> problem was due to Han Unification under Unicode. This is clearly
> illustrated on the Unicode page for this character:
> http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=%E5%A5%A0&useutf8=true
>
> However, if these sorts of problems are indeed widespread and do
> cause significant difficulties, they can be solved by simply adding
> extra Unicode code points to disambiguate the character forms.
> Unicode is extensible enough to permit this.
This is an interesting problem, and it is actually more complicated
than just code points. The browser you're using and the page's markup
will also affect which variant of the character is shown.
The two browsers I regularly use are Firefox and Safari. Firefox is
smarter about this kind of thing: if it detects that a page is written
in Chinese (based on the language declaration in the doctype element),
it will render using a Chinese typeface; Japanese for Japanese. The
hanzismatter page in question is declared as English, and Firefox
defaults to Japanese, which is higher up in my linguistic food-chain.
Safari is not so smart: because Japanese is higher up in my linguistic
food-chain, it renders Chinese characters in a Japanese font whenever
it can.
You can play around with this by downloading the hanzismatter page and
changing the language type. In fact, you can declare language
attributes on individual elements to see how that affects rendering
for identical text.
I've created a test page where you can see this effect in action:
http://8stars.org/misc/test_unihan.html
Adam Rice :: Austin TX USA :: adam...@8stars.org :: http://8stars.org
Alan Siegrist
> I've created a test page where you can see this effect in action:
> http://8stars.org/misc/test_unihan.html
Thanks, Adam. That illustrates the effect perfectly.
I think this is a perfect illustration of the importance of page authors
declaring the language of text. Not being very HTML-savvy, I did not realize
that it is possible to declare the language for each individual bit of text
but now I know.
Thanks for the tutorial!
Best,
JimBreen
> Adam Rice writes:
> > I've created a test page where you can see this effect in action:
> >http://8stars.org/misc/test_unihan.html
> Thanks, Adam. That illustrates the effect perfectly.
>
> I think this is a perfect illustration of the importance of page authors
> declaring the language of text. Not being very HTML-savvy, I did not realize
> that it is possible to declare the language for each individual bit of text
> but now I know.
sensitive to to the locale/language, NOT by adding extra codepoints to
cater for minor glyph differences. Imagine what text searching and
indexing would be like if we had different codepoints for Italic,
Gothic,
etc, in alphabetic languages.
Jim
Alan Siegrist
> > I think this is a perfect illustration of the importance of page authors
> > declaring the language of text. Not being very HTML-savvy, I did not
> > realize that it is possible to declare the language for each individual
> > bit of text but now I know.
>
> The issue is appropriately handled by having display/rendering systems
> sensitive to to the locale/language, NOT by adding extra codepoints to
> cater for minor glyph differences.
Yes, I see that now. I think it would be very helpful if display systems
like browsers would allow the user to change the locale/language setting on
the fly if the page author neglects to declare a specific locale or
language. This could be done in the same way that one can set the encoding
properly if you see mojibake on a page.
> Imagine what text searching and indexing would be like if we had different
> codepoints for Italic, Gothic, etc, in alphabetic languages.
Yes, we see this difficulty now when searching in Korean, for example. There
are actually two completely different sets of codepoints for the hanja of
common Korean names such as 金 (Kim), 李 (Lee) and 林 (Lim), despite the
respective character forms being identical. It appears to be random as to
which of the two codepoints would be used by any particular page author,
since both character forms look identical.
If one is searching for the name of a specific Korean person in hanja, it is
often necessary to do two different searches with two sets of characters to
be sure to get all hits. The number of searches required increases further
if the name has multiple characters with non-unique codepoints.
Regards,
Keith Wilkinson
JimBreen
> ...... I think it would be very helpful if display systems
> the fly if the page author neglects to declare a specific locale or
> language. This could be done in the same way that one can set the encoding
> properly if you see mojibake on a page.
need is to get the people who create text in WWW, etc. pages to
add the language tags, especially if there are mixed languages.
That W3C page Keith posted is on the message.
> > Imagine what text searching and indexing would be like if we had different
> > codepoints for Italic, Gothic, etc, in alphabetic languages.
>
> Yes, we see this difficulty now when searching in Korean, for example. There
> are actually two completely different sets of codepoints for the hanja of
> common Korean names such as 金 (Kim), 李 (Lee) and 林 (Lim), despite the
> respective character forms being identical. It appears to be random as to
> which of the two codepoints would be used by any particular page author,
> since both character forms look identical.
>
> If one is searching for the name of a specific Korean person in hanja, it is
> often necessary to do two different searches with two sets of characters to
> be sure to get all hits. The number of searches required increases further
> if the name has multiple characters with non-unique codepoints.
more than
one reading, so they did a codepoint for each. (Imagine if JIS codes
had
15+ codepoints for 生.) The old Taiwanese "BIG5" code had one hanzi
with
two codepoints, but that was a mistake. 8-)
Oh, and the North Korean code tables (yes, they have their own) have
special codepoints for the characters for Kim Jong-Il and Kim Il-Sung.
Cheers
Jim
Keith Wilkinson
known:
http://www.w3.org/TR/i18n-html-tech-lang/#ri20040429.113217290
Keith Wilkinson
http://www.w3.org/TR/css3-ruby
https://addons.mozilla.org/en-US/firefox/addon/1935
https://addons.mozilla.org/ja/firefox/addon/6812
https://addons.mozilla.org/en-US/firefox/addon/6178
https://addons.mozilla.org/en-US/firefox/addon/2471 rikaichan
https://addons.mozilla.org/ja/thunderbird/addon/145 dictionary
Keith Wilkinson
Alan Siegrist
> As I heard it, the multiple codepoints are because those hanja have more
> than one reading, so they did a codepoint for each.
I would be interested in hearing more about this.
> (Imagine if JIS codes had 15+ codepoints for 生.)
That would certainly be a mess. That does not seem to make sense, so I
wonder why a similar thing was done for Korean, if that was in fact the
case.
> Oh, and the North Korean code tables (yes, they have their own) have
> special codepoints for the characters for Kim Jong-Il and Kim Il-Sung.
What is the Korean word for "cult of personality"?
JimBreen
> Jim Breen writes:
> > As I heard it, the multiple codepoints are because those hanja have more
> > than one reading, so they did a codepoint for each.
>
> I would be interested in hearing more about this.
the 1st edition and 143+ in the 2nd.
"Hanja [..] with multiple readings are encoded more than once."
"Due to the multiple readings of some hanja, 268 of the 4,888
hanja in KS X 1001:2004 are genuine duplicate characters ...
this effectively means there are 4,620 unique hanja in
KS X 1001..."
> > (Imagine if JIS codes had 15+ codepoints for 生.)
>
> That would certainly be a mess. That does not seem to make sense, so I
> wonder why a similar thing was done for Korean, if that was in fact the
> case.
Cheers
Jim
Brian Chandler
> Jim Breen writes:
>
> > As I heard it, the multiple codepoints are because those hanja have more
> > than one reading, so they did a codepoint for each.
>
>
> That would certainly be a mess. That does not seem to make sense, so I
> wonder why a similar thing was done for Korean, if that was in fact the
> case.
to sort it without further information (i.e. the readings). If in
Korean all but a very small number of characters have unambiguous
readings, then one way to make text sortable is to embed the reading
in alternative "versions" of the same (visual) character. I presume
this was the strategy that the Korean committee adopted -- it solves
one problem, even if it leaves others (like visual cut-n-paste is not
reliable).
The basis of Unicode is not to get involved in weighing up tens, if
not hundreds, of competing ideas about how character encoding might
best be done, but simply to provide the functionality of each of the
national character sets. So if Korean has two copies of a character,
so does Unicode. Just as Unicode has two copies of (Roman) 'A', one
for the living fossils of "EAST ASIAN FULL WIDTH MUMBLE JUMBLE". and
the other for most other versions. This didn't mean that the Unicode
people thought it was a good idea to have different codepoints for
different typesetting conventions, any more than they (necessarily)
thought it was a good idea to have different codepoints for the same
character pronounced differently -- everything is about providing
"round-trip" compatibility with the existing national character sets.
Notice that although there's no reason to suppose that "EAST ASIAN"
typesetting of Cyrillic is any different from [the same] typesetting
of Roman, there are no Unicode "full-width" Cyrillic letters -- not
because of a sudden attack of reason, but because there is no national
character set that includes two typographical variants of the Cyrillic
alphabet.
The other thing to remember is that every time a distinction is added,
a commonality is lost... (You need to paste each of these *different*
keywords into Google to see)
MOCKBA : 229,000 googits
МОСКВА : 193 million
МOCKBA : 6870
МОСКBА : 238
(It's hard to made too much sense of the numbers: I'm using
http://winrus.com/klava.htm for input, and some mixed versions provide
more hits than the pure Roman 229,000. Also, once you've typed
something, it's more or less impossible to work out which it was.)
Brian Chandler
JimBreen
> The basis of Unicode is not to get involved in weighing up tens, if
> not hundreds, of competing ideas about how character encoding might
> best be done, but simply to provide the functionality of each of the
> national character sets.
hanzi, kanji and hanja, where the national sets were merged and
rationalized.
> ........ So if Korean has two copies of a character,
> so does Unicode.
Yes, but only one in the regular Unicode set. The other
KZ X 1001 duplicates are pushed off into the F900+
"compatibility" block for people who worry about
round-trip issues.
> Just as Unicode has two copies of (Roman) 'A', one
> for the living fossils of "EAST ASIAN FULL WIDTH MUMBLE JUMBLE". and
> the other for most other versions.
> ..... This didn't mean that the Unicode
> different typesetting conventions, any more than they (necessarily)
> thought it was a good idea to have different codepoints for the same
> character pronounced differently -- everything is about providing
> "round-trip" compatibility with the existing national character sets.
My hope is that all of these will eventually wither away,
including the horrible 半角カナ.
Jim