Locale to Unicode Codepoint mapping?
Peter Olcott
every regional dialect of a human language. How can I go
about finding this information?
Paavo Helde
_K_WnZ2dnU...@giganews.com:
> I need to know the set of Unicode codepoints associated with
> every regional dialect of a human language. How can I go
> about finding this information?
James Kanze
www.unicode.org. Go to the code charts under the standard.
In general, for such things, look for the reference site. This
one is actually very well done, but even in the case of those
that are less well done, the reference is the reference.
--
James Kanze
Peter Olcott
"James Kanze" <james...@gmail.com> wrote in message
news:991f32d8-319c-45ad...@s31g2000yqs.googlegroups.com...
I already studied these for many hours. They do not have
what I need. I need to know the exact subset of Unicode code
points that apply to a specific dialect of a specific
language. A superset of these will not do, I must have the
minimum set of code points. This has to be known because the
keyboard must know which glyphs to produce.
I am working on internationalizing this system:
www.OCR4Screen.com
Paavo Helde
5ZSW6nWnZ2d...@giganews.com:
>
> "James Kanze" <james...@gmail.com> wrote in message
> news:991f32d8-319c-45ad...@s31g2000yqs.googlegroups.com...
>> On Dec 23, 6:27 pm, "Peter Olcott" <NoS...@SeeScreen.com>
>> wrote:
>>> I need to know the set of Unicode codepoints associated
>>> with
>>> every regional dialect of a human language. How can I go
>>> about finding this information?
>>
>> www.unicode.org. Go to the code charts under the
>> standard.
>>
>> In general, for such things, look for the reference site.
>> This
>> one is actually very well done, but even in the case of
>> those
>> that are less well done, the reference is the reference.
>>
>> --
>> James Kanze
>
> I already studied these for many hours. They do not have
> what I need. I need to know the exact subset of Unicode code
> points that apply to a specific dialect of a specific
> language. A superset of these will not do, I must have the
> minimum set of code points.
Do you realize that there are thousands of languages and dialects?
WikiAnswers offers a number like 2261 for the number of languages having a
writing system.
In any case, the local alphabete ultimately depends on the local national
decisions and conventions - which characters are considered to belong to
the national character set, and which are not. This can change over time.
> This has to be known because the
> keyboard must know which glyphs to produce.
Knowing what characters some vendor has decided to include in a keyboard
mapping for given language might help of course. However, this must not be
relied on too much.
> I am working on internationalizing this system:
> www.OCR4Screen.com
>
For OCR-like tasks one also has to consider that text can often contain
foreign characters and even words. For example, in Russian technical texts
it is common to type technical terms in Latin alphabete. To make things
worse, there are many symbols which have the same look in Cyrillic and in
Latin, but correspond to different Unicode code points. Recognizing them
correctly is no easy task.
5-minute Google search produced these pages, maybe these are of any help:
http://developer.mimer.com/charts/index.tml
http://www.evertype.com/alphabets/
Paavo
Peter Olcott
"Paavo Helde" <myfir...@osa.pri.ee> wrote in message
news:Xns9CECB7082...@216.196.109.131...
Those links are exactly what I needed, thanks!
James Kanze
> "James Kanze" <james.ka...@gmail.com> wrote in message
> news:991f32d8-319c-45ad...@s31g2000yqs.googlegroups.com...
> > On Dec 23, 6:27 pm, "Peter Olcott" <NoS...@SeeScreen.com>
> > wrote:
> >> I need to know the set of Unicode codepoints associated
> >> with every regional dialect of a human language. How can I
> >> go about finding this information?
> >www.unicode.org. Go to the code charts under the standard.
> > In general, for such things, look for the reference site.
> > This one is actually very well done, but even in the case of
> > those that are less well done, the reference is the
> > reference.
> I already studied these for many hours. They do not have what
> I need. I need to know the exact subset of Unicode code points
> that apply to a specific dialect of a specific language.
I doubt you'll find much more. I'm not even sure that "the
exact subset of Unicode code points that apply to a specific
dialect of a specific language" has a concrete meaning. Local
dialects generally don't have formal spellings, and different
authors treat them differently. And the formalized written
forms of languages tend to contain a lot of accents that only
show up in one or two words (e.g. "naïve" in English); trying to
root out all of those accents is probably a hopeless task.
> A superset of these will not do, I must have the minimum set
> of code points. This has to be known because the keyboard must
> know which glyphs to produce.
What does the keyboard have to do with this? It's a known
problem that the keyboard doesn't have enough keys for most
languages, and that various alternatives have been developed.
(Even in English: does your keyboard have "“"? It's a
standard character in both American and British English.)
> I am working on internationalizing this system:
> www.OCR4Screen.com
Wouldn't that entail understanding all characters, even if they
wouldn't normally be used in the current locale?
--
James Kanze
James Kanze
> "Peter Olcott" <NoS...@SeeScreen.com> wrote in news:EMCdnTXL-
[...]
> In any case, the local alphabete ultimately depends on the
> local national decisions and conventions - which characters
> are considered to belong to the national character set, and
> which are not. This can change over time.
Exactly. It can also vary regionally---Swiss German doesn't use
exactly the same character set as standard German---, and
culturally---if nothing else, most languages use a different set
of characters when the data must be entered via a standard
keyboard.
> > This has to be known because the keyboard must know which
> > glyphs to produce.
> Knowing what characters some vendor has decided to include in
> a keyboard mapping for given language might help of course.
> However, this must not be relied on too much.
Especially as, as mentionned above, keyboard input often follows
different rules: in French, the standard double quotes would be
« ... » ; when entering text from a keyboard, " ... " is
considered acceptable (although normally, French doesn't have a
" character).
> > I am working on internationalizing this system:
> > www.OCR4Screen.com
> For OCR-like tasks one also has to consider that text can
> often contain foreign characters and even words. For example,
> in Russian technical texts it is common to type technical
> terms in Latin alphabete. To make things worse, there are many
> symbols which have the same look in Cyrillic and in Latin, but
> correspond to different Unicode code points. Recognizing them
> correctly is no easy task.
> 5-minute Google search produced these pages, maybe these are of any help:
> http://developer.mimer.com/charts/index.tml
> http://www.evertype.com/alphabets/
Just a note, very relevant with regards to your previous comment
about changing over time: the list of characters for French in
the second doesn't include ü. Which is present in words like
aigüe. At least since the last spelling reforme---before, one
would have written aiguë. (But ü was still present in names
like Saül. And ÿ, which is given, only occurs in five or six
proper nouns---all the names of small villages.)
--
James Kanze