Re: unnecessary translation between dialects
Helmut Wollmersdorfer
> I was not talking just about recognizing languages. I was talking
> about problems of identifying character-to-byte(s) format encoding
> algorithm that was used by the message sender.
> Your above list of frequently occurring words cannot be identified
> without knowing how the characters were encoded into single
> or multiple bytes.
Exactly. Therefore you need the best match against language AND
encoding. You cannot guess the encoding without guessing the language.
> But if it isn't Unicode then the identification is not
> so easy.
Been there, done that. It's not so easy, but it's easy enough.
Helmut Wollmersdorfer
Ruud Harmsen
<hel...@wollmersdorfer.at>: in sci.lang:
>pauljk wrote:
>
>> I was not talking just about recognizing languages. I was talking
>> about problems of identifying character-to-byte(s) format encoding
>> algorithm that was used by the message sender.
>
>> Your above list of frequently occurring words cannot be identified
>> without knowing how the characters were encoded into single
>> or multiple bytes.
>
>Exactly. Therefore you need the best match against language AND
>encoding. You cannot guess the encoding without guessing the language.
Not always true. UTF8 is pretty well detectable in pretty much any
language.
>
> > But if it isn't Unicode then the identification is not
> > so easy.
>
>Been there, done that. It's not so easy, but it's easy enough.
Right.
--
Ruud Harmsen, http://rudhar.com
Aidan Kehoe
Ar an seachtú lá is fiche de mí na Nollaig, scríobh Ruud Harmsen:
> Mon, 27 Dec 2010 21:35:41 +0100: Helmut Wollmersdorfer
> <hel...@wollmersdorfer.at>: in sci.lang:
>
> >pauljk wrote:
> >
> >> I was not talking just about recognizing languages. I was talking
> >> about problems of identifying character-to-byte(s) format encoding
> >> algorithm that was used by the message sender.
> >
> >> Your above list of frequently occurring words cannot be identified
> >> without knowing how the characters were encoded into single
> >> or multiple bytes.
> >
> >Exactly. Therefore you need the best match against language AND
> >encoding. You cannot guess the encoding without guessing the language.
>
> Not always true. UTF8 is pretty well detectable in pretty much any
> language.
It doesn’t have a huge impact on real-world applications, but to be exact,
English-language text in ISO 8859-1 and the same text in UTF-8 are often,
maybe usually, identical.
--
“Apart from the nine-banded armadillo, man is the only natural host of
Mycobacterium leprae, although it can be grown in the footpads of mice.”
-- Kumar & Clark, Clinical Medicine, summarising improbable leprosy research
Helmut Richter
> Ar an seachtú lá is fiche de mí na Nollaig, scríobh Ruud Harmsen:
> > Not always true. UTF8 is pretty well detectable in pretty much any
> > language.
>
> It doesn't have a huge impact on real-world applications, but to be exact,
> English-language text in ISO 8859-1 and the same text in UTF-8 are often,
> maybe usually, identical.
Most texts I deal with are German, where this is not so. But I was enough
fed up with coding issues of ISO-8859-1 and UTF-8 that I wrote a piece of
code which would accept both, or any mixture of them, and transform it
into UTF-8. Sequences of ISO-8859-1 characters that could also be
interpreted as UTF-8 characters do not occur in the wild.
--
Helmut Richter
Peter Moylan
> Mon, 27 Dec 2010 21:35:41 +0100: Helmut Wollmersdorfer
> <hel...@wollmersdorfer.at>: in sci.lang:
>
>> pauljk wrote:
>>
>>> I was not talking just about recognizing languages. I was talking
>>> about problems of identifying character-to-byte(s) format encoding
>>> algorithm that was used by the message sender.
>>> Your above list of frequently occurring words cannot be identified
>>> without knowing how the characters were encoded into single
>>> or multiple bytes.
>> Exactly. Therefore you need the best match against language AND
>> encoding. You cannot guess the encoding without guessing the language.
>
> Not always true. UTF8 is pretty well detectable in pretty much any
> language.
In fact you can pick any of the Unicode variants by examining the first
four bytes in the file, independently of language.
>>> But if it isn't Unicode then the identification is not
>>> so easy.
>> Been there, done that. It's not so easy, but it's easy enough.
>
> Right.
--
Peter Moylan, Newcastle, NSW, Australia. http://www.pmoylan.org
For an e-mail address, see my web page.
Ruud Harmsen
in sci.lang:
> > >Exactly. Therefore you need the best match against language AND
> > >encoding. You cannot guess the encoding without guessing the language.
> >
> > Not always true. UTF8 is pretty well detectable in pretty much any
> > language.
>
>It doesn’t have a huge impact on real-world applications, but to be exact,
>English-language text in ISO 8859-1 and the same text in UTF-8 are often,
>maybe usually, identical.
Of course. That's because both share ASCII as a subset.
Ruud Harmsen
>> It doesn't have a huge impact on real-world applications, but to be exact,
>> English-language text in ISO 8859-1 and the same text in UTF-8 are often,
>> maybe usually, identical.
Mon, 27 Dec 2010 22:38:46 +0100: Helmut Richter <hh...@web.de>: in
sci.lang:
>Most texts I deal with are German, where this is not so. But I was enough
>fed up with coding issues of ISO-8859-1 and UTF-8 that I wrote a piece of
>code which would accept both, or any mixture of them, and transform it
>into UTF-8. Sequences of ISO-8859-1 characters that could also be
>interpreted as UTF-8 characters do not occur in the wild.
I did the same years ago (2 years, actually):
http://rudhar.com/lingtics/utf8mixd/
Except that it works the other way around: the output is ISO8859-1 or
optionally CP1252.
Ruud Harmsen
<inv...@peter.pmoylan.org.invalid>: in sci.lang:
>> Not always true. UTF8 is pretty well detectable in pretty much any
>> language.
>
>In fact you can pick any of the Unicode variants by examining the first
>four bytes in the file, independently of language.
If the file has such a prefix, which in the case of UTF8 is
unnecessary and unrequired.
UTF8 does not have a byte order issue, like most other Unicode
encodings have.
Helmut Richter
> Mon, 27 Dec 2010 22:38:46 +0100: Helmut Richter <hh...@web.de>: in
> sci.lang:
> >Most texts I deal with are German, where this is not so. But I was enough
> >fed up with coding issues of ISO-8859-1 and UTF-8 that I wrote a piece of
> >code which would accept both, or any mixture of them, and transform it
> >into UTF-8. Sequences of ISO-8859-1 characters that could also be
> >interpreted as UTF-8 characters do not occur in the wild.
>
> I did the same years ago (2 years, actually):
> http://rudhar.com/lingtics/utf8mixd/
>
> Except that it works the other way around: the output is ISO8859-1 or
> optionally CP1252.
Yes, this is an additional option which, alas, accounts for most of the code.
UTF-8 target code is easy (about 10 lines of Perl) but other target code
requires dealing with characters that are not in the target code. For
instance, a c with a hacek should be rendered as c, not as ? or something
else.
--
Helmut Richter
Adam Funk
Sounds extremely useful! Is it publicly available, or would you mind
sharing it?
--
Le beau est aussi utile que l'utile. [Victor Hugo]
Helmut Richter
It is rather new with the more sophisticated options. Give me two more
weeks of testing so that the most embarrassing blunders have a chance of
being removed. (Of course, there is never any warranty.)
--
Helmut Richter
Adam Funk
I recently discovered this Perl module, which you might find useful
(well, only if you use Perl).
http://search.cpan.org/~jgmyers/Encode-Detect-0.01/Detect.pm
Ubuntu package libencode-detect-perl
I've only used it a little, but so far it seems to work quite well and
it's easy to use. I don't know how it would work on a "mixed" string
containing more than one encoding, though.
--
It is probable that television drama of high caliber and produced by
first-rate artists will materially raise the level of dramatic taste
of the nation. (David Sarnoff, CEO of RCA, 1939; in Stoll 1995)