> In iso-2022-jp-3, there are even more ways to encode the same string.
Wow, that's not easy to see why would anyone ever want that? Is there any logic behind this?
In your samples both of unicode("\x1b(BHallo","iso-2022-jp") and unicode("\x1b(JHallo","iso-2022-jp") give u"Hallo" -- does this mean that the ignored/lost bytes in the original strings are not illegal but *represent nothing* in this encoding?
I.e. in practice (in a context limited to the encoding in question) should this be considered as a data loss, or should these strings be considered "equivalent"?
On Fri, 28 Dec 2007 03:00:59 -0800, mario wrote: > On Dec 27, 7:37 pm, "Martin v. Löwis" <mar...@v.loewis.de> wrote: >> Certainly. ISO-2022 is famous for having ambiguous encodings. Try >> these:
>> In iso-2022-jp-3, there are even more ways to encode the same string.
> Wow, that's not easy to see why would anyone ever want that? Is there > any logic behind this?
> In your samples both of unicode("\x1b(BHallo","iso-2022-jp") and > unicode("\x1b(JHallo","iso-2022-jp") give u"Hallo" -- does this mean > that the ignored/lost bytes in the original strings are not illegal > but *represent nothing* in this encoding?
They are not lost or ignored but escape sequences that tell how the following bytes should be interpreted '\x1b(B' switches to ASCII and '\x1b(J' to some "roman" encoding which is a superset of ASCII, so it doesn't matter which one you choose unless the following bytes are all ASCII. And of course you can use that escape prefix as often as you want within a string of ASCII byte values.
> I.e. in practice (in a context limited to the encoding in question) > should this be considered as a data loss, or should these strings be > considered "equivalent"?
Equivalent I would say. As Unicode they contain the same characters. Just differently encoded as bytes.
Now also suppose that A, B, and C are not completely different, but have slight overlap - and you get ambiguous encodings.
ISO-2022 works that way. IPSJ maintains a registry of character sets for ISO, and assigns escape codes to them. There are currently about 200 character sets registered.
Somebody decoding this would have to know all the character sets (remember it's a growing registry), hence iso-2022-jp restricts the character sets that you can use for that particular encoding. (Likewise, iso-2022-kr also restricts it, but to a different set of sets).
It's a mess, sure, and one of the primary driving force of Unicode (which even has the unification - ie. lack of ambiguity - in its name).
> In your samples both of unicode("\x1b(BHallo","iso-2022-jp") and > unicode("\x1b(JHallo","iso-2022-jp") give u"Hallo" -- does this mean > that the ignored/lost bytes in the original strings are not illegal > but *represent nothing* in this encoding?
See above, and Marc's explanation. ESC ( B switches to "ISO 646, USA Version X3.4 - 1968"; ESC ( J to "ISO 646, Japanese Version for Roman Characters JIS C6220-1969"
These are identical, except for the following differences: - The USA version has "reverse solidus" at 5/12; the Japanese version "Yen sign" - The USA version has "Tilde (overline; general accent)" at 7/14 (depicted as tilde); the Japanese version "Overline" (depicted as straight overline) - The Japanese version specifies that you can switch between roman and katakana mode by sending shift out (SO, '\x0e') and shift-in (SI, '\x0F') respectively; this switches to the "JIS KATAKANA character set". (source: http://www.itscj.ipsj.or.jp/ISO-IR/006.pdf http://www.itscj.ipsj.or.jp/ISO-IR/014.pdf )
> I.e. in practice (in a context limited to the encoding in question) > should this be considered as a data loss, or should these strings be > considered "equivalent"?
These particular differences should be considered as irrelevant. There are some cases where Unicode had introduced particular compatibility characters to accommodate such encodings (specifically, the "full-width" latin (*) and "half-width" Japanese characters). Good codecs are supposed to round-trip the relevant differences to Unicode, and generate the appropriate compatibility characters.
Bad codecs might not, and in some cases, users might complain that certain compatibility characters are lacking in Unicode so that correct round-tripping is not possible. I believe the Unicode consortium has resolved all these complaints by adding the missing characters; but I'm not sure.
Regards, Martin
(*) As an example for full-width characters, consider these two strings: Ｈｅｌｌｏ Hello Should they be equivalent, or not? They are under NFKD, but not NFD.
Thanks a lot Martin and Marc for the really great explanations! I was wondering if it would be reasonable to imagine a utility that will determine whether, for a given encoding, two byte strings would be equivalent. But I think such a utility will require *extensive* knowledge about many bizarrities of many encodings -- and has little chance of being pretty!
In any case, it goes well beyond the situation that triggered my original question in the first place, that basically was to provide a reasonable check on whether round-tripping a string is successful -- this is in the context of a small utility to guess an encoding and to use it to decode a byte string. This utility module was triggered by one that Skip Montanaro had written some time ago, but I wanted to add and combine several ideas and techniques (and support for my usage scenarios) for guessing a string's encoding in one convenient place. I provide a write-up and the code for it here:
> Thanks a lot Martin and Marc for the really great explanations! I was > wondering if it would be reasonable to imagine a utility that will > determine whether, for a given encoding, two byte strings would be > equivalent.
But that is much easier to answer:
s1.decode(enc) == s2.decode(enc)
Assuming Unicode's unification, for a single encoding, this should produce correct results in all cases I'm aware of.
If the you also have different encodings, you should add
This would flatten out compatibility characters, and ambiguities left in Unicode itself.
> But I think such a utility will require *extensive* > knowledge about many bizarrities of many encodings -- and has little > chance of being pretty!
See above.
> In any case, it goes well beyond the situation that triggered my > original question in the first place, that basically was to provide a > reasonable check on whether round-tripping a string is successful -- > this is in the context of a small utility to guess an encoding and to > use it to decode a byte string. This utility module was triggered by > one that Skip Montanaro had written some time ago, but I wanted to add > and combine several ideas and techniques (and support for my usage > scenarios) for guessing a string's encoding in one convenient place.
Notice that this algorithm is not capable of detecting the ISO-2022 encodings - they look like ASCII to this algorithm. This is by design, as the encoding was designed to only use 7-bit bytes, so that you can safely transport them in Email and such (*)
If you want to add support for ISO-2022, you should look for escape characters, and then check whether the escape sequences are among the ISO-2022 ones: - ESC ( - 94-character graphic character set, G0 - ESC ) - 94-character graphic character set, G1 - ESC * - 94-character graphic character set, G2 - ESC + - 94-character graphic character set, G3 - ESC - - 96-character graphic character set, G1 - ESC . - 96-character graphic character set, G2 - ESC / - 96-character graphic character set, G3 - ESC $ - Multibyte ( G0 ) G1 * G2 + G3 - ESC % - Non-ISO-2022 (e.g. UTF-8)
If you see any of these, it should be ISO-2022; see the Wiki page as to what subset may be in use.
G0..G3 means what register the character set is loaded into; when you have loaded a character set into a register, you can switch between registers through ^N (to G1), ^O (to G0), ESC n (to G2), ESC o (to G3) (*)
> > Thanks a lot Martin and Marc for the really great explanations! I was > > wondering if it would be reasonable to imagine a utility that will > > determine whether, for a given encoding, two byte strings would be > > equivalent.
> But that is much easier to answer:
> s1.decode(enc) == s2.decode(enc)
> Assuming Unicode's unification, for a single encoding, this should > produce correct results in all cases I'm aware of.
> If the you also have different encodings, you should add
> This would flatten out compatibility characters, and ambiguities > left in Unicode itself.
Hmmn, true, it would be that easy.
I am now not sure why I needed that check, or how to use this version of it... I am always starting from one string, and decoding it... that may be lossy when that is re-encoded, and compared to original. However it is clear that the test above should always pass in this case, so doing it seems superfluos.
On Jan 2, 9:34 pm, "Martin v. Löwis" <mar...@v.loewis.de> wrote:
> > In any case, it goes well beyond the situation that triggered my > > original question in the first place, that basically was to provide a > > reasonable check on whether round-tripping a string is successful -- > > this is in the context of a small utility to guess an encoding and to > > use it to decode a byte string. This utility module was triggered by > > one that Skip Montanaro had written some time ago, but I wanted to add > > and combine several ideas and techniques (and support for my usage > > scenarios) for guessing a string's encoding in one convenient place.
> Notice that this algorithm is not capable of detecting the ISO-2022 > encodings - they look like ASCII to this algorithm. This is by design, > as the encoding was designed to only use 7-bit bytes, so that you can > safely transport them in Email and such (*)
Well, one could specify decode_heuristically(s, enc="iso-2022-jp") and that encoding will be checked before ascii or any other encoding in the list.
> If you want to add support for ISO-2022, you should look for escape > characters, and then check whether the escape sequences are among > the ISO-2022 ones: > - ESC ( - 94-character graphic character set, G0 > - ESC ) - 94-character graphic character set, G1 > - ESC * - 94-character graphic character set, G2 > - ESC + - 94-character graphic character set, G3 > - ESC - - 96-character graphic character set, G1 > - ESC . - 96-character graphic character set, G2 > - ESC / - 96-character graphic character set, G3 > - ESC $ - Multibyte > ( G0 > ) G1 > * G2 > + G3 > - ESC % - Non-ISO-2022 (e.g. UTF-8)
> If you see any of these, it should be ISO-2022; see > the Wiki page as to what subset may be in use.
> G0..G3 means what register the character set is loaded > into; when you have loaded a character set into a register, > you can switch between registers through ^N (to G1), > ^O (to G0), ESC n (to G2), ESC o (to G3) (*)
OK, suppose we do not know the string is likely to be iso-2022, but we still want to detect it if it is. I have added a "may_do_better" mechanism to the algorithm, to add special checks on a *guessed* algorithm. I am not sure this will not however introduce more or other problems than the one it is addressing...
I have re-instated checks for iso-8859-1 control chars (likely to be cp1252), for special symbols in iso-8859-15 when they occur in iso-8859-1 and cp1252, and for the iso-2022-jp escape sequences. To flesh out with other checks is mechanical work...
I still have issues with what happens in situations when for example a file contains iso-2022 esc sequences but is anyway actally in ascii or utf-8? e.g. this mail message! I'll let this issue turn for a little while...
> > I will be very interested in any remarks any of you may have!
> From a shallow inspection, it looks right. I would have spelled > "losses" as "loses".
