the utf8 flag (was Re: [perl #41527] decode_utf8 sets utf8 flag on plain ascii strings)
Darren Duncan
> The documentation for the 'decode' function in Encode.pm states:
>
> ...the utf8 flag for $string is on unless $octets entirely
> consists of ASCII data...
>
> but it appears that decode turns on the flag even if the input string is
> plain ASCII. A test case demonstrating this is appended below.
Considering that I like to write modern programs that simply use
Unicode end-to-end as possible, and at least internally, which keeps
everything simple and compatible, it would be easier for me if the
meaning of the utf8 flag was updated to officially be the new
behaviour.
I believe that a true utf8 flag should mean that the string contains
data that is valid utf8, not just that it has utf8 characters outside
the ASCII range.
As far as I know, the conceptual purpose of the utf8 flag is to
indicate whether Perl considers a string to be unambiguous character
data or binary data which could be ambiguous character data, and thus
how Perl will treat it by default.
If I have a library that wants to work internally with unambiguous
character data, and to keep things simple will require the user code
to remove any ambiguity by doing any decoding itself and passing the
library the result, then it would be simpler if the input checking
code of the library could just do this:
sub expects_text {
my ($v) = @_;
confess q{Bad arg; it is undefined.}
if !defined $v;
confess q{Bad arg; Perl 5 does not consider it to be a char str.}
if !Encode::is_utf8( $v );
# $v is okay, so do whatever ...
}
Instead, the older documented utf8 flag behaviour would require this
unnecessary extra work in order to accept all valid input:
sub expects_text {
my ($v) = @_;
confess q{Bad arg; it is undefined.}
if !defined $v;
confess q{Bad arg; Perl 5 does not consider it to be a char str.}
if !Encode::is_utf8( $v ) and $v =~ m/[^\x00-\x7F]/xs;
# $v is okay, so do whatever ...
}
I would expect the use of the regular expression, which would be
called for any ASCII data, would be considerably slower than just
checking the flag, especially since we already know the data is valid
Unicode characters in order for decode() to possibly set the flag in
the first place.
Now, if there is some concern that character-oriented regexes and
such are considerably slower for ASCII data than alternatives, and
this is a problem and it can't be otherwise dealt with, we could
perhaps have an additional flag which has the meaning that I ascribed
to utf8; eg, is_chars() or is_text() etcetera; but in my mind it
would be simpler to just leave the meaning of is_utf8 adjusted to
mean is unambiguous character data.
Thank you. -- Darren Duncan
P.S. On a tangent, it would be nice if there was a simple test to
see if an SV currently considered its numerical or integer or string
etc component to be the authoratative one, so eg I could just check
that rather than using looks_like_number or some such more
complicated solution. Though maybe there is already, perhaps in a
bundled debugging or some such module, and I haven't found it yet?
Juerd Waalboer
> I believe that a true utf8 flag should mean that the string contains
> data that is valid utf8, not just that it has utf8 characters outside
> the ASCII range.
How often should Perl check for this? Directly after decoding only, or
also after mutating operations like substr, or s///?
> As far as I know, the conceptual purpose of the utf8 flag is to
> indicate whether Perl considers a string to be unambiguous character
> data or binary data which could be ambiguous character data, and thus
> how Perl will treat it by default.
The *conceptual* purpose of the UTF8 flag isn't there. Conceptually,
every string can be a unicode string, and you're not supposed to look
at, know, or set the UTF8 flag yourself. It's an internal bit, like IOK
and NOK. [1]
> confess q{Bad arg; Perl 5 does not consider it to be a char str.}
> if !Encode::is_utf8( $v );
As said, this is not the purpose of the flag, and you're not supposed to
use is_utf8 for this. It is documented with the "[INTERNAL]" flag, for a
good reason.
Perl conceptually has a single numeric type, and a single string type.
The distinction between integer and float, and between iso-8859-1 and
utf-8, is internal.
This could be changed, but will introduce incompatibilities and a severe
loss of performance for strings that fit in iso-8859-1.
What I want (and I think you want too) is a real type system, to have
two different distinct types: byte strings and character strings. It
would be bad to use a flag called "UTF8" for this, because a byte string
can also be UTF8 encoded. Perl already suffers from this problem, but
because the UTF8 flag is *INTERNAL*, it's not a big deal. It would be if
it surfaced and was used by Perl coders.
A whole type system is a bit too much to implement in Perl 5, I think.
Our current unicode string semantics are a great way to deal with not
having types, in my opinion.
> Instead, the older documented utf8 flag behaviour would require this
> unnecessary extra work in order to accept all valid input:
No.
If your subroutine expects text, it can only assume that it gets text,
and it should not (must not?) make any distinction based on the internal
encoding.
The string it gets is a Unicode string. Not a UTF8 string, not a latin1
string.
> if !Encode::is_utf8( $v ) and $v =~ m/[^\x00-\x7F]/xs;
This check is wrong. If the flag is not set, that means only that the
internal encoding is iso-8859-1 if the string is a text string, not that
the string is a byte string.
The reverse is true, however: if the flag is set, the string will not be
a byte string. But lack of UTF8 flag is no indication of byte versus
character.
> I would expect the use of the regular expression, which would be
> called for any ASCII data
Note that Perl internally uses iso-8859-1 (8 bit) and utf-8 (variable
whole-octet), not ascii (7 bit).
The character Γ© (eacute) may be stored internally as the single octet
233 (decimal) and does not by itself cause an internal upgrade to UTF-8.
[1] Some parts of Perl break this concept. The regex engine is one of
them, and has different semantics depending on the presence of the flag.
This is a bug, but any fix would be incompatible.
--
korajn salutojn,
juerd waalboer: perl hacker <ju...@juerd.nl> <http://juerd.nl/sig>
convolution: ict solutions and consultancy <sa...@convolution.nl>
Ik vertrouw stemcomputers niet.
Zie <http://www.wijvertrouwenstemcomputersniet.nl/>.
John Berthels
> Unicode end-to-end as possible, and at least internally, which keeps
> everything simple and compatible, it would be easier for me if the
> meaning of the utf8 flag was updated to officially be the new
> behaviour.
Well, perl goes to some lengths (implicit conversion) for you to be
able to mix untagged-all-ascii string values and tagged-non-ascii
transparently in your program. And you can happily write modern
programs using Unicode end-to-end doing so. Both types of strings
consist of character data.
> I believe that a true utf8 flag should mean that the string contains
> data that is valid utf8, not just that it has utf8 characters outside
> the ASCII range.
Well, I think is_utf8 is poorly named either way (with several years
of hindsight - I don't think I would have made a better choice at the
time). I don't think that Perl's internal representation for unicode
strings is guaranteed to be utf8. The flag more properly means "please
treat this as character data, taking special care to realise that some
of the character values may be > 255". And it's the 'special care' bit
which can cost performance.
> As far as I know, the conceptual purpose of the utf8 flag is to
> indicate whether Perl considers a string to be unambiguous character
> data or binary data which could be ambiguous character data, and thus
> how Perl will treat it by default.
Yes, agreed. And it's really a bit of perl's internals which
application code shouldn't really want to examine or change directly.
[snip example of using is_utf8 to check that a perl value contains
'character data']
Why would your library routine care? It can manipulate the string as a
sequence of characters in either case. It will produce the wrong
results if passed the wrong data, but that will always be true, since
it could be passed wrong data tagged as utf8. If your routine wants
specific sequences of characters it can check for those, regardless of
the is_utf8ness of the string.
> Now, if there is some concern that character-oriented regexes and
> such are considerably slower for ASCII data than alternatives, and
> this is a problem and it can't be otherwise dealt with
I think the unicode regex engine can never be as fast as the
byte-oriented one. It has more to consider. There's some example code
(vaguely like the sort of templating where I noticed the problem),
which shows unicode running 2-3 times as slow (17s instead of 6s) as
the byte engine.
> we could
> perhaps have an additional flag which has the meaning that I ascribed
> to utf8; eg, is_chars() or is_text() etcetera; but in my mind it
> would be simpler to just leave the meaning of is_utf8 adjusted to
> mean is unambiguous character data.
I'm having trouble thinking of an example where application code might
want to check this. It's part of perl's internals, surely?
> P.S. On a tangent, it would be nice if there was a simple test to
> see if an SV currently considered its numerical or integer or string
> etc component to be the authoratative one, so eg I could just check
> that rather than using looks_like_number or some such more
> complicated solution. Though maybe there is already, perhaps in a
> bundled debugging or some such module, and I haven't found it yet?
I'd rather is_utf8 disappeared from the public API, since it's really
an internal flag and (I think) poorly named. Internally, it could then
be renamed requires_unicode_engine or something.
But what I really care about is the ability to just tell perl "data
from this source is in this encoding", "data going to this destination
is in this encoding" and get all the nice automagic handling of
conversions for me without paying the unicode engine cost on ascii
data.
regards,
jb
Bench output:
Rate udata data
udata 588/s -- -63%
data 1572/s 167% --
Code:
#!/usr/bin/perl
use warnings;
use strict;
use Encode;
use Benchmark;
my $data = "";
my $count = 10;
while ($count-- > 0) {
$data = "<%-$count tag with some text $data $count-%>";
}
my $udata = $data;
Encode::_utf8_on($udata);
my $do_what = shift || "bench";
my $run_count = shift || 10000;
if ($do_what eq 'bench') {
Benchmark::cmpthese(-20, {
data => sub { stress($data); },
udata => sub { stress($udata); },
});
}
elsif ($do_what eq 'bytes') {
stress($data) for (1..$run_count);
}
elsif ($do_what eq 'chars') {
stress($udata) for (1..$run_count);
}
else {
die "Don't understand what you wanted me to do: $do_what";
}
sub stress {
my $data = shift;
my $oldlen;
while ($data =~ s/<%-(\d+)([^<]*?).*%-\1>/reverse($2)/e) {
if ($oldlen) {
die "didn't match [$data]" unless length $data < $oldlen;
}
$oldlen = length $data;
}
}
Marc Lehmann
> > As far as I know, the conceptual purpose of the utf8 flag is to
> > indicate whether Perl considers a string to be unambiguous character
> > data or binary data which could be ambiguous character data, and thus
> > how Perl will treat it by default.
>
> The *conceptual* purpose of the UTF8 flag isn't there. Conceptually,
> every string can be a unicode string, and you're not supposed to look
> at, know, or set the UTF8 flag yourself. It's an internal bit, like IOK
> and NOK. [1]
Thats not how current perl works.
> Perl conceptually has a single numeric type, and a single string type.
> The distinction between integer and float, and between iso-8859-1 and
> utf-8, is internal.
I would love if that were the case, but the powers to be decided that every
perl progarmmer has to know those internals, and needs to be able to deal
with them.
> Note that Perl internally uses iso-8859-1 (8 bit) and utf-8 (variable
> whole-octet), not ascii (7 bit).
No, Perl exposes this. For example, see the recent example of Compress::Zlib:
unpack ('CCCCVCC', $$string);
that code is broken because the powers to be decided that "C" exposes the
internal encoding, while "V" doesn't. That requires every perl programmer
who decodes file headers etc. using unpack to know about those internals.
This is especially bad as not only has the meaning of "C" been shifted from
decoding bytes to something else (instead of using a new modifier), but no
alternative has been provided to get the old meaning of "C", so basically all
code that doesn't utf8::downgrade is broken now by this change in meaning.
(Worse is the fact that its wrongly documented to decode an octet even in
the presence of Unicode, but it doesn't decode an octet, unless you define
"octet" in Perl to mean that "\xa0" is either one or two octets)
The same is true for many XS modules: in older versions of perl, SvPV gave
you the 8-bit version of a scalar, but in current versions, it randomly
gives you either 8-bit or utf-8 encoded. SvPV was renamed to SvPVbyte.
Both of those gratitiously backwards-incompatible changes break lots of
existing code.
And the problem is that those bugs are not considered bugs but features.
> [1] Some parts of Perl break this concept. The regex engine is one of
> them, and has different semantics depending on the presence of the flag.
> This is a bug, but any fix would be incompatible.
In fact, some parts of perl break this concept and make perfectly working
code (in 5.005) not working anymore, or working randomly, and thats not
considered a bug.
I wonder why it is ok to break large amounts of perl and xs code silently,
without even documenting how to fix it[1], while at the same time 5.10
introduced "use feature" to shield against possible breakage with far less of
an impact then the changes above.
[1] If it is documented, then anybody please show me why this:
utf8::downgrade $s;
unpack "C", $s;
is documented to have different effects from:
unpack "C", $s;
i.e., where is it documented that perl doesn't upgrade the scalar in between
those lines? If you think it is obvious, how about this:
my $s = chr 255; # to me, this is one octet. to perl, it might be one or
# two, or maybe more, who knows.
warn unpack "C", $s;
"$s\x{672c}";
warn unpack "C", $s;
$s .= "\x{672c}"; substr $s, 1, 1, "";
warn unpack "C", $s;
Can a pure-Perl programmer tell what the output of this program is without
trying it? Should he be able to? I would say the answer is no to both.
It is beyond me how people can introduce so much breakage to existing code
so lightly, forcing many modules to be changed and forcing pure-Perl
programmer to understand the perl interpreter sources to get their unicode
right.
Thats a broken unicode model, and as long as those kind of bugs are
considered features, perl programmers very well have to care about that
internal utf-x, utf-8, whatever flag.
--
The choice of a
-----==- _GNU_
----==-- _ generation Marc Lehmann
---==---(_)__ __ ____ __ p...@goof.com
--==---/ / _ \/ // /\ \/ / http://schmorp.de/
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE
Nicholas Clark
> The same is true for many XS modules: in older versions of perl, SvPV gave
> you the 8-bit version of a scalar, but in current versions, it randomly
> gives you either 8-bit or utf-8 encoded. SvPV was renamed to SvPVbyte.
>
> Both of those gratitiously backwards-incompatible changes break lots of
> existing code.
>
> And the problem is that those bugs are not considered bugs but features.
I certainly consider this one a bug.
I didn't create the release that messed this up, and didn't realise the
implications of the change until some time after it happened.
You might consider me slow for this.
> I wonder why it is ok to break large amounts of perl and xs code silently,
> without even documenting how to fix it[1], while at the same time 5.10
> introduced "use feature" to shield against possible breakage with far less of
> an impact then the changes above.
Problem is now that I can't see how to fix it without breaking other code that
plays by the different, new, rules.
Nicholas Clark
Marc Lehmann
> > And the problem is that those bugs are not considered bugs but features.
>
> I certainly consider this one a bug.
So fix it. It is easy to do, and I documented it years ago (during 5.6).
> I didn't create the release that messed this up, and didn't realise the
> implications of the change until some time after it happened.
> You might consider me slow for this.
I do not consider you slow for not creating the release that messed this
up, no :) If it all, its a pity you didn't.
> > I wonder why it is ok to break large amounts of perl and xs code silently,
> > without even documenting how to fix it[1], while at the same time 5.10
> > introduced "use feature" to shield against possible breakage with far less of
> > an impact then the changes above.
>
> Problem is now that I can't see how to fix it without breaking other code that
> plays by the different, new, rules.
I have yet to see that other code outside the testsuite that reliably
relies on broken 5.6 unicoded semantics and is considered worth keeping. I
challenge you to show me, and I promise to show you another example from
CPAN or elsewhere that breaks. OR maybe even two. Or three.
Besides, without any doubt, the code that relies on psuedo-random
behaviour is certainkly in the minority. The amount of code in the wild
that relies on "C" having 5.5 semantics is much larger. I doubt _anybody_
except me (or at leats not very many people) understands that he has to
downgrade scalars before passing them into unpack to decode structures.
As the amount of breakage will only increase over time as unicode becomes
more and more used in perl.
The solution to this bug is fixing. The earlier, the better.
At the very least, it needs to be documented, and *hard* rules on when
perl uphgraded or downgrades would need to be established, as, right now,
behaviour is pretty random over versions. Of course, down that path lies
madness and perl5 will ever stay a failed experiment of how to do unicode
correctly (namely, abstracted away from the actual encoding).
Besides, don't you think an agrument of the form "yes, it breaks lots of
code, but some code might rely on it, so lets keep it" sounds pretty,
sorry to be so honest, stupid?
Marc Lehmann
> At the very least, it needs to be documented, and *hard* rules on when
> perl uphgraded or downgrades would need to be established, as, right now,
> behaviour is pretty random over versions. Of course, down that path lies
> madness and perl5 will ever stay a failed experiment of how to do unicode
> correctly (namely, abstracted away from the actual encoding).
In fact, I teach a lot of people about unicode in perl. And the problem
is not that the unicode model doesn't work or isn't simple, the problem
is those "features" that remind people that perl has no abstract unicode
model, that they do have to understand the internals of the UTF-X bit.
Thats really the problem: if perl had a pure 5.005_5x model, then it would
be far less easy, but at least consistent. If perl had the abstract model
juerd dreams of, then perl would have a very easy unicode model that boils
down to what I talked about on the perl workshop: encode/decode when doing
I/O, oherwise, enjoy.
As it is, perl has this abstract model that nobody understands because
they are constantly reminded that it isn't fully implemented and they do
have to care about the UTF-X flag themselves, as perl seemingly randomly
doesn't.
Nicholas Clark
> On Fri, Mar 30, 2007 at 01:07:22PM +0100, Nicholas Clark <ni...@ccl4.org> wrote:
> > > And the problem is that those bugs are not considered bugs but features.
> >
> > I certainly consider this one a bug.
>
> So fix it. It is easy to do, and I documented it years ago (during 5.6).
"this one" that I was confident is a bug is the change of meaning on SvPV()
And in turn what I'm not confident about is the fix.
> Besides, without any doubt, the code that relies on psuedo-random
> behaviour is certainkly in the minority. The amount of code in the wild
> that relies on "C" having 5.5 semantics is much larger. I doubt _anybody_
> except me (or at leats not very many people) understands that he has to
> downgrade scalars before passing them into unpack to decode structures.
I don't know enough about "C" in pack offhand to know what the right thing to
do is.
I don't like anything Perl space that lets the abstraction leak, and "C" is
one of them.
The third thing that you didn't mention which I consider distinct from the two
behaviours you did is that the encoding effects how regexps match, and
lc/uc/lcfirst/ucfirst.
Nicholas Clark
Marc Lehmann
> >
> > So fix it. It is easy to do, and I documented it years ago (during 5.6).
>
> "this one" that I was confident is a bug is the change of meaning on SvPV()
> And in turn what I'm not confident about is the fix.
Sorry. I can understand that it might be difficult as perl itself likely
relies on the current meaning of SvPV.
However, some of the obvious fixes would be to change ExtUtils/typemap so
that stuff such as "const char *" does no longer boil down to random bytes.
Example:
SV *compress (const char *data);
the right thing here is to use SvPVbyte, at leats in the majority of
cases. The reason is that existing users either have to clal downgrade
explicitly themselves or suffer from random problems.
etc.
> > Besides, without any doubt, the code that relies on psuedo-random
> > behaviour is certainkly in the minority. The amount of code in the wild
> > that relies on "C" having 5.5 semantics is much larger. I doubt _anybody_
> > except me (or at leats not very many people) understands that he has to
> > downgrade scalars before passing them into unpack to decode structures.
>
> I don't know enough about "C" in pack offhand to know what the right thing to
> do is.
The right thing to do is the follow the documentation and existing code.
Could you tell me why almost every other 5.6 bug was fixed in 5.8, but
gratitious breakage of large parts of CPAN are accepted with this change?
Whats the rationale behind keeping this 5.6 bug, while fixing the rest?
For example, take a network protocol that sends packets prefixed with a
2-byte length header, a type, and data. There is currently no unpack format
available to do this, as:
unpack "Cn", $data
Gives different results depending in the history of the string in $data.
If there were a pack type that gave me 5.005 behaviour of returning a
single character, I could use it:
unpack "Wn", $data;
but there simply isn't. Besides, all code does use "C", so the right thing is
to move the new pack type to a different modifier.
(In my personal opinion, of course, pack should not expose internal
encoding at all. Use Devel::Peek or so, or one of the functions in the
utf8:: module. The first one who shows me code that would need the
peculiar nondeterministic behaviour of unpack "C" gets a prize).
> I don't like anything Perl space that lets the abstraction leak, and "C" is
> one of them.
So why not fix it? Nobody made such a fuss when they fixed the remaining bugs
from 5.6. For example, PApp, one of my older modules using unicode, is full
of code such as this:
Convert::Scalar::utf8_on($_); # DEVEL7952 bug workaround #d# #FIXME#
For various values of DEVEL and workaround. Some of that code broke in 5.8
because 5.8 did the right thing (not 5.8.0, mind you, as this fixing went
on during 5.8.x).
*Nobody* argued my case of "it breaks existing code", not even me, because
its clearly a bugfix that lets perl code just work, both old code and new
code (which is the beauty of the perl unicode model).
> The third thing that you didn't mention which I consider distinct from the two
> behaviours you did is that the encoding effects how regexps match, and
> lc/uc/lcfirst/ucfirst.
The difference is that I haven't seen code break so badly because of
that. I see lots of code break because of the incompatible change in the
meaning of "C", though.
(In fact, I haven't even seen a difference, apart from when use locale is
active, which is a rare case).
The other difference to that case is that those bugs are getting fixed,
while in the case of "C", people just ignore the problem, which increases
over time, saying they don't know why to fix this bug.
And as I said, there is no pack-type that gives me the old meaning of
"C" that every structure-decoding program relies on. Thats gratitious
undocumented breakage. (It really is undocumented because all of the perl
documentation tells me that the internal encoding doesn't surface, and the
small hint in the pack description for "C" seems to reinforce this as it
tells me it works "even in the presence of Unicode"!).
In any case, please could you answer to me why you accept obvious breakage
of old code in this case? I really wanna know.
The only argument in favour I have heard os far is that the camelbook
documents it in some obscure way. But that cannot be a reason to keep a
bug. If the camelbook describes buggy behaviour, it needs a fix. It is
insane to force every existing perl program that uses that feature to
be changed in a way that contradicts the rest of the documentation, is
unintuitive and generaly useless (again, show me a useful application for
unpack "C" with 5.8 semantics).
Jarkko Hietaniemi
> gratitious breakage of large parts of CPAN are accepted with this change?
> Whats the rationale behind keeping this 5.6 bug, while fixing the rest?
>
> from 5.6.
Oh, for heavens sake. I'm sorry but I have VERY hard time of
listening to your wailing and sitting still. So I won't.
Perl 5.8 was in development for quite close to two years (5.7.0 in
2000-Sep, but work started already in July or so - 5.8.0 in 2002-Jul),
and 5.8.1 (the "cleanup for oopses" for 5.8.0) took another year. So
three years before we had really a useable 5.8.
Since then Nicholas picked up and has admirably and thanklessly
released SEVEN maintenance releases of 5.8 over three years, meaning
that about every six months there has been a change of fixing
something that is very broken.
How serious a breakage can be if in three years of development and
three years of maintenance it hasn't gotten enough attention to be
fixed? There is no hidden conspiracy of keeping things broken.
I'm the first one to admit that I wasn't brave enough to REALLY fix
the Unicode brokenness of 5.6.
(1) The more strongly typed scheme, where there would be really
forcibly separate "byte strings" and "Unicode strings" *would* have
been possible, if I only had had the guts. But it was mostly the
regex engine that scared me too much. For basic strings manipulation
and I/O it would not have been a problem to implement.
(2) Another big mistake (due to lack of courage) was the decision to
stick with "Latin-1" as the default 8-bit legacy "type". I should
have broken that assumption, too, and stuck with pure ASCII (or
EBCDIC). (As a side thing, the 8-bit locale support should have been
ejected, too: it is just not worth the trouble: it should have been
replaced with something pluggable so that people could have plugged in
CLDR or Windows locales or whatever they want.)
I'm just getting really, really tired of people whining about Perl's Unicode.
--
There is this special biologist word we use for 'stable'. It is
'dead'. -- Jack Cohen
Nicholas Clark
> On Fri, Mar 30, 2007 at 01:31:22PM +0100, Nicholas Clark <ni...@ccl4.org> wrote:
> However, some of the obvious fixes would be to change ExtUtils/typemap so
> that stuff such as "const char *" does no longer boil down to random bytes.
> Example:
>
> SV *compress (const char *data);
>
> the right thing here is to use SvPVbyte, at leats in the majority of
> cases. The reason is that existing users either have to clal downgrade
> explicitly themselves or suffer from random problems.
This seems a sane idea. However, I'm not going to change it for 5.8.9
5.10 is a different matter, but also not my call.
> Could you tell me why almost every other 5.6 bug was fixed in 5.8, but
> gratitious breakage of large parts of CPAN are accepted with this change?
> Whats the rationale behind keeping this 5.6 bug, while fixing the rest?
No, I can't.
5.8.0 and 5.8.1 were not my releases, *and* I wasn't aware that 'C' was a
problem at that time.
I *think* that the reason may have been because "it is documented in
Programming Perl" that it behaves the 5.6.0 way.
*but*
I went looking, and the closest I can find to an assertion about how it works
is:
* the pack/unpack letters "c" and "C" do /not/ change, since they're often
used for byte-orientated formats. (Again, think "char" in the C language.)
However, there is a new "U" specifier that will convert between UTF-8
characters an integers:
pack("U*", 1, 20 ,300, 4000) eq v1.20.300.4000
* The chr and ord functions work on characters
chr(1).chr(20).chr(300).chr(4000) eq v1.20.3000.4000
In other words, chr and ord are like pack("U") and unpack("U"), not like
pack("C") and unpack("C"). In fact, the latter two are how you now emulate
byte-orientated chr and ord if you're too lazy to use bytes.
[3rd edition, page 408]
> > I don't like anything Perl space that lets the abstraction leak, and "C" is
> > one of them.
>
> So why not fix it? Nobody made such a fuss when they fixed the remaining bugs
> from 5.6. For example, PApp, one of my older modules using unicode, is full
I'm not going to change anything this late in 5.8.x.
Whether 5.10 changes is not something I have the final say on.
> And as I said, there is no pack-type that gives me the old meaning of
> "C" that every structure-decoding program relies on. Thats gratitious
> undocumented breakage. (It really is undocumented because all of the perl
> documentation tells me that the internal encoding doesn't surface, and the
> small hint in the pack description for "C" seems to reinforce this as it
> tells me it works "even in the presence of Unicode"!).
>
> In any case, please could you answer to me why you accept obvious breakage
> of old code in this case? I really wanna know.
>
> The only argument in favour I have heard os far is that the camelbook
> documents it in some obscure way. But that cannot be a reason to keep a
> bug. If the camelbook describes buggy behaviour, it needs a fix. It is
> insane to force every existing perl program that uses that feature to
> be changed in a way that contradicts the rest of the documentation, is
> unintuitive and generaly useless (again, show me a useful application for
> unpack "C" with 5.8 semantics).
I agree with the obscure now.
Reading the wording of the Camel book carefully, this behaviour
$ perl5.00503 -le 'print unpack "c", chr (256+78)'
78
$ perl5.00503 -le 'print unpack "C", chr (256+78)'
78
"unchanged" actually means to me that it would produce the same output.
The only thing that seems to define the current 5.6 behaviour is the
comparison of unpack("C") with ord under use bytes in the paragraph on chr
and ord.
Nicholas Clark
Nicholas Clark
> Since then Nicholas picked up and has admirably and thanklessly
> released SEVEN maintenance releases of 5.8 over three years, meaning
> that about every six months there has been a change of fixing
> something that is very broken.
But to be fair to Marc, I think that he reported issues with pack after
5.8.3 was released and at the time I wasn't sure how to fix it and punted.
(He may have reported it earlier, that's the earliest I remember saying
roughly "I don't know how to fix this safely")
So I haven't dealt with it for about 2 years. It's one of those icky things
where it's always easier to volunteer to deal with something else first.
The only point I partially dealt with it was during my TPF grant.
5.8.9 will be the first stable release after that, due to the small delay
of a job that tried to drive me insane.
Nicholas Clark
Juerd Waalboer
> Well, perl goes to some lengths (implicit conversion) for you to be
> able to mix untagged-all-ascii string values and tagged-non-ascii
> transparently in your program.
As Jarkko already mentioned, Perl internally makes a distinction between
latin1 and utf8. BOTH are fully ASCII-compatible, but no special case
exists for strings that are fully ASCII-only.
> Well, I think is_utf8 is poorly named either way (with several years
> of hindsight - I don't think I would have made a better choice at the
> time).
Agreed, but for different reasons. I think it should be called
Internals::internal_encoding_is_utf8_not_latin1, with a user friendly
wrapper called Internals::encoding that returns either "latin1" or
"utf8".
> I don't think that Perl's internal representation for unicode
> strings is guaranteed to be utf8.
Indeed. It can also be latin1. The flag indicates (negated) if this is
the case.
> The flag more properly means "please treat this as character data
As far as Perl is concerned, ALL strings consist of character data.
Internally-latin1 strings are special because there, bytes and
characters can safely be considered equal.
> And it's the 'special care' bit which can cost performance.
My guess is that the performance costs are mostly associated with utf8
being variable width, which means that you need to scan through the
string to do just about anything.
> [The UTF8 flag is] really a bit of perl's internals which application
> code shouldn't really want to examine or change directly.
Well said.
> >Now, if there is some concern that character-oriented regexes and
> >such are considerably slower for ASCII data than alternatives, and
> >this is a problem and it can't be otherwise dealt with
> I think the unicode regex engine can never be as fast as the
> byte-oriented one.
Unicode versus bytes is a weird comparison. Unicode strings are stored
as bytes too!
But it's true that a naive octet matcher is faster. When you're
particularly sure about the encoding and you're matching literal strings
(no case insensitive stuff) and don't care about character offsets and
don't care that the captures might not be correct character strings, you
can sometimes gain some performance by encoding (e.g. utf8::encode for
max performance) both the subject and the regex before matching and
decoding afterwards. But be careful that this way also have an adverse
effect, so always benchmark first.
> It has more to consider. There's some example code (vaguely like the
> sort of templating where I noticed the problem), which shows unicode
> running 2-3 times as slow (17s instead of 6s) as the byte engine.
I'd like to see and examine that. Templating is a trade that sometimes
allows for naive handling, so there might be room for improvement.
> I'd rather is_utf8 disappeared from the public API, since it's really
> an internal flag and (I think) poorly named.
There's nothing wrong with an internal thing being part of a public API.
Perl has that everywhere. There is, however, something wrong with people
who access these internals not realising that they are, in fact,
internals, even though the documentation clearly indicated this.
Encode::is_utf8 is very clearly labeled as "[INTERNAL]" in the
documentation.
The function may certainly be useful sometimes, like when you can output
either latin1 or utf8 and just want to get the data out, without the
performance loss of re-encoding:
binmode $fh, ":raw";
print {$fh}
"Content-Type: text/plain; charset=",
(Encode::is_utf8($body) ? "UTF-8" : "ISO-8859-1"),
"\n\n", $body;
> Internally, it could then be renamed requires_unicode_engine or
> something.
Unicode semantics are also needed, when the string is not encoded as
utf-8 internally. Don't forget that Unicode and UTF-8 are different
things.
Regardless of the internal encoding, "x" and "Γ©" are unicode characters,
with lots of unicode properties, like that they are lower case
alphabetic characters.
> But what I really care about is the ability to just tell perl "data
> from this source is in this encoding"
binmode $source, ":encoding(...)";
Though when your source is different, you may need to write your own
wrappers.
> "data going to this destination is in this encoding"
binmode $destination, ":encoding(...)";
Same caveat.
> and get all the nice automagic handling of conversions for me without
> paying the unicode engine cost on ascii data.
The conversions themselves may need "the unicode engine" to realise that
no further action is required for your ASCII data.
> while ($data =~ s/<%-(\d+)([^<]*?).*%-\1>/reverse($2)/e) {
If you reverse, you need to know where characters end. If the internal
encoding is utf-8, knowing where individual characters end, requires
scanning through the string.
That's one of the reasons that Perl DOES NOT USE utf-8, when latin1
suffices. Here, you forced the issue with _utf8_on.
Perl already has this optimized. It's not in the regex engine, but in
the very implementation of strings themselves, so that other operations
may benefit too.
Juerd Waalboer
> > The *conceptual* purpose of the UTF8 flag isn't there. Conceptually,
> > every string can be a unicode string, and you're not supposed to look
> > at, know, or set the UTF8 flag yourself. It's an internal bit, like IOK
> > and NOK. [1]
> Thats not how current perl works.
We must have differing definitions, somewhere.
> > Perl conceptually has a single numeric type, and a single string type.
> > The distinction between integer and float, and between iso-8859-1 and
> > utf-8, is internal.
> I would love if that were the case, but the powers to be decided that every
> perl progarmmer has to know those internals, and needs to be able to deal
> with them.
The best approach to programming with unicode in mind, in Perl, is to
(pretend to) be completely ignorant about Perl's internals with regards
to encoding and the UTF8 flag.
The only exception is the regex engine, which has a big bug. This can be
worked around, again without any knowledge of the internals, by
utf8::upgrade'ing both sides of the regex before trying the match.
Your powers-that-be, might be different. Also, don't confuse "you can
know what Perl does internally" with "you have to know what Perl does
internally".
Just being able to access internal metadata doesn't mean you should
actually do so on a daily basis.
It's entirely possible to make undef writable, and have it equal 42.
No-one is complaining about that, and only very few people ever get the
idea of changing the value of undef.
It's also entirely possible to set the internal flag "UTF8" on an
existing string. But for some reason a lot of people are complaining
about that, and even more people have actually set UTF8 flags
themselves...
> > Note that Perl internally uses iso-8859-1 (8 bit) and utf-8 (variable
> > whole-octet), not ascii (7 bit).
> No, Perl exposes this. For example, see the recent example of Compress::Zlib:
> unpack ('CCCCVCC', $$string);
> that code is broken because the powers to be decided that "C" exposes the
> internal encoding, while "V" doesn't.
Yes, any byte-specific operation on a text string (which I keep separate
from character strings) will use the internal encoding. It has to use
/some/ encoding, because it cannot see whether the string was meant as a
byte string or a text string. Perl does not have strong typing.
Personally, I think that unpack with a byte-specific signature should
die, or at least warn, when its operand has the UTF8 flag set. That'll
catch at least some of the cases, because the UTF8 flag always
positively indicates that the string is a text string. (The reverse,
however, is not true: a string without the UTF8 string might be either a
text string or a byte string.)
> That requires every perl programmer who decodes file headers etc.
> using unpack to know about those internals.
No, it requires every Perl programmer to keep track of the function of
every string.
Byte strings and text strings must never be combined, and text strings
must never undergo byte-specific operations.
This again requires no knowledge of the actual encoding that Perl uses
internally, whatsoever.
> The same is true for many XS modules: in older versions of perl, SvPV gave
> you the 8-bit version of a scalar, but in current versions, it randomly
> gives you either 8-bit or utf-8 encoded. SvPV was renamed to SvPVbyte.
Unfortunately, I lack knowledge of these internals, so I cannot comment
about this (yet).
Note that XS writers must have knowledge of Perl's internals. This has
always been true, and is not specific to this fancy new Unicode thing.
> And the problem is that those bugs are not considered bugs but features.
Some bugs are acknowledged as bugs, but won't be fixed anyway, because
there is already a lot of code in the wild that depends on the bugs.
> > [1] Some parts of Perl break this concept. The regex engine is one of
> > them, and has different semantics depending on the presence of the flag.
> > This is a bug, but any fix would be incompatible.
> In fact, some parts of perl break this concept and make perfectly working
> code (in 5.005) not working anymore, or working randomly, and thats not
> considered a bug.
Personally I'm only interested in 5.8.2 and later, but I still would
like to learn about this history.
> unpack "C", $s;
The C template for unpack is specifically documented as byte-specific.
It should never be used on text strings. If you properly keep text and
byte strings separate, that means that your byte string was never
upgraded, and that unpacking with "C" is reliable and predictable.
If upgrading happened even though the string was not mixed with text
strings or used with unicode semantics, that is a bug. I'm very
interested in these silent upgrades that you are experiencing.
> If you think it is obvious, how about this:
>
> my $s = chr 255; # to me, this is one octet. to perl, it might be one or
> # two, or maybe more, who knows.
> warn unpack "C", $s;
> "$s\x{672c}";
> warn unpack "C", $s;
> $s .= "\x{672c}"; substr $s, 1, 1, "";
> warn unpack "C", $s;
> Can a pure-Perl programmer tell what the output of this program is without
> trying it?
Not relevant.
> Should he be able to?
No, because the author of this program made a big mistake in the line
"$s\x{672c}".
The casual reader can easily figure out that $s was meant as a byte
string: it is used with unpack "C", which is known to be a byte
operation. Because it is a byte string, the chr 255 is just a 0xFF
octet, not a Ρ (ÿ) conceptually.
The casual reader can also easily figure out that \x{672c} is meant as a
text string: any codepoint higher than \x{FF} is always a character,
never a single byte.
Then, the author of this snippet uses both the byte string $s and the
text sting "\x{672c}" joined in one string "$s\x{672c}". People not
interested in fixing the code can stop reading there: the code is broken
and its semantics not terribly relevant. People who wish to fix it, will
have to try and figure out what the author really wanted to do here.
Because it's a contrived case, that's very hard to figure out. But I'm
sure that given real world values and variable names, there would be a
clear and logical solution, to be found somewhere along the lines of
encoding and decoding explicitly.
> Thats a broken unicode model
So far, I've only seen a broken understanding of the unicode model, and
a broken regex engine.
Juerd Waalboer
> In fact, I teach a lot of people about unicode in perl.
At the German Perl Workshop, I saw your unicode presentation. I don't
know if this is a good representation for your teaching of unicode, but
I noticed that you used utf8::encode and utf8::decode, not the similar
functions from Encode.pm that are more commonly used and advised. These
utf8:: in-place encode/decode functions are efficient, but using them
means that the same SV changes from byte string to text sting or vice
versa, which makes the code hard to follow, and any attempt to use
hungarian notation in code examples impossible.
Whenever I teach the Perl Unicode model, I try to call my strings
$byte_string and $text_string, or similar. But
utf8::decode($byte_string) makes $byte_string a text string, and
utf8::encode($text_string) makes $text_string a byte string, so after
these statements, the names are no longer correct.
(And of course, I try not to teach people the Unicode model, because
that's something that's quite internal. I try to teach the difference
between text strings and byte strings, and how to use encodings (which
are byte representations of text strings). I treat UTF-8 exactly the
same way as KOI8-R. That helps a lot!)
> If perl had the abstract model juerd dreams of
and uses in day-to-day coding, without encountering ANY of the problems
that you describe (only the regex engine still manages to surprise me,
but that's because I'm too stubborn to utf8::upgrade explicitly).
It kind of makes one wonder if this dream might be reality (and your
reality a dream?)
> then perl would have a very easy unicode model that boils down to
> what I talked about on the perl workshop: encode/decode when doing
> I/O, oherwise, enjoy.
And keep text strings and byte strings separate!!!!!!!!!!!!!eleven
Whenever you must mix text strings and byte strings, consider the byte
strings I/O and encode/decode accordingly.
So, recap: encode/decode when doing I/O, keep text strings and byte
strings separate, otherwise, enjoy.
Tels
Hash: SHA1
Moin,
On Friday 30 March 2007 20:09:29 Juerd Waalboer wrote:
> Marc Lehmann skribis 2007-03-30 14:24 (+0200):
> > In fact, I teach a lot of people about unicode in perl.
>
> At the German Perl Workshop, I saw your unicode presentation. I don't
> know if this is a good representation for your teaching of unicode, but
> I noticed that you used utf8::encode and utf8::decode, not the similar
> functions from Encode.pm that are more commonly used and advised. These
> utf8:: in-place encode/decode functions are efficient, but using them
> means that the same SV changes from byte string to text sting or vice
> versa, which makes the code hard to follow, and any attempt to use
> hungarian notation in code examples impossible.
However, if you have 200Mbyte of ASCII string, it is more efficient to *not*
copy the data around just to find out that, yes, all of it is 7bit :)
But otherwise, I basically agree with you.
All the best,
Tels
- --
Signed on Fri Mar 30 22:31:28 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
"Retsina?" - "Ja, Papa?" - "Schach Matt." - "Is gut, Papa."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg2QGXcLPEOTuEwVAQIy+Qf8DETRmN30yEFJSgd2yO8kezpOiT6yErsB
c2EUa0XJ1nl+pEQ1givBZ4Y/Ci7QlfeyuDFCBL30Ld1JKPBqP2p6AJgwoOAKk2VU
AQcnTUloimSqzanuzs8+v5S7APUDQbBuEpaxliepHuMAJvfxFjN81A8nWcXDWNUO
XG/YSLiDvoZoj8RE5rpE5DQ7hIuoyxq/h6fBlIwNB7ATl3XOPC8Ji8rKCIglzW88
DNHiovC0Mo5V6VNE2tYfKlkxZBm1qtOjenUurgUjdh4NoivyxAg9CCvbFoWDE6f4
zeOio94e9JEf5e4ZlK+plwqFSonpVbRO7Fdk1EcxrjG5sQaaBNzNxQ==
=a8Qq
-----END PGP SIGNATURE-----
Tels
Hash: SHA1
Moin,
On Friday 30 March 2007 21:00:37 Marvin Humphrey wrote:
> On Mar 30, 2007, at 12:53 PM, Juerd Waalboer wrote:
> > Perl does not have strong typing.
>
> If it is so deadly to collide byte-oriented data with character data,
> it should not be so easy to do so accidentally.
It can happen everytime you concatenate two strings. Maybe we could add a
new warning?
use warnings 'upgrade';
my $a = 'a';
$a .= "\x100"; # warns
In an application I am currently bringing up to speed in regard to Unicode I
opted for a "string" struct, that contains essentially:
* the lenght in bytes
* the lenght in characters (not always set, e.g. can be unknown)
* the storage buffer (containing the data, plus some optional padding)
* the encoding
Every action between two stings thus becomes very clearly defined as you can
compare their encodings before doing anything. (for instance upgrading one
or both strings before comparing them etc.)
In Perl, you have only one bit to tell you the encoding (utf8), and it seems
this is not enough as strings without that bit set can be either ASCII, or
ISO-8859-1, or the local locale (maybe?), or utf-8 which hasn't yet tagged
as UTF-8 etc. In short, it becomes a mess.
All the best,
Tels
- --
Signed on Fri Mar 30 23:11:40 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
"Call me Justin, Justin Case."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg2ag3cLPEOTuEwVAQKxjwf/Tu2blhDuAawXoTbNOCA9wBnWtvxvwL05
PoIZOI9vSivXF78ooL8/Hta8pC4o2/TgFdYzORyzNGCGNSdkkj/4vnriZ+f67uV2
BQGhzceu7r5U2Byl1xBS/egDB8FOSzB9kX3BcviD+ePjB/gAys0XagCQxfzLiFEa
mCAp3LVVANmXei0/AgoI/Mj2gO+iz4XX3QvqoL/4tr7Dg734pG/SkYvNE5DL2sc0
OfTvQPGc8NmLHseEM8Vt0jY/gApHLK0LFn9yh98BbJaGNIaCzNZxtPABGYWjFoFS
JI1qEVVO4xu0FOJktdEaOSdONTGBincL+4jZ4HbXpi7EMCCZJNLLyw==
=t2+L
-----END PGP SIGNATURE-----
Marvin Humphrey
On Mar 30, 2007, at 12:53 PM, Juerd Waalboer wrote:
If it is so deadly to collide byte-oriented data with character data,
it should not be so easy to do so accidentally.
>> Thats a broken unicode model
>
> So far, I've only seen a broken understanding of the unicode model,
> and
> a broken regex engine.
That so many users, including those as expert as Marc, possess a
"broken" understanding of Perl's Unicode model suggests a flawed
design. We have been set up to fail.
(My admiration for the Unicode integration effort remains
undiminished by its flaws.)
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
Juerd Waalboer
> >Perl does not have strong typing.
> If it is so deadly to collide byte-oriented data with character data,
> it should not be so easy to do so accidentally.
I agree. But Perl chose to have the same single data type for all
strings, and to maintain compatibility with older Perls by assuming that
your byte string is a latin1 string if you start using it as a text
string. After all, in a strictly 8 bit world, there's no need for a
distinction, so people were never careful about it.
(Well, there was a need, but ignorance being bliss ignoring that was
better for anyone's sanity.)
It kind of bothers me that people constantly whine about this decision
years after it was made. The time to influence the decision has past. It
just seems so counter-productive to keep bringing it up, while there are
bugs to be discovered and fixed.
I wasn't active in p5p back then, and if I had been, I would probably
not have overseen the consequences, just like the porters then didn't.
But wonderfully, a rather consistent and usable plus useful model was
invented, with better/easier Unicode/encodings support than any other
programming language. Of course it's never good enough, but let's first
focus on finding and fixing bugs.
> That so many users, including those as expert as Marc, possess a
> "broken" understanding of Perl's Unicode model suggests a flawed
> design.
I think the design is solid, but the implementation (see regex) slightly
broken and documentation wildly misleading.
The documentation thing I'm trying to fix with perlunitut, perlunifaq,
and a lot of changes to existing documentation, all of which are now
part of bleadperl and will probably be part of the next Perl release.
In addition, I'm maintaining a consise list of best practices at
http://juerd.nl/perluniadvice, and spending tuits on teaching people
(including module maintainers) about the One Way To Do It, because there
is, in fact, just one way that really works well in this case. You just
have to find it, and stick to it. TIMTOWTDI doesn't always apply.
> We have been set up to fail.
Maybe so, but you haven't given up yet, and I hope you won't. Please
join us in the effort to deal with the problems at hand. It's a hell of
a lot more productive than praying for the opportunity to undo recent
years of Perl.
Surely you must know a way in which Perl's unicode support can be
improved, or accidents avoided, without trying to change all of Perl,
CPAN, and a gazillion lines of code that we can't even reach. Let's hear
it! :)
Thanks,
Juerd Waalboer
> However, if you have 200Mbyte of ASCII string, it is more efficient to *not*
> copy the data around just to find out that, yes, all of it is 7bit :)
Indeed, but this is an optimization. Optimization isn't part of teaching
how things work, it always comes after.
Information overload is probably the single most problematic thing in
Perl's unicode documentation. Constantly people are told all those
internal implementation details that they don't have to know. It's no
wonder that they start assuming that they actually need this
information, and use manual setting of UTF8 flags as their first resort
in case of trouble.
Juerd Waalboer
> > If it is so deadly to collide byte-oriented data with character data,
> > it should not be so easy to do so accidentally.
> It can happen everytime you concatenate two strings. Maybe we could add a
> new warning?
Eh, no, because Perl does not have any metadata telling you if this
non-UTF8 string is a latin1 text string, or just a random byte string.
There is no way to tell Perl how you intended your string to be used,
and there is no way for Perl to tell you the same thing about a string
it returned.
> use warnings 'upgrade';
This already exists on CPAN, authored by Audrey Tang, as
encoding::warnings:
use encoding::warnings;
But it will warn when Perl upgrades latin1 to utf-8, without knowing if
that is a bug or a feature, because it doesn't know if the "latin1"
string was meant as a text string or a byte string.
It's a useful debugging tool, to find unintended upgrades, but you
shouldn't try to avoid upgrading altogether. That just hurts, because
upgrading is part of the way the Perl Unicode model was intended.
> * the lenght in bytes
> * the lenght in characters (not always set, e.g. can be unknown)
> * the storage buffer (containing the data, plus some optional padding)
> * the encoding
Hey, cool, Perl has almost the same thing, only it supports just two
encodings: latin1 and utf8. It uses a single bit to indicate the
encoding, the UTF8 flag, which can be on or off. When it's off, the
string is latin1, when it's on, the string is UTF-8.
Maybe you should try Perl; you'll like the way it's built, because it
very closely matches your own design!
The same type of string can be used for binary data, because in the
unicode encoding "latin1", all 256 codepoints map to the same byte
values.
> In short, it becomes a mess.
Yes, with strong typing, especially with string subtypes for arbitrary
encodings, it would be cleaner. But it would also not look like Perl 5.
Tels
Hash: SHA1
Moin,
On Friday 30 March 2007 21:28:44 Juerd Waalboer wrote:
> Tels skribis 2007-03-30 22:32 (+0000):
> > However, if you have 200Mbyte of ASCII string, it is more efficient to
> > *not* copy the data around just to find out that, yes, all of it is
> > 7bit :)
>
> Indeed, but this is an optimization. Optimization isn't part of teaching
> how things work, it always comes after.
I almost agree. :)
Some decisions really need to be done early on, in the design phase. You
cannot optimize when the design is broken. E.g. if your data needs to be
copied around *per design*, the best you can achive is O(N). When you do
not have to copy the data, you suddenly can achive O(1). This distinctions
is quite important, and not something you can fix aftwards apart from
redesigning (aka let's break and re-assemble it :)
A recent (non-Perl) example for such a methodology/design change was
zero-copy networking - I remember there being a lot of talk about this,
especially in Unix/Linux world. Basically, when you want to send data to
the network it is wastefull to copy it many times around just to output it
to the hardware - up to the point where the copy takes more time than all
the rest of work to be done. However, avoidn the copy isn't that easy :)
I know it is hard to design your code so that it works fine for small data
("A") and large data ("A" x 10000000) alike, but usually, these things need
to be considered early on, or you end up with a system that is only usefull
for demos and toying around and breaks under real-world access :)
Just like security, a performant design usually can't just bolted on later.
And how to design your program to be secure, ast, reliable etc. should be
teached, too. Maybe not in the same hour, but close :-)
Just saying... :)
> Information overload is probably the single most problematic thing in
> Perl's unicode documentation. Constantly people are told all those
> internal implementation details that they don't have to know. It's no
> wonder that they start assuming that they actually need this
> information, and use manual setting of UTF8 flags as their first resort
> in case of trouble.
I think I agree. Luckily I managed to completely avoid this whole issue by
ignoring unicode until very recently - and then the doc and code had
improved quit a lot so that Unicode is really usable in Perl (Thank you
guys! especially Jarkko!)
All the best,
Tels
- --
Signed on Fri Mar 30 23:55:12 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
"Elliot, Sie Schwachkopf!"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg2lXXcLPEOTuEwVAQKsEQf/REU2lTQdaOjP7MBeC+Uw6zdQaSB26FgY
cZn9ob0M6Jz2l2+2hukhZQpFbff09QxzVPIPmL3RtUx3SIEdF/3WjFQ7CvLxfQR8
S0KG3zkhMclrdEAspOlUrW2g+PlC9PuWGSPhUGg+LSvGVkNmQtor7dMoEVQ0BD1b
4kVRU4s7Jb4A7kyoFYksBumofNg/Qw1Y2Jr2ccn9WU3G6EHNOM4dYWDieq+BW1Ci
YcGAx+gSS523OvBh73VxYsCDz3RgY1aRWqULmvCCp38F6fluDcDAc14PQnoDz8j0
PAgkS4wiChq/uSY28wp9IZuoYU8k8+gB3eJtraRGTem+DiW7vgT/yA==
=3Z3P
-----END PGP SIGNATURE-----
Juerd Waalboer
> > Indeed, but this is an optimization. Optimization isn't part of teaching
> > how things work, it always comes after.
> I almost agree. :)
> Some decisions really need to be done early on, in the design phase. You
> cannot optimize when the design is broken. E.g. if your data needs to
> be copied around *per design*, the best you can achive is O(N). When
> you do not have to copy the data, you suddenly can achive O(1). This
> distinctions is quite important, and not something you can fix
> aftwards apart from redesigning (aka let's break and re-assemble it :)
With Data::Alias you have a fair chance.
utf8::decode and utf8::encode provide similar opportunities to optimize
after the fact.
I don't see how decoding and encoding in-place or copying would be part
of your design, or greatly influence it.
Tels
Hash: SHA1
Moin,
First for the record:
The application I am outfitting is written in C, for speed, and quite large.
So there is NO way I would even consider to rewrite it in Perl. I'm just
using the right tool for the right job. That doesn't mean I do not like
Perl, or the way Perl does things. Sorry if this sounded like it.
Anyway, I wasn't aware that any non-utf8 data in Perl is *always*
ISO-8859-1, I thought that, when not specified, this depended on some other
stuff. Guess I need to reread the tutorials. :)
However, this also poses the question: How does Perl know that your data is
in KOI8-R?
(Yes, that's a trick question, but I would like to hear your answer to that,
in any case, just to make it clear to me. No offence meant!)
One of the limitations of the "there can be only two encodings" of Perl
seems to be that strings are permanently upgraded:
$iso_8859_1 = '...';
$utf8 = '...';
if ($iso_8859_1 eq $utf8) { ... }
Please correct me if I am wrong, but I do think it is not be possible to
keep both variables in their current encoding and only temporarily upgrade
them to utf8 (for the common encoding that contains both of them)?
After reading this discussion here, a lot of problems also seem to stem from
the fact that the upgrade to utf8 is permanent, silently and
done "behind-the-scenes". Just like 1 + 2.0 will result in 3.0 and not 3
and we all know how much confusion this creates :) (heh, I fell for it
today, even tho I should have know better :)
> The same type of string can be used for binary data, because in the
> unicode encoding "latin1", all 256 codepoints map to the same byte
> values.
This sounds like a circular definition, because in CP1250, also all 256
codepoints map to the same byte values. Except it are different byte
values :)
In my application, I also considered having a "BINARY" encoding, but in the
end I opted to make ISO-8859-1 the default encoding for BINARY stuff. (Ha,
great minds sink alike or so) And since unlike in Perl, upgradings are
never done permanently, you can keep your BINARY string and compare it to
UTF-8 whatever, and it never gets "corrupted".
I am not sure how one could achive that in Perl. Making the SV read-only?
> > In short, it becomes a mess.
>
> Yes, with strong typing, especially with string subtypes for arbitrary
> encodings, it would be cleaner. But it would also not look like Perl 5.
Over the years, I come to the insight that I want to build reliable and fast
programs. (easy to maintain, reliable, fast, pick two :-)
So maybe we really need "use strict 'encodings';" :-)
All the best,
Tels
- --
Signed on Sat Mar 31 00:04:29 2007 with key 0x93B84C15.
Get one of my photo posters: http://bloodgate.com/posters
PGP key on http://bloodgate.com/tels.asc or per email.
"Blogebrity: Wow, guess what this one stands for? Too easy. Hey, anyone
can do it: take a blogger who's a chef, and you get: BLEF. A blogger
who's a dentist? BENTIST. A female blogger with an itch? You guessed it:
a BITCH."
-- maddox from xmission
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg2pBHcLPEOTuEwVAQKxJQf/UKYZhHUkTkH6wpP/uLQ+zkEO/8ptDA4i
7lQipjOIkGlcLc0peF0sr2jlNu59XWSVbDeYdSSdJGWYvydYbeToP180xaBms40a
GdL/5QWlgUalQ1sifs93r1pfx+AQv1Pc4TivybFj/SbYY5WYe7pcaZDZ80/luYtp
ftxd+96KLVshZ/2bMtxjJ7yo2k7oD0uwA2MF1SFiytjSFZZ+QRol2G7PbsIaqonc
ITDrEm+R+djp9FLFKlXQIs3/jNx2wOhoS5z6Q3HKIi9KrXfMngyZa4cvpSmm071l
ETbRT4gy+1O7fFvsFG8xrtyajO95LpSPhZ1aeYR7fPpj0zLP6KNqxQ==
=jV6Z
-----END PGP SIGNATURE-----
Marc Lehmann
> Information overload is probably the single most problematic thing in
> Perl's unicode documentation. Constantly people are told all those
> internal implementation details that they don't have to know.
Exactly. If they wouldn't have to care for those internals it would be much
simpler, abstracted away. But thats not reality.
> wonder that they start assuming that they actually need this
> information, and use manual setting of UTF8 flags as their first resort
> in case of trouble.
If you send a compressed string over the network using JSON and decompress
it, you need to know that. Evem if you do pure perl only. And, as I do a
lot of network protocols, this never hurts me as I know how and when perl
upgrades/downgrades and whats broken or not.
It does hurt other people constantly though, and I do not understand why
it has to be that way if the fix were conceptually simple and aligned with
existing usage.
In my talk for example I only hinted at the implementation details and
told people to ignore it, but when they get weird bugs, they might look
into that.
My problem is not that there are bugs. My problem is that those bugs are
not beign fixed because of truely hilarious reasons such as that obscure
rfereence in the camelbook, so all have to suffer, while other, similar
bugs, have official bug status and get fixed.
I am really frustrated at that. It makes perl as a whole rather questionable
for unicode use, as you constantly have to think about the internals.
And yes, that simply shouldn't be the case.
Marc Lehmann
> Marc Lehmann skribis 2007-03-30 14:24 (+0200):
> > In fact, I teach a lot of people about unicode in perl.
>
> At the German Perl Workshop, I saw your unicode presentation. I don't
> know if this is a good representation for your teaching of unicode, but
It is, if a bit short (and I consider it a matter of taste).
> > If perl had the abstract model juerd dreams of
>
> and uses in day-to-day coding, without encountering ANY of the problems
> that you describe
Frankly, that is not a very good sign. It means eitehr you are extremely
lucky or you don't use any of the many XS modules that silently break, or
even the Perl modules (such as the example from Compress::Zlib) that break
less silently, but more miraciously.
> It kind of makes one wonder if this dream might be reality (and your
> reality a dream?)
The dream isn't reality. If it ere, people would not report bugs against
JSON::XS because it happens to create scalar values with the UTF-X bit set.
And they do so for some of my other modules doing that, too. And there are
two options to me: either tlel them perl is broken w.r.t. to e.g. "C", or
their code is broken becasue they do not call downgrade.
Obviously, I prefer the former over the latter, but last time I was told
unpack "C" was mentioned to break the abstraction in the camelbook, so its
correct.
Which suddenly invalidates a lot of code.
> > then perl would have a very easy unicode model that boils down to
> > what I talked about on the perl workshop: encode/decode when doing
> > I/O, oherwise, enjoy.
>
> And keep text strings and byte strings separate!!!!!!!!!!!!!eleven
I find "text strings" and "byte strings" not adequate either, as Perl
makes no difference between those two concepts (being typeless), and
they do not map well to encoded/decoded text either. Perl only knows
how toc oncatenate characters, it does not know anything about byte or
text, so utf8::encode does not necesarily create a byte string out of a
text string. It could juts as well create a text string out of a byte
string (think JSON, which creates json _text_ out of e.g. byte strings by
encoding them to UTF-8).
> So, recap: encode/decode when doing I/O, keep text strings and byte
> strings separate, otherwise, enjoy.
I do not think that maps clearly to Perl (or my programs either). It might
be a good and simplified advice to a beginner, though, although I prefer
to never tell people simplified (but wrong) things. The perl unicode model
is rather simple, but leaves you in control, and I found teaching people
about how perl just allows more than 0..255 for a character index works
best (although people differ).
Juerd Waalboer
> Anyway, I wasn't aware that any non-utf8 data in Perl is *always*
> ISO-8859-1, I thought that, when not specified, this depended on some other
> stuff. Guess I need to reread the tutorials. :)
Note that they are unicode strings, and that Perl is theoretically free
to change the internal representation at any time.
> However, this also poses the question: How does Perl know that your data is
> in KOI8-R?
Because you tell it that it is with "decode". The resulting string is a
unicode string, which may have any encoding internally. (Practically,
this is limited to latin1 and utf8.)
my $text_string = decode("koi8-r", $byte_string);
or, if you prefer different terminology:
my $unicode_string = decode("koi8-r", $koi8r_string);
> One of the limitations of the "there can be only two encodings" of Perl
> seems to be that strings are permanently upgraded:
> $iso_8859_1 = '...';
> $utf8 = '...';
> if ($iso_8859_1 eq $utf8) { ... }
$iso_8859_1 is temporarily upgraded to utf8 for this comparison.
(Yes, this copies data, and then throws it away. Again, optimization
does require knowing internals. The easiest optimization here is to
utf8::upgrade $iso_8859_1, after which the variable name no longer makes
sense :))
> Just like 1 + 2.0 will result in 3.0 and not 3 and we all know how
> much confusion this creates :) (heh, I fell for it today, even tho I
> should have know better :)
Doesn't really cause me any headaches, to be honest.
> > The same type of string can be used for binary data, because in the
> > unicode encoding "latin1", all 256 codepoints map to the same byte
> > values.
> This sounds like a circular definition, because in CP1250, also all 256
> codepoints map to the same byte values. Except it are different byte
> values :)
I said "unicode encoding", but should have said "unicode codepoints".
Codepoints 0..256 in latin1 map to byte values 0..256. That makes it
special.
> > > In short, it becomes a mess.
> > Yes, with strong typing, especially with string subtypes for arbitrary
> > encodings, it would be cleaner. But it would also not look like Perl 5.
> Over the years, I come to the insight that I want to build reliable and fast
> programs. (easy to maintain, reliable, fast, pick two :-)
I do that with Perl. Really, you should check that language out! You'll
LOVE it! :)
Marc Lehmann
> This seems a sane idea. However, I'm not going to change it for 5.8.9
Sure.
> 5.10 is a different matter, but also not my call.
Sure.
I know all that...
> > Could you tell me why almost every other 5.6 bug was fixed in 5.8, but
> > gratitious breakage of large parts of CPAN are accepted with this change?
> > Whats the rationale behind keeping this 5.6 bug, while fixing the rest?
>
> No, I can't.
> 5.8.0 and 5.8.1 were not my releases, *and* I wasn't aware that 'C' was a
> problem at that time.
Yes, you can. You control 5.8, and you said it won't gonna happen. So either
you have a reason and can tell me of it, or not.
The reason I wanna know is because I want to know what to tell
people. Either it is "your code is broken, unpack "C" without downgrade
is a bug in your code" or "it is a bug in perl, you can work around by
enabling ->shrink for the time being".
> I *think* that the reason may have been because "it is documented in
> Programming Perl" that it behaves the 5.6.0 way.
I would argue it doesn't behave the 5.6 way, though: 5.6 had a completely
broken unicode implementation, and lots of bugs. In 5.6 it would give me one
"character", because 5.6 often exposed the utf-8 encoding explicitly, so one
character in the 5.6 model often was a single internal byte.
Also, I still think it is a mistake to break working code without giving
an alternative(!) for unpack that isn't "you have to downgrade and keep
your fingers crossed".
> I went looking, and the closest I can find to an assertion about how it works
> is:
>
> * the pack/unpack letters "c" and "C" do /not/ change, since they're often
> used for byte-orientated formats. (Again, think "char" in the C language.)
> However, there is a new "U" specifier that will convert between UTF-8
> characters an integers:
>
> pack("U*", 1, 20 ,300, 4000) eq v1.20.300.4000
Exactly. But "C" somehow works on UTF-8, while it shouldn't. It should
work on characters, as documented (just like in C, char array[]; array[i]
is one character, regardless of how many bits a character in C has, or how
it is encoded).
> * The chr and ord functions work on characters
>
> chr(1).chr(20).chr(300).chr(4000) eq v1.20.3000.4000
>
> In other words, chr and ord are like pack("U") and unpack("U"), not like
> pack("C") and unpack("C"). In fact, the latter two are how you now emulate
> byte-orientated chr and ord if you're too lazy to use bytes.
So due to that documentation insanity it is now suggested that all code that
used "C" beforee muts use "U" now to get the same effect as in earlier perl
versions?
Then why was "use feature" introduced in the first place? Just document
existing programs to be broken. I am quite convinced (whatever that means
to you :) that that would result in less and less silent breakage then
renimong "C" to "U".
Besides, perl 5.8 does not follow that description:
perl -e '$x = "\xc3\xbc"; die unpack "U*", $x'
This gives me 195188, two characters, although it is a single UTF-8
character, so why does it wrongly give me two? $x certainly is utf-8-encoded
(try Encode::encode_utf8 chr 252, it results in the above string).
Whoever wrote that part, simply said, was completely confused about unicode.
Thats fine, Sarathy had to hammer it into me too, and then made a mistake
himself after he did so. And it took me years to understand how it should be.
It is hard to do from an implementors standpoint because you are so near the
actual code.
But that doesn't mean it is right. Fact is, the above documentation is
simply wrong, either with regards to how it should be, and in regards to how
it is implemented.
> [3rd edition, page 408]
(Thanks for digging it out, btw, I haven't seen that yet).
> > So why not fix it? Nobody made such a fuss when they fixed the remaining bugs
> > from 5.6. For example, PApp, one of my older modules using unicode, is full
>
> I'm not going to change anything this late in 5.8.x.
> Whether 5.10 changes is not something I have the final say on.
Ok, so I will tell people to replace "C" by "U" in theor code then.
Thanks! (And go on with your good work, btw., it seems that wasn't quite
clear to some people, so again: you are doing tremendously good work! :).
> "unchanged" actually means to me that it would produce the same output.
>
> The only thing that seems to define the current 5.6 behaviour is the
> comparison of unpack("C") with ord under use bytes in the paragraph on chr
> and ord.
Right, while the documentation on unpack "U" disagrees with it, as it talks
about UTF-8. The documentation clearly does not apply to current perls, it
clearly applies to the 5.005_5x model where perl ahd no UTF-X flag.
Juerd Waalboer
> > > If perl had the abstract model juerd dreams of
> > and uses in day-to-day coding, without encountering ANY of the problems
> > that you describe
> Frankly, that is not a very good sign. It means eitehr you are extremely
> lucky or you don't use any of the many XS modules that silently break, or
> even the Perl modules (such as the example from Compress::Zlib) that break
> less silently, but more miraciously.
Most of the time, it's a question of realising that the module doesn't
do the Perl unicode model, and considering communication with the module
I/O, i.e. only feed it bytes, and only get bytes back. Encode and decode
as appropriate.
I maintain a short list of some modules at
http://juerd.nl/perluniadvice. If you encounter modules that I can test
easily without setting up complete environments, please let me know!
Compress::Zlib sounds like it uses zlib, which compresses byte streams.
i.e. don't give it unicode strings, because unicode strings have no
bytes (the bytes are internal only, but you don't know what encoding is
used there). Encode explicitly.
> And they do so for some of my other modules doing that, too. And there are
> two options to me: either tlel them perl is broken w.r.t. to e.g. "C", or
> their code is broken becasue they do not call downgrade.
Their code is probably broken because they mix text strings with byte
strings. This can be solved most easily by explicitly encoding your text
string as soon as you feel you must join it with a byte string. The
joined string as a byte string. Decoding it to make a text string may or
may not make sense, depending on the data format.
> I find "text strings" and "byte strings" not adequate either, as Perl
> makes no difference between those two concepts (being typeless)
Indeed. Programmers have to track this themselves. Sometimes that sucks,
but in my experience, you need to know what kind of data your variable
contains anyway.
If you ++ a reference, you're in for trouble too. How come that's never
been a problem? Probably because programmers are pretty good at knowing
what functions their variables have.
It's just that this is something you haven't needed to know before, so
you're not /trained/ yet to think about it. But you can't go from 256
characters to several thousands without changing the way you think :)
> they do not map well to encoded/decoded text either
Oh, but they do. Please read perlunitut, which tries to redefine the
universe into four important definitions (and succeeds).
1. Byte strings (aka binary strings)
2. Text strings (aka unicode strings or "internal format" strings)
3. Decoding is byte --> text
4. Encoding is text --> byte
> Perl only knows how toc oncatenate characters, it does not know
> anything about byte or text, so utf8::encode does not necesarily
> create a byte string out of a text string.
I don't get the causal connection you're illustrating.
utf8::encode takes any text string (or unicode string, if you prefer
that term) and turns it into a UTF-8 encoded byte string in place.
That is,
utf8::encode($foo);
is the efficient equivalent of:
$foo = encode("utf8", $foo);
Note that whenever a string has an encoding attach to it, conceptually,
it's automatically a byte string. Text strings don't have encodings,
because encodings are a byte thing, and text strings don't have bytes;
they have characters. (Text strings have encodings and bytes
/internally/, just like numbers do have bytes /internally/, encoded in
one way or another, that allows values greater than 255 or less than 0.)
> It could juts as well create a text string out of a byte string (think
> JSON, which creates json _text_ out of e.g. byte strings by encoding
> them to UTF-8).
utf8::encode is a text operation. It will assume that whatever you give
it, is a text string. Its characters are considered Unicode codepoints.
You shouldn't give it a byte string.
To understand what happens if you do give utf8::encode a byte string,
you need to know some internals. But I stress that this is not required
knowledge, because it's so much easier to just remember not to do this
weird thing. Why would you try to encode a byte string to UTF-8, anyway?
That makes no sense, because UTF-8 is a means of representing
characters. Byte strings consist of bytes, not characters.
Here's what happens internally: Any byte string used as a text
string is considered to be encoded in latin1, because Perl doesn't
know the difference.
> (or my programs either). It might be a good and simplified advice to a
> beginner
The theory is very simple, but not simplified. It just isn't any harder.
I'm sorry if you want a more complex programming tool. But apparently
you have found ways to make it hard for yourself already :)
> though, although I prefer to never tell people simplified (but wrong)
> things.
I agree. Whenever I use a simplified view, that will be obvious or
mentioned. Metadata ("this information is wrong, but useful anyway") is
very important.
> The perl unicode model is rather simple, but leaves you in control,
> and I found teaching people about how perl just allows more than
> 0..255 for a character index works best (although people differ).
That's a great explanation of how unicode strings work. But when people
write programs, these programs typically accept input and also have some
output. And then you're doing I/O, which is done with bytes, and
requires character encodings in order to communicate characters. You
used to be able to ignore this fact when everyone still used iso-8859-1,
I mean CP437, I mean CP850, I mean koi8-r, I mean Windows-1252. Right,
we never did all use exactly the same encoding. We've just chosen to
remain ignorant all this time. Explicit re-encoding, or decoding and
encoding has been necessary all this time. It's just that with more than
256 codepoints, it became much more apparent :)
Marc Lehmann
> > > at, know, or set the UTF8 flag yourself. It's an internal bit, like IOK
> > > and NOK. [1]
> > Thats not how current perl works.
>
> We must have differing definitions, somewhere.
No. I have explained elsewhere that we quite agree on how it should be. It is
just that you make strange claims:
> The best approach to programming with unicode in mind, in Perl, is to
> (pretend to) be completely ignorant about Perl's internals with regards
> to encoding and the UTF8 flag.
It doesn't work even when not having unicode in mind. See unpack.
> The only exception is the regex engine, which has a big bug.
Uhm, no.
> Your powers-that-be, might be different. Also, don't confuse "you can
> know what Perl does internally" with "you have to know what Perl does
> internally".
In the example I gave, you have to.
> Just being able to access internal metadata doesn't mean you should
> actually do so on a daily basis.
Whats the alternative? Replace all my uses of unpack with explicit calls
to ord? Sorry, but thats completely unrealistic.
> It's also entirely possible to set the internal flag "UTF8" on an
> existing string. But for some reason a lot of people are complaining
> about that, and even more people have actually set UTF8 flags
> themselves...
Yes. Because you have to when interfacing with a gazillion of existing
modules (or at the very least clear or downgrade).
If perl wouldn't force people to know the internals so often, one could
certainly get away with telling them: do not touch downgrade/upgrade, and
certainly never utf8_on or is_utf8, it is form the dveil.
But thats far from reality.
> > unpack ('CCCCVCC', $$string);
> > that code is broken because the powers to be decided that "C" exposes the
> > internal encoding, while "V" doesn't.
>
> Yes, any byte-specific operation on a text string (which I keep separate
> from character strings) will use the internal encoding. It has to use
> /some/ encoding, because it cannot see whether the string was meant as a
> byte string or a text string. Perl does not have strong typing.
Thats wrong. There is a perfectly good definition for character and byte:
the one from C. It is a single element of a string. The same thing was true
in perl: one byte is one character, and it should be true under the new
model.
Nothing in pack or unpack requires a speciifc encoding, just as nothign in
perl should require me to know the specific encoidng of "chr 200". It is a
single byte/character, regardles sof how perl stores it internally.
> Personally, I think that unpack with a byte-specific signature should
> die, or at least warn, when its operand has the UTF8 flag set.
Thats pure insanity. Then people would again be forced to know the internal
encoding. How can you tell people to not worry about internal encoding and in
the next paragraph force them to know because suddenly they are not allowed
to call unpack unless some _internal_ flag has some specific value.
I severely doubt you understood perls unicode model: It works by abstracting
away the internal flag completely, not forcing the user to deal with it.
Forcing her to deal with it is *wrong*.
> catch at least some of the cases, because the UTF8 flag always
> positively indicates that the string is a text string.
No, absolutely not. You are confused. The UTF-X flag only marks a specific
encoding used by perl internally. It says nothing about text or not text. You
cna store binary just fine in a UTF-X marked string.
> (The reverse,
> however, is not true: a string without the UTF8 string might be either a
> text string or a byte string.)
As might a string with the UTF-X flag set. Perl is typeless, it doesn't know
anything about text vs. binary.
> > That requires every perl programmer who decodes file headers etc.
> > using unpack to know about those internals.
>
> No, it requires every Perl programmer to keep track of the function of
> every string.
No. A binary string is a binary string because it contains no characters
higher then 255. It is that simple.
> Byte strings and text strings must never be combined, and text strings
> must never undergo byte-specific operations.
That is certainly wrong.
> This again requires no knowledge of the actual encoding that Perl uses
> internally, whatsoever.
It does, for unpack, both in current perl as well as in your proposed change.
> Note that XS writers must have knowledge of Perl's internals. This has
> always been true, and is not specific to this fancy new Unicode thing.
Right. But why gratitiously break old code? In perl, it is broken by at least
unpack, in XS, it is broken by changing the meaning of SvPV.
> > And the problem is that those bugs are not considered bugs but features.
>
> Some bugs are acknowledged as bugs, but won't be fixed anyway, because
> there is already a lot of code in the wild that depends on the bugs.
Again, I know a lot of code that is currently broken because of that
bug. I asked, but nobody found code "in the wild" that relies on that
specific bug.
> > unpack "C", $s;
>
> The C template for unpack is specifically documented as byte-specific.
No, it is specifically documented as being character-specific. Read your
manpage carefully:
c A signed char value.
C An unsigned C char (octet) even under Unicode.
(Note that byte and character is the same thing in C). That leavs us with
"octet". An octet is a number between 0 and 255 (you can give alternative
definitions thta are equivalent to mine, though).
In perl this is an octet:
$x = chr 200;
Yet unpack under some circumstances returns two values for this single
octet, and sometimes not. And the only way to know is to inspect the
internal UTF-X flag.
> It should never be used on text strings.
Perl is typeless. There is no such thing as a text string in Perl. The
problem, however, is not that it doesn't work on "text strings",m whatever
that might be, the problem is that unpack doesn't work on binary strings,
ro at least not all the time.
> If you properly keep text and byte strings separate, that means that
> your byte string was never upgraded, and that unpacking with "C" is
> reliable and predictable.
Uhhh, who guarentees that? JSON::XS does no such thing, and cannot
guarantee that, because Perl has no type for "text string" vs. "binary
string". So how do you suggest JSON::XS keeps text and byte strings
separate, if there is no way to detect the type of a string or make a
useful difference between those two?
> If upgrading happened even though the string was not mixed with text
> strings or used with unicode semantics, that is a bug. I'm very
> interested in these silent upgrades that you are experiencing.
Concatenating strings might upgrade them (e.g. in debugging output). More
so, JSON::XS currently can return either UTF-X encoded strings or non
UTF-X-encoded strings.
You can that buggy. So please tell me how to fix that bug. How do I, when
decoding a JSON string, know wther it is one of your text or byte strings?
Whats the difference, if neither JSON nor Perl make one?
> > If you think it is obvious, how about this:
> >
> > my $s = chr 255; # to me, this is one octet. to perl, it might be one or
> > # two, or maybe more, who knows.
> > warn unpack "C", $s;
> > "$s\x{672c}";
> > warn unpack "C", $s;
> > $s .= "\x{672c}"; substr $s, 1, 1, "";
> > warn unpack "C", $s;
> > Can a pure-Perl programmer tell what the output of this program is without
> > trying it?
>
> Not relevant.
Very relevant.
> > Should he be able to?
>
> No, because the author of this program made a big mistake in the line
> "$s\x{672c}".
Are you sure that upgraded? And why is it a mistake? I very much differ in
that-
> The casual reader can easily figure out that $s was meant as a byte
> string
I cannot, from that short fragment. Neither can Perl.
> it is used with unpack "C", which is known to be a byte
> operation. Because it is a byte string, the chr 255 is just a 0xFF
> octet, not a Ρ (ÿ) conceptually.
Exactly. But unpac does not return 255 for that byte string.
> The casual reader can also easily figure out that \x{672c} is meant as a
> text string: any codepoint higher than \x{FF} is always a character,
> never a single byte.
Why? Lots of people use those higher codepoints. Perl certainly does
not mandate anything like that, so why do you try to enforce it? People
routinely do stuff like join "\x{100}", @png_images to seperate them, and
it works fine.
Perls unicode model does not enforce a meaning of the codepoints used in
strings. It simply allows me to use more character indices than in 5.005.
> Then, the author of this snippet uses both the byte string $s and the
> text sting "\x{672c}" joined in one string "$s\x{672c}". People not
> interested in fixing the code can stop reading there: the code is broken
> and its semantics not terribly relevant.
Thanks for gratitiously calling my code broken. In any case, explain to me
how to fix it in general, I only gave an example of silent upgrades.
use JSON::XS;
my $x = (from_json to_json [$y])[0];
is another silent upgrade users need to know about.
> People who wish to fix it, will
> have to try and figure out what the author really wanted to do here.
Exactly that.
> Because it's a contrived case, that's very hard to figure out.
Not at all. You are just guessing, and getting it wrong.
> sure that given real world values and variable names, there would be a
> clear and logical solution, to be found somewhere along the lines of
> encoding and decoding explicitly.
See above, figure it out in the real world then.
> > Thats a broken unicode model
>
> So far, I've only seen a broken understanding of the unicode model, and
> a broken regex engine.
Same here. Your model requires people knowing about the UTF-X flag (at
leats in unpack). Mine doesn't, and I think mine is much closer to what
you want to achieve: not having to tell people about it. In your model
you would have to tell people to downgrade before unpacking string, or
alternatively, you rule out a lot of perfectly fine Perl code on the
assumption that it is easy to figure out that it is broken. Sorry, but I
differ very much.
Marc Lehmann
> >>Thats a broken unicode model
> >
> >So far, I've only seen a broken understanding of the unicode model,
> >and
> >a broken regex engine.
>
> That so many users, including those as expert as Marc, possess a
> "broken" understanding of Perl's Unicode model suggests a flawed
> design. We have been set up to fail.
My "broken" understand bails down to users not having to know about the
internal UTF-X flag. If that is indeed wrong, then going back to the
explicit 5.005_5x model where you indeed had to track your encoding
manually is the right thing.
However, I claim it would a great loss. Back to the assembly programming
of unicode.
Juerd Waalboer
> If you send a compressed string over the network using JSON and decompress
> it, you need to know that.
Does JSON compress arbitrary data? If so, then the user must do the
decoding and encoding, because arbitrary data only exists in byte form.
Once you dictate any specific encoding, it's no longer arbitrary.
On the other hand, if JSON does text data only, it can just use any UTF
encoding on both sides, and document it like that.
Unless both sides are exactly the same platform (e.g. both Perl), you
need to establish a protocol for sending data anyway. And that protocol
should also describe encoding. If sender and receiver don't agree, you
have a problem.
> I am really frustrated at that. It makes perl as a whole rather
> questionable for unicode use, as you constantly have to think about
> the internals. And yes, that simply shouldn't be the case.
I maintain that it isn't the case, for almost any programming job,
unless you're indeed doing things with internals.
Marvin Humphrey
On Mar 30, 2007, at 2:25 PM, Juerd Waalboer wrote:
>> That so many users, including those as expert as Marc, possess a
>> "broken" understanding of Perl's Unicode model suggests a flawed
>> design.
>
> I think the design is solid, but the implementation (see regex)
> slightly
> broken and documentation wildly misleading.
I strongly disagree with this assessment. In particular, I think
insisting that the user be responsible for manually segregating
character and byte-oriented data without any help from Perl is
totally unreasonable.
Look at how easily Marc made the "mistake" of commingling the two
types of data. It's debatable whether the fact that Perl allowed him
to do that without complaint is a flaw with the design or the
implementation, but it's one or the other and it's serious.
Additionally, as Marc points out, there are lots of broken XS modules
out there -- including one of mine. (KinoSearch 0.15 -- Unicode
support is fixed as of 0.20_01, which breaks backwards
compatibility.) Few or none of them would be broken if Perl made it
more difficult to move between character data and byte-oriented data
-- errors would be flying right and left and the broken modules would
get fixed right away.
Of course I understand why that cannot be the case, but it's
astonishing to me that you see this as a problem which can be solved
via documentation.
I hope that Perl 6 does not opt to replicate Perl 5's behavior in
this area (my understanding is that it will not, but I'm not
following development closely). I hope that project is taking into
account the lessons we have learned in the wake of very difficult
compromises about how to balance the addition of Unicode with
preserving backwards compatibility.
> Surely you must know a way in which Perl's unicode support can be
> improved, or accidents avoided, without trying to change all of Perl,
> CPAN, and a gazillion lines of code that we can't even reach. Let's
> hear
> it! :)
How about encouraging the use of encoding::warnings in perlunitut?
How about adding it to core and having 'use 5.10;' turn it on?
Marc Lehmann
> I maintain a short list of some modules at
> http://juerd.nl/perluniadvice. If you encounter modules that I can test
> easily without setting up complete environments, please let me know!
>
> Compress::Zlib sounds like it uses zlib, which compresses byte streams.
> i.e. don't give it unicode strings, because unicode strings have no
> bytes (the bytes are internal only, but you don't know what encoding is
> used there). Encode explicitly.
The difference between us, and thats what it boils down to, is that you give
the internal UTF-X bit meaning. You equate UTF-X flag set == Unicode string.
To me, a unicode string is a concept outside of perl. I would consider any
text string using the unicode codepoints a unicode string. For example:
"hallo" is a unicode string. Any any binary string is not a unicode string.
The problem with your approach is that you have to expose the UTF-X flag
to users. Which comes with a lot of problems.
Please note that in the actual problem, nobody is passing unicode to
compress::zlib. Instead, a binary string is passed to Compress::Zlib that
happens to be UTF-X encoded internally because it was transferred using a
protocol that encodes bytes as UTF-8 (namely JSON), and the decoder opted
not to make another copy of the data for speed reasons.
Compress::Zlib is not buggy. Neither is the caller. The bug is that unpack
treats the same string differently depending on an internal flag that might
be set for a variety of reasons outside the programmers control.
Initially I thought you, too, wanted a unicode model where the UTF-X bit is
not exposed to the perl level. But in fact the opposite is true: you force
knowledge of the UTF-X bit on users, even though it should be transparent.
Thats the problem. As logn as you call UTF-X-encoded strings Unicode
strings and something else byte strings and try to give them meaning
the programmer has to know about it, as functions behave semantically
differently depending on that flag.
All I want is a perl that behaves semnatically consistent, regardless
of some internal flag that is documented not to be of concern to a Perl
programmer.
> Their code is probably broken because they mix text strings with byte
> strings. This can be solved most easily by explicitly encoding your text
> string as soon as you feel you must join it with a byte string. The
> joined string as a byte string. Decoding it to make a text string may or
> may not make sense, depending on the data format.
my $bytestring = "zlib-encoded string";
my $transfer = Encode::encode_utf8 $bytestring;
my $bytes = Encode::decode_uf8 $transfer;
$bytes is the same string, but depending on implementation details of
Perl, it is treated diferently in different contexts, sometimes it is
treated like the binary string it is, sometimes it is trated as if it were
utf-8 encoded, which it isn't, as I decoded it.
> > I find "text strings" and "byte strings" not adequate either, as Perl
> > makes no difference between those two concepts (being typeless)
>
> Indeed. Programmers have to track this themselves. Sometimes that sucks,
> but in my experience, you need to know what kind of data your variable
> contains anyway.
the problem is you want them to track the UTF-X flag in addition to that.
Because putting a "byte string" into unpack should not work if that bit
happens to be set. So you force people who want to use unpack to learn about
that flag, when it is set, when they have to downgrade etc. etc.
> If you ++ a reference, you're in for trouble too. How come that's never
> been a problem?
Because perl treats it consistently.
> It's just that this is something you haven't needed to know before, so
> you're not /trained/ yet to think about it. But you can't go from 256
> characters to several thousands without changing the way you think :)
Yes. Thats not a problem, I understand unicode quite well, and I udnerstand
quite well how Perl stores unicode.
What the problem is is that I separate internal encoding (unicode can be
encoded both in UTF-X as well as in octets, as can byte strings) from the
unicode model in Perl, while you mix them together, forcing the user to
know their UTF-X bits on their scalars in addition to tracking wether they
are binary or not.
> > they do not map well to encoded/decoded text either
>
> Oh, but they do. Please read perlunitut, which tries to redefine the
> universe into four important definitions (and succeeds).
I do not have that manpage.
> 1. Byte strings (aka binary strings)
>
> 2. Text strings (aka unicode strings or "internal format" strings)
>
> 3. Decoding is byte --> text
>
> 4. Encoding is text --> byte
That doesn't reflect reality, of course, if it were so.
However, those four definitions, as I said, do not map well to
encoded/decoded text. Because "internal format" strings can store binary
data just as well, and often does.
I am talking purely about the perl level strings. If perlunitut confused
the issue by talking about internal encoding it completely failed its
mission, imho.
> I don't get the causal connection you're illustrating.
>
> utf8::encode takes any text string (or unicode string, if you prefer
> that term) and turns it into a UTF-8 encoded byte string in place.
No. It converts characters to UTF-X encoded octets. Wether my characters
are bytes or not is of no consequence.
> Note that whenever a string has an encoding attach to it, conceptually,
> it's automatically a byte string.
Yes. And that encoding is completely independent of the internal UTF-X
flag. Or should be, but isn't, in current perls.
> Text strings don't have encodings,
> because encodings are a byte thing, and text strings don't have bytes;
> they have characters. (Text strings have encodings and bytes
Perl doesn't know about that. It only knows about characters. The problem
is that some parts of perl make a difference bewteen the very same string,
depending on how it is encoded internally, _even if the encoding is the
same on the Perl level_.
> /internally/, just like numbers do have bytes /internally/, encoded in
> one way or another, that allows values greater than 255 or less than 0.)
Exatcly. But nothing in perl forces those indices to be unicode characters.
Certainly not the indices 0..255. Yet still, the UTF-X flag might be set or
cleared, resulting in changes in interpretation.
I want those to go away and make perl treat my binary data as binary data,
regardless of how the interpreter treats them.
> utf8::encode is a text operation. It will assume that whatever you give
> it, is a text string. Its characters are considered Unicode codepoints.
Where does it say so?
> You shouldn't give it a byte string.
Please leave it up to me what I should or should not to. This whole
discussion of what I should or should not to is completey besides the
point.
The point is that Perl treats my strings the same in utf8::encode, regardless
of how the UTF-X flag is set, because upgrading or downgrading does not
change the semantics of my characters.
But in unpack, it does. Thats the problem. Bot what I should or should not
do. The problem is givign unpack a binary strings makes it return garbage
sometimes (if the binary string happens to be encoded internally in UTF-X).
This whole "force the user to track the UTF-X bit is useless". If you
really want that, then go back to 5.005_5x, which forces you to track
your UTF-8 on your own. The whole point of the big change in 5.6 was that
programmers should not care about how perl internally encodes stuff, and I
certainly do not want to give this up. Thats what makes perl so good.
> To understand what happens if you do give utf8::encode a byte string,
A byte string is a string containing only octets, that is, values between 0
and 255.
Without knowing any intenals, utf8::encode will encode it into a UTF-8
encoded sequence.
> you need to know some internals.
Wrong. I need know no internals, the result is always well-defined: put
characters into utf8::encoede, and get utf-8-encoded characters. No need for
internals knowledge, regardless of wether my characters are 0..255 or some of
them happen to be larger. Perl doesn't care, nor does UTf-8 care, nor do I
care.
The problem is, perl cares in unpack, and when handing strings over to XS
modules.
> That makes no sense, because UTF-8 is a means of representing
> characters. Byte strings consist of bytes, not characters.
Not in C, which is what the documentation constantly refers to, mind
you. And no, a byte always has been a character. It is the very definition
of byte in C, regardless of how many bits it has. And the same is true in
perl: a single bate is represented by a single character, havign an index
no higher than 255.
> > (or my programs either). It might be a good and simplified advice to a
> > beginner
>
> The theory is very simple, but not simplified. It just isn't any harder.
It doesn't map to reality.
> I'm sorry if you want a more complex programming tool. But apparently
> you have found ways to make it hard for yourself already :)
Just stop your ad-hominem, please. I told you before that I find it
rather easy, but users of my module find it rather hard, for example. I
worked around a lot of bugs in 5.6 easily, and can slap an occasional
utf8::up/downgrade into my code. But I think its simply wrong to force
every programmer to know as much about the internals as I do.
> > The perl unicode model is rather simple, but leaves you in control,
> > and I found teaching people about how perl just allows more than
> > 0..255 for a character index works best (although people differ).
>
> That's a great explanation of how unicode strings work.
You think so? Then why do you want to force people to know about how
128..255 is encoded internally then? Because you do when say that UTF-X
always means text (which is not true in reality, mind you), and you want
unpack to fail on binary strings that happen to be UTF-X encoded?
> we never did all use exactly the same encoding. We've just chosen to
> remain ignorant all this time. Explicit re-encoding, or decoding and
> encoding has been necessary all this time. It's just that with more than
> 256 codepoints, it became much more apparent :)
Right. But at leats when dealing with decoded stuff (such as binary data),
Perl should behave consistently and correctly, but it doesn't.
Juerd Waalboer
> The reason I wanna know is because I want to know what to tell
> people. Either it is "your code is broken, unpack "C" without downgrade
> is a bug in your code" or "it is a bug in perl, you can work around by
> enabling ->shrink for the time being".
If a downgrade is "needed", it means that your byte string was
accidentally upgraded. This should only happen if you mix it with a text
string. If it happens without mixing it with a text string, that is a
bug. Please report.
So, neither "your code is broken, unpack "C" without downgrade is a bug
in your code" nor "it is a bug in perl".
Instead: "your code is broken, don't mix text strings with byte strings"
or "it is a bug in perl that your string got upgraded in the first
place."
> Exactly. But "C" somehow works on UTF-8, while it shouldn't.
Agreed!
Things that specifically handle bytes, and bytes only, should DIE (or at
least warn) when used with a string that has the UTF-8 flag on. This
still lets users get away with naively assuming that byte == character
for latin1 strings, as designed, but at least catches the cases when you
know that the user does something stupid.
> It should work on characters, as documented (just like in C, char
> array[]; array[i] is one character, regardless of how many bits a
> character in C has, or how it is encoded).
A C "char" is a byte, not a multibyte character, ever.
Besides that, the "C" in Perl's pack() is documented as a single byte.
I think that "char value" should be either removed from perlfunc, or
explained in more detail. It's NOT OBVIOUS to those who don't know C.
> > * The chr and ord functions work on characters
> > chr(1).chr(20).chr(300).chr(4000) eq v1.20.3000.4000
> > In other words, chr and ord are like pack("U") and unpack("U"), not like
> > pack("C") and unpack("C"). In fact, the latter two are how you now emulate
> > byte-orientated chr and ord if you're too lazy to use bytes.
> So due to that documentation insanity it is now suggested that all code that
> used "C" beforee muts use "U" now to get the same effect as in earlier perl
> versions?
The earlier Perl versions didn't support character values greater than
255, and if you never have those characters, C still works perfectly.
But yes, if you're dealing with characters and want your program to be
able to handle those fancy new >255 characters, you should change that C
to a U.
> Besides, perl 5.8 does not follow that description:
> perl -e '$x = "\xc3\xbc"; die unpack "U*", $x'
> This gives me 195188, two characters, although it is a single UTF-8
> character, so why does it wrongly give me two? $x certainly is utf-8-encoded
> (try Encode::encode_utf8 chr 252, it results in the above string).
You asked for the codepoints U+00C3 and U+00BC, and got them.
It's a UTF-8 encoded byte string, alright, but "U" is for Unicode, not
UTF-8.
> Ok, so I will tell people to replace "C" by "U" in theor code then.
If they do Unicode text strings, that's indeed very good advice.
But you still want C for byte strings, simply because some protocols or
formats expect a byte value. :)
> Right, while the documentation on unpack "U" disagrees with it, as it talks
> about UTF-8.
That would be a bug, but I can't find it in my copy (5.8.8). It only
says "Encodes to UTF-8 internally" for pack(), which as far as I can
tell, is true.
Juerd Waalboer
> one byte is one character, and it should be true under the new model.
You must be kidding.
Juerd Waalboer
> I strongly disagree with this assessment. In particular, I think
> insisting that the user be responsible for manually segregating
> character and byte-oriented data without any help from Perl is
> totally unreasonable.
That is okay. You are not alone.
In fact, I would also like to have real types.
But Perl has had its current model for quite a while. If you don't
agree, there are a few things that you can do:
1. Find a way to do it better, in a backwards compatible way, and then
either
1a. implement it yourself
1b. document it and hope that someone else has the tuits
2. Find a way to do it better, in a non-backwards compatible way, and
then either
2a. fork perl and implement it yourself
2b. document it and hope that someone else has the tuits
3. Just use the tools that Perl currently provides.
Any of these are totally valid options. Gerard Goossen, for example, has
picked 2a. I picked 3. Which one is your favourite?
> I hope that Perl 6 does not opt to replicate Perl 5's behavior in
> this area (my understanding is that it will not, but I'm not
> following development closely).
You are right. Perl 6 will have distinct byte string types ("buf" for
"buffer"), and character string types ("str" for "string").
I guess Perl 6 follows the 2nd path, with both a and b simultaneously :)
> How about encouraging the use of encoding::warnings in perlunitut?
See my other post about why this module is not what you want.
Tels
Hash: SHA1
Moin,
On Friday 30 March 2007 22:38:19 Juerd Waalboer wrote:
> Tels skribis 2007-03-31 0:19 (+0000):
> > Anyway, I wasn't aware that any non-utf8 data in Perl is *always*
> > ISO-8859-1, I thought that, when not specified, this depended on some
> > other stuff. Guess I need to reread the tutorials. :)
>
> Note that they are unicode strings, and that Perl is theoretically free
> to change the internal representation at any time.
>
> > However, this also poses the question: How does Perl know that your
> > data is in KOI8-R?
>
> Because you tell it that it is with "decode". The resulting string is a
> unicode string, which may have any encoding internally. (Practically,
> this is limited to latin1 and utf8.)
>
> my $text_string = decode("koi8-r", $byte_string);
>
> or, if you prefer different terminology:
>
> my $unicode_string = decode("koi8-r", $koi8r_string);
I thought you would say this :)
My question was posed because I wanted to know how to *keep* a KOI8 (or any
other random binary) string in Perl without converting it to Unicode. It
seems to me this is not easily possible because there are literally dozend
places where your KOI8 string might get suddenly upgraded to UTF-8 (and
thus get corrupted because Perl treats it is ISO-8859-1). Or did I get this
wrong?
In an ideal world, you could either just keep everything in utf-8 (that's
too slow for some things and not fool-proof either), or rely on no other
code to corrupt your data - especially this random third party module you
pulled from CPAN last night. :)
OMHO the problem arises from the fact that Perl makes no distinction between
a byte string like "a" and a text string like "a", and furthermore,
manipulating byte string (for instance appending a byte) is done with
typical string operators. So:
$byte_string = 'something random bytes';
# works if $y is 7bit and no utf8 flag
# but fails if $y is 7bit with utf8 flag
$byte_string .= $y;
As you said, all is well as long as you can keep these two beasts seperate,
but the slightest problem might mangle your data. Such as a decode_utf8
setting the UTF8 bit on a 7bit ASCII string, therefore changing the 7bit
byte string to a text string.
Hm, maybe one could write a module that always tackles the encoding to an SV
via magic. And then you could have a special encoding called "BINARY" (or
absence of an encoding means it is treated as binary), so that if you ever
try to fuse two strings together where one of them is tagged binary, you
get an exception (but only then!).
As you said, the current warnings::encode can't decide between the case
of "BINARY + UTF_8" and "ISO-8859-1 + UTF_8" as Perl makes no distinction
between binary data and ISO-8859-1. And this missing distinction is
certainly a bother :)
> > One of the limitations of the "there can be only two encodings" of Perl
> > seems to be that strings are permanently upgraded:
> > $iso_8859_1 = '...';
> > $utf8 = '...';
> > if ($iso_8859_1 eq $utf8) { ... }
>
> $iso_8859_1 is temporarily upgraded to utf8 for this comparison.
> (Yes, this copies data, and then throws it away. Again, optimization
> does require knowing internals. The easiest optimization here is to
> utf8::upgrade $iso_8859_1, after which the variable name no longer makes
> sense :))
Nah, in this case I wanted the temporarily upgrade :)
> > Just like 1 + 2.0 will result in 3.0 and not 3 and we all know how
> > much confusion this creates :) (heh, I fell for it today, even tho I
> > should have know better :)
>
> Doesn't really cause me any headaches, to be honest.
Yeah, I am not a genius :/ (Sometimes I wish I could upgrade my brain :)
> > > The same type of string can be used for binary data, because in the
> > > unicode encoding "latin1", all 256 codepoints map to the same byte
> > > values.
> >
> > This sounds like a circular definition, because in CP1250, also all 256
> > codepoints map to the same byte values. Except it are different byte
> > values :)
>
> I said "unicode encoding", but should have said "unicode codepoints".
>
> Codepoints 0..256 in latin1 map to byte values 0..256. That makes it
> special.
Erm, I don't buy this because:
Codepoints 0..256 in KOI8-R (to pick one) map to byte values 0.256. That
would make it special, too.
(I don't nec. disagree with you, I just don't understand what you mean).
> > > > In short, it becomes a mess.
> > >
> > > Yes, with strong typing, especially with string subtypes for
> > > arbitrary encodings, it would be cleaner. But it would also not look
> > > like Perl 5.
> >
> > Over the years, I come to the insight that I want to build reliable and
> > fast programs. (easy to maintain, reliable, fast, pick two :-)
>
> I do that with Perl. Really, you should check that language out! You'll
> LOVE it! :)
Yeah, maybe one day I actually start real programming work in Perl. ;)
All the best,
Tels
PS: I think this discussion has become a bit off-topic, so we should
probably keep it off-list. Just for the original topic and the record, when
you have pure 7bit ASCII data, Perl (decode etc) should not set the utf8
flag on the data, as that makes things go slower and is just a waste. In
fact, it shouldn't even copy the data around etc., it should only make
exactly one run through the data to count the high-bit bytes.
PPS: Thanx for the discussion, this really helps me to understand things
better.
PΒ³S: Unrelated to this thread, I was working on benchmarking Encode and the
ISO-8859-1 to UTF-8 upgrade code. Stay tuned :)
- --
Signed on Sat Mar 31 01:18:34 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
". . . my work, which I've done for a long time, was not pursued in
order to gain the praise I now enjoy, but chiefly from a craving after
knowledge, which I notice resides in me more than in most other men. And
therewithal, whenever I found out anything remarkable, I have thought it
my duty to put down my discovery on paper, so that all ingenious people
might be informed thereof."
-- Antony van Leeuwenhoek. Letter of June 12, 1716
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg27uncLPEOTuEwVAQIkXAf+O+FgERCl2lcyr28XpeLcCl17pKtfeVBd
kQn/j7sqMGLYuqzcZMrNIn4gKskw8L1T19Q0XcoJBVb4phlHHKrZttmbBrhN++KA
YfXPd9WH/qg9exYHH/+TDdAWCaJYDYcG2B8xI1NTKrDgwFBt8sJJyt9J2jrJoPJE
6rPpAL9vun1wqv6MJeRacxHWmWk7wXflCIrUt9bf8c+feEpMJ51/331Kgb0tjcFs
85IpfzV9TuFn8I17it//7rPrzJfb1NOSwOcgk/6dj5msIoZv1psmNYZcaysAIGpu
evEdhAjpmiVh+DSnGRZEoWfzGwoJfVwGCOmoaQ2O44e9u+AVmx6x0A==
=gDih
-----END PGP SIGNATURE-----
Marc Lehmann
> If a downgrade is "needed", it means that your byte string was
> accidentally upgraded. This should only happen if you mix it with a text
> string. If it happens without mixing it with a text string, that is a
> bug. Please report.
Thats extrenely far from reality. Lots of things can cause a text string
to be upgraded. Forcing people to learn all that is just stupid when you
could just make it work logically without telling people about internals
(note that the internals come into play by your peculiar efinition of
"tetx strings" having the UTF-X bit set, which isn't reality and in my
opinion is an extremely stupid limitation that 96% of perl does not
follow).
> Instead: "your code is broken, don't mix text strings with byte strings"
> or "it is a bug in perl that your string got upgraded in the first
> place."
See my json example. Nothing gets mixed.
> > Exactly. But "C" somehow works on UTF-8, while it shouldn't.
>
> Agreed!
>
> Things that specifically handle bytes, and bytes only, should DIE (or at
> least warn) when used with a string that has the UTF-8 flag on.
So you force people to know about the internal flag, lest they cannot avoid
the die.
This completely contradicts your claim that you want to abstratc the UTF-X
flag away from the Perl level.
> still lets users get away with naively assuming that byte == character
> for latin1 strings, as designed, but at least catches the cases when you
> know that the user does something stupid.
But the user does not do anythign stupid when feeding binary strings (my
definition, indices 0..255) into Compress::Zlib. It is only your request
for a die that makes problems. Zlib would work just fine if perl gave
downgraded data to perl and XS code that wants it.
> > It should work on characters, as documented (just like in C, char
> > array[]; array[i] is one character, regardless of how many bits a
> > character in C has, or how it is encoded).
>
> A C "char" is a byte, not a multibyte character, ever.
Exactly. The same as in Perl I would assume, as Perl uses characters to
store bytes, it doesn't use multibyte characters on the Perl level.
Hope you get it this time :)
> Besides that, the "C" in Perl's pack() is documented as a single byte.
"A C "char" is a byte".
Your words.
But here you say a byte is not a character. Thats a contradiction.
You are deeply confusing the internal encoding Perl uses (Which might be
single octets for characters, or UTF-X encoded octets, for characters)
with the language proper.
In C, a single byte is a character, even if it happens to have a value
higher than 255 (although very few compilers allow that, usually, a byte
is an octet, although it is common on DSPs to have 32 bit bytes).
Even if Perl encoded a single character into multiple C bytes/octets, that
does not mean its more than a single character.
The documentation is completely contradictory when it comes to "C" and can
easily be interpreted to mean a single character in the C sense.
Fact is "even under Unicode" it doesn't work as advertised, becasue Unicode
can be internally represented in multiple ways in Perl.
> I think that "char value" should be either removed from perlfunc, or
> explained in more detail. It's NOT OBVIOUS to those who don't know C.
To those who do know C it has perfectly clear meaning, namely a single
character.
> The earlier Perl versions didn't support character values greater than
> 255, and if you never have those characters, C still works perfectly.
Nothing in C limits you to 256 characters. A byte in C is exactly a
character. It can store at least 256 different values, but nothing in C
limits you to that, many compilers use larger bytes. And the same is true
in Perl: Perl only supported bytes 0..255 in earlier versiosn, and now the
perl byte can be up to 64 bits (or maybe a bit less, I forgot).
> But yes, if you're dealing with characters and want your program to be
> able to handle those fancy new >255 characters, you should change that C
> to a U.
I do not want to handle those fancy >255 characters. I only want to handle
a single octet. But unpack doesn't do that.
In fact, thats thr problem: all old code that uses unpack "C" would need
to be changed to use "U". Thats the compatibility breakage I was talking
about. Code that uses "C" expects the single-octet meaning form perl
5.005, it does not expect the "sometimes returns half of a utf-x encoded
character, sometimes not" meaning it has in current perls.
It is especially weird as it suddenly has become incompatible with regards
to the other template characters such as "n", which correctly decode
bytes regardless of internal encoding.
> > Besides, perl 5.8 does not follow that description:
> > perl -e '$x = "\xc3\xbc"; die unpack "U*", $x'
> > This gives me 195188, two characters, although it is a single UTF-8
> > character, so why does it wrongly give me two? $x certainly is utf-8-encoded
> > (try Encode::encode_utf8 chr 252, it results in the above string).
>
> You asked for the codepoints U+00C3 and U+00BC, and got them.
No, I asked for UTF-8 encoded characters. Again, read the documentation:
* If the pattern begins with a "U", the resulting string will
* be treated as UTF-8-encoded Unicode.
thats for pack, unfortunately.
U A Unicode character number. Encodes to UTF-8
internally
uh, that internal thing again. So how many characters will pack "U", 200
give me? According to the documentation, 2, as UTF-8 requires that. That
is not what happens, though.
Thats the problem. Perfectly working code using unpack "CN" suddenly
stops working because "N" works on bytes, while "C" works on the internal
encoding, regardless of what that might be.
> It's a UTF-8 encoded byte string, alright, but "U" is for Unicode, not
> UTF-8.
You cna store unicode in UTF-8. IF you say "UTF-8 encoded unicode" then you
very well have UTF-8, even though it still is unicode.
> > Ok, so I will tell people to replace "C" by "U" in theor code then.
>
> If they do Unicode text strings, that's indeed very good advice.
Unfortunately, thats what they have to do when dealing with binary
strings, as C doesn't work on them.
> But you still want C for byte strings, simply because some protocols or
> formats expect a byte value. :)
Exactly. And then I have to use "U" to get it. Because a byte in perl is a
character. Is and always has been, just as in C.
And to get those bytes for use in such protocols you have to use "U" now,
instead of "C" as in earlier versions.
> > Right, while the documentation on unpack "U" disagrees with it, as it talks
> > about UTF-8.
>
> That would be a bug, but I can't find it in my copy (5.8.8). It only
> says "Encodes to UTF-8 internally" for pack(), which as far as I can
> tell, is true.
So it talks about using UTF-8, so, according to you, it is a bug. Fine
with me.
Tels
Hash: SHA1
Moin,
I think just documenting isn't enough. We do have things like "strict", so
if the current Perl model doesn't allow you to even detect when you mix the
wrong kind of data, then we need module/pragma that catches these errors.
Of course warnings::encode exists, but it seems to not be able to
distinguish between "untagged" data and real ISO-8859-1 strings as Perl
itself doesn't make this distinction.
> How about encouraging the use of encoding::warnings in perlunitut?
>
> How about adding it to core and having 'use 5.10;' turn it on?
If I understand correctly, that would not be enough due to the "is this
binary or really iso-8859-1 encoded data" problem mentioned above.
all the best,
tels
- --
Signed on Sat Mar 31 01:42:47 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
"In 1988, Jack Thompson ran against Janet Reno for DA of Dade County:
Thompson's unique campaign message was that Reno was unfit for the job
because, as a closeted lesbian with a drinking problem, she was great
candidate for blackmail by the criminal element. Jack never explained
why this remained a threat even after he exposed her 'secret'. Reno
cruised at the polls."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg29jncLPEOTuEwVAQJALAf/SsSjz5VB4l3Zcggd18SNmdTq8DpBLUtP
pxiPCs0fYrEtDny/HvDCbQss/nEaGmFwPaVpAA+kFp8jss3h3xzklW6MwAm7Aisy
+EiZO0JEcADXRWr9CChJpWfMr0qllmzsUUKHa6wc9iXagD6kPoiL49Ay5bkqPBDT
OKOfcJIRDqk12VKATpdQlBIHR3cEpnUMdh8QKhmAArkXAsV5cZGBC9EGm8l+dgeK
Uc2k7pxvLXdjCZu6YbJfPwwdiLlugL23Bci7sZrCO/JyboBOK3ch5dWYohZ8QoMw
SahL/axgJ1DeFTP2ryL6wvnM1djF+HSbzoaLD1E+d7XJqB700Qxdfg==
=eI9w
-----END PGP SIGNATURE-----
Marc Lehmann
> > codepoints map to the same byte values. Except it are different byte
> > values :)
>
> I said "unicode encoding", but should have said "unicode codepoints".
>
> Codepoints 0..256 in latin1 map to byte values 0..256. That makes it
> special.
Yes, and the exact same is true for unicode (both have a 1-1 mapping
between 0..255 and octets), trivially, of course, as unicode explicitly is
a superset of latin1.
Juerd Waalboer
> The difference between us, and thats what it boils down to, is that you give
> the internal UTF-X bit meaning. You equate UTF-X flag set == Unicode string.
No, that's a unidirectional thing.
I've said it on p5p at least a dozen times, but I'll say it again:
If the UTF8 flag is set, you can be sure that you have a text string.
If the UTF8 flag is not set, it can be either a byte string or a text
string.
If you have a text string, the UTF8 flag may or not be set. If you have
a byte string, the UTF8 string is not set (or it was set because you
treated the byte string as a text string).
> The problem with your approach is that you have to expose the UTF-X flag
> to users. Which comes with a lot of problems.
Again: you're kidding, right?
I'm constantly very explicitly and verbosely telling people to NOT look
at the flag, NOT set it manually, etcetera.
Heck, I've even explained that I think you should try to (pretend to) be
ignorant about the internals, in response to your message even!
I do not understand how you are able to misinterpret this message even
after this many posts in this thread alone. Have you ever read
perlunitut, even?
> Initially I thought you, too, wanted a unicode model where the UTF-X bit is
> not exposed to the perl level. But in fact the opposite is true: you
> forc> knowledge of the UTF-X bit on users, even though it should be
> transparent.
> ...
> the problem is you want them to track the UTF-X flag in addition to that.
> ...
> Then why do you want to force people to know about how
> 128..255 is encoded internally then?
That's not what I said, nor what I meant. In fact, quite the opposite.
If you're just spending this evening just to get on my nerves, then
congratulations!
> > Oh, but they do. Please read perlunitut, which tries to redefine the
> > universe into four important definitions (and succeeds).
> I do not have that manpage.
http://www.google.com/search?q=perlunitut&btnI=I'm+Feeling+Lucky
> Because "internal format" strings can store binary data just as well,
> and often does.
Yes, and when you use such a byte string as a text string, its bytes are
considered to be codepoints, just like in latin1.
> I am talking purely about the perl level strings. If perlunitut confused
> the issue by talking about internal encoding it completely failed its
> mission, imho.
I strongly suggest that you READ the document before whining about its
supposed failure.
> The problem is that some parts of perl make a difference bewteen the
> very same string, depending on how it is encoded internally, _even if
> the encoding is the same on the Perl level_.
Those are bugs. Report them, and they might get fixed.
> > utf8::encode is a text operation. It will assume that whatever you give
> > it, is a text string. Its characters are considered Unicode codepoints.
> Where does it say so?
Well, you have already denied that "encoding is going from characters to
bytes" is a real world fact, so I guess there's little point in pointing
out the places where exactly the same thing is explained.
> > you need to know some internals.
> Wrong. I need know no internals
A certain Marc Lehmann once said:
"I would love if that were the case, but the powers to be decided that
every perl progarmmer has to know those internals, and needs to be able
to deal with them."
> > That makes no sense, because UTF-8 is a means of representing
> > characters. Byte strings consist of bytes, not characters.
> Not in C, which is what the documentation constantly refers to, mind
> you.
And that is bad, I agree. Perl programmers should not be expected to
speak C in order to understand Perl documentation. This is a big
problem in Perl's documentation, but who's going to fix it?
Juerd Waalboer
> > How about encouraging the use of encoding::warnings in perlunitut?
> > How about adding it to core and having 'use 5.10;' turn it on?
> If I understand correctly, that would not be enough due to the "is this
> binary or really iso-8859-1 encoded data" problem mentioned above.
You understand correctly :)
Thanks.
Marc Lehmann
> Anyway, I wasn't aware that any non-utf8 data in Perl is *always*
> ISO-8859-1, I thought that, when not specified, this depended on some other
> stuff. Guess I need to reread the tutorials. :)
He, because its not true :)
> However, this also poses the question: How does Perl know that your data is
> in KOI8-R?
It doesn't. Perl ideally only interprets character indices as unicode
codepoints (I am ignoring use locale and similar issues here). So when you
want to match your koi8-r data aginst a regex, you need to decode it first.
Perl doesn't know that and will *then* treat your character data as KOI8-R
(and afterwards as unicode).
Unless you force perl to apply unicode interpretations to your characters,
they are completely encoding-free.
> One of the limitations of the "there can be only two encodings" of Perl
> seems to be that strings are permanently upgraded:
Thats the root of the problem. There aren't two encodings. There is only one:
characters concatenated to form strings.
Internally, Perl currently has two forms for that, just as perl can store
real integers and doubles in a scalar.
But on the Perl level, "5", "5.0", 5 and utf8-encoded 5 are all the same
scalar.
> if ($iso_8859_1 eq $utf8) { ... }
>
> Please correct me if I am wrong, but I do think it is not be possible to
> keep both variables in their current encoding and only temporarily upgrade
> them to utf8 (for the common encoding that contains both of them)?
It is, but likely not very efficient as in most such cases you actually
want utf-x internally. Except for optimisation purposes (where I see
downgrade and upgrade as well-warranted), you do not have to care, as perl
handles thta automatically.
> After reading this discussion here, a lot of problems also seem to stem from
> the fact that the upgrade to utf8 is permanent, silently and
> done "behind-the-scenes". Just like 1 + 2.0 will result in 3.0 and not 3
> and we all know how much confusion this creates :) (heh, I fell for it
> today, even tho I should have know better :)
No, there is no problem in most cases, as the upgrade does not change the
scalar in any way (except, again, for speed). Or at least should.
Perl achieves that goal by transparentlxy re-encoding its internal format
as required. re-coding in that way does not change the semantics of the
string, except:
- when you hit a bug in perl
- when you use unpack "C".
So in a bug-free perl without unpack, everythign just works and you never
need to care about wether perl stores the data as UCS-4, UTF-X or octets
in memory.
Thats the "sane" model introduced with 5.6 and mostly achieves with 5.8.8.
The problem are thre remainign bugs AND unpack, the latter of which breaks
existing programs that assume unpack "C" has byte semantics, when, in
fact, it returns the internal encoding that perl normally hides from you
and tells you to ignore.
If those remaining problems were fixed (that included SvPV), the only
difference between utf-x encoding and octet-encoding within perl would be
speed, but not semantics.
Thats the beauty.
Juerds goal of having the UTF-X flag exposed and having you to think about
when perl upgrades and downgrades (and making you avoid the upgrades) is
horrible, as it forces a lot of administration on the programmer, a lot of
which perl already claims to do, as only in a few cases you have to know
your UTF-X flag at the moment.
> > The same type of string can be used for binary data, because in the
> > unicode encoding "latin1", all 256 codepoints map to the same byte
> > values.
latin1 is not a unicode encoding in the first place.
Also, I find it much more natural to represent bytes as characters 0..255 in
perl, as opposed to Juerds definition of characters 0..255 with the internal
UTF-X flag cleared.
I just don't see why the programmer has to learn about that internal flag
at all. If he has to, then perl could become much much faster by forcing
her to do that all the time, instead of only in unpack or XS cases.
> great minds sink alike or so) And since unlike in Perl, upgradings are
> never done permanently, you can keep your BINARY string and compare it to
> UTF-8 whatever, and it never gets "corrupted".
In the 5.5 model, nothing ever gets "corrupted", too. Thats the beauty of it.
Because scalars with the UTF-X flag set behave the same way as scalars not
having it set, everything is compatible with each other.
Its only the cases _where_ it makes a difference where this is a problem
and in fact stuff gets corrupted.
> I am not sure how one could achive that in Perl. Making the SV read-only?
By fixing the remaining bugs and making the UTF-X flag truely internal, so
you do not have to worry about modules corrupting your stuff.
Thats what perl does for you in the vast majority of cases already, and it
should simply do that all the time, so programmers have their typeless perl
that they love again.
> > > In short, it becomes a mess.
> >
> > Yes, with strong typing, especially with string subtypes for arbitrary
> > encodings, it would be cleaner. But it would also not look like Perl
> > 5.
I beg to differ. Strong typing makes programming hard. Until Perl6 came and
destroyed it, the typeless nature of Perl was a feature, not a problem.
Why should perl suddenly introduce types for strings when a single abstratc
string type works just as wonderful as the single abstract scalar type works
in perl already?
Having strongly typed integers/doubles/utf-8-strings etc. is a step
backwards from perl towards Java.
Programmers using Perl do not want to worry about strict typing. They can
use C++ or Java anytime for that.
> Over the years, I come to the insight that I want to build reliable and
> fast programs. (easy to maintain, reliable, fast, pick two :-)
>
> So maybe we really need "use strict 'encodings';" :-)
What for, so that your program crashes at runtime instead of degrading to a
slower but corretc case in case it happens to hit binary data? You surely do
not want this, or do you?
Juerd Waalboer
> So you force people to know about the internal flag, lest they cannot avoid
> the die.
No, you don't have to know about the UTF8 flag, just that Perl can't
always know if your string is a text string, but is there to help you
when it does.
> > Besides that, the "C" in Perl's pack() is documented as a single byte.
> "A C "char" is a byte".
> Your words.
> But here you say a byte is not a character. Thats a contradiction.
"C char" ne "Perl character".
> No, I asked for UTF-8 encoded characters. Again, read the documentation:
> * If the pattern begins with a "U", the resulting string will
> * be treated as UTF-8-encoded Unicode.
Resulting string, not input string.
The word "internally" is missing here. I will do my best to correct
that.
> thats for pack, unfortunately.
> U A Unicode character number. Encodes to UTF-8
> internally
> uh, that internal thing again. So how many characters will pack "U", 200
> give me? According to the documentation, 2, as UTF-8 requires that.
One character. Note again that "character" isn't the same as a "C char".
We in Perl land, and the people over in Unicode land, use different
words, sometimes.
Most of the time, a Perl "character" means codepoint.
> > > Right, while the documentation on unpack "U" disagrees with it, as it talks
> > > about UTF-8.
> > That would be a bug, but I can't find it in my copy (5.8.8). It only
> > says "Encodes to UTF-8 internally" for pack(), which as far as I can
> > tell, is true.
> So it talks about using UTF-8, so, according to you, it is a bug. Fine
> with me.
This was for pack, you were talking about unpack. Also, the word
"internally" was probably not added without reason.
Tels
Hash: SHA1
Moin,
On Saturday 31 March 2007 00:04:53 Juerd Waalboer wrote:
> Marc Lehmann skribis 2007-03-31 1:33 (+0200):
> > The difference between us, and thats what it boils down to, is that you
> > give the internal UTF-X bit meaning. You equate UTF-X flag set ==
> > Unicode string.
>
> No, that's a unidirectional thing.
>
> I've said it on p5p at least a dozen times, but I'll say it again:
>
> If the UTF8 flag is set, you can be sure that you have a text string.
> If the UTF8 flag is not set, it can be either a byte string or a text
> string.
>
> If you have a text string, the UTF8 flag may or not be set.
So you are basically saying that you can have any string (text or byte) with
either the flag set, or not. Er, and how do we find out which combination
is which?
I think we all should go to bed and have a nice rest. What you wrote above
makes no sense at all to me now anymore.
So, good night for now,
Te"pls don't tell mom it's 2 already"ls
- --
Signed on Sat Mar 31 02:12:15 2007 with key 0x93B84C15.
Get one of my photo posters: http://bloodgate.com/posters
PGP key on http://bloodgate.com/tels.asc or per email.
Like my code? Want to hire me to write some code for you? Send email!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg3ET3cLPEOTuEwVAQJ7QQf/QmX+IUIaVxgJMSfCrGFnQDRlXzKEHXBk
fIsz1cCNmwPeRJsskLxxkRsC2TlufgccRx3RSN0HcI56l79ldBAvN7uqNgRHEZ2x
JRsIFdT6B13YPFwjAsnSNwl9kIYoRmaXVsFugQELqIbKAKqe/7BGCgnG9qLfN8a0
n6+T3tbpoyWL5MWcDGi6Z+r+GL3bb3GQQQY9GHa4sNU5aWsDcdEOTM9g9KKgINY1
0OIt5nXxPjLEcpOsuqxFA/Xk9kA/EPr/oz4VpZN+9WlahBkL31BJ5Vb3QjbC6eo5
amOAJ+qg04jFu2rLTMBtjunc+/Hvebiz8JsK1Bcb5VeG3GEJKKRTRw==
=2fEH
-----END PGP SIGNATURE-----
Juerd Waalboer
> Yes, and the exact same is true for unicode (both have a 1-1 mapping
> between 0..255 and octets), trivially, of course, as unicode explicitly is
> a superset of latin1.
Unicode is a character set, not a character encoding.
While for 8 bit character sets, the encoding is the same thing, once you
get past the 8 bit boundary, the difference begins to matter.
A unicode string is a sequence of codepoints, not octets. They don't map
1:1 to octets either. To express a unicode string in octects, you need
to encode it. For this, there are several possibilities, including
UTF-8, UTF-16, ...
Unicode is a superset of the latin1 character set, not the latin1
character encoding. We'd need bigger bytes for the latter :)
Marc Lehmann
> My question was posed because I wanted to know how to *keep* a KOI8 (or any
> other random binary) string in Perl without converting it to Unicode. It
> seems to me this is not easily possible because there are literally dozend
> places where your KOI8 string might get suddenly upgraded to UTF-8 (and
> thus get corrupted because Perl treats it is ISO-8859-1). Or did I get this
> wrong?
Yes, you did get that wrong, liekly because Juerd wants users to care about
that. But in fact, if you try it, nothing will get corrupted unless you use
unpack "C" to get the first byte of your KOI8-string. Then you might get
surprised (current perl) or an exception (Juerd's idea).
> In an ideal world, you could either just keep everything in utf-8 (that's
> too slow for some things and not fool-proof either), or rely on no other
> code to corrupt your data - especially this random third party module you
> pulled from CPAN last night. :)
In an ideal world, you would just want to manipulate bytes == characters in
Perl, and do not care about how it treats it internally. It should treat it
as fast as possible, of course.
The same is true for other things in perl: you do not wan tto care wether
your scalar contains an integer, floatingpoint, or strings. Use decides that
in perl: if you print an integer scalar, it (also) turns into a string. If you add
a floating point number to an integer-only scalar, you get the expected
floatingpoint result.
Perl converts between all those "encodings" transparently in a way that makes
most sense. And the same thing is true for character data.
There is a small diference, as Perl can have scalars that have both a string
and a double value, for example, and can then choose the fastest
representation. Perl could just as well keep both an UTF-X encoded as well as
a octet-encoded version of string around to optimise for speed.
Of course, that optimisation would need a lot of memory, so the trade-off
choosen in the current implementation is to upgrade/downgrade when needed,
transparently, so your KOI8-bytes stay KOI8-bytes all the time.
It is the few cases where perl doesn't do that I am concerned about.
> OMHO the problem arises from the fact that Perl makes no distinction between
> a byte string like "a" and a text string like "a", and furthermore,
> manipulating byte string (for instance appending a byte) is done with
> typical string operators. So:
Yeah. It also makes no difference between numbers and strings. Thats Perl.
> # works if $y is 7bit and no utf8 flag
> # but fails if $y is 7bit with utf8 flag
> $byte_string .= $y;
>
> As you said, all is well as long as you can keep these two beasts seperate,
> but the slightest problem might mangle your data. Such as a decode_utf8
> setting the UTF8 bit on a 7bit ASCII string, therefore changing the 7bit
> byte string to a text string.
No, only in Juerd's model where binary data encoded in UTF-X is a bug. In
real-world perl, that just works fine,a dn thats what I expect, and thats I
think what users expect, too: not having to deal with the internal types.
In the same way, you do not have a module that converts numbers to strings,
you just print them:
my $x = 5;
print $x;
Again, pelr transparently handles the details (which includes(!) character
encoding for the outside world!).
> As you said, the current warnings::encode can't decide between the case
> of "BINARY + UTF_8" and "ISO-8859-1 + UTF_8" as Perl makes no distinction
> between binary data and ISO-8859-1. And this missing distinction is
> certainly a bother :)
Only when you hit bugs, or unpack.
Greetings,
Juerd Waalboer
> My question was posed because I wanted to know how to *keep* a KOI8 (or any
> other random binary) string in Perl without converting it to Unicode. It
> seems to me this is not easily possible because there are literally dozend
> places where your KOI8 string might get suddenly upgraded to UTF-8 (and
> thus get corrupted because Perl treats it is ISO-8859-1). Or did I get this
> wrong?
A koi8r string is a byte string. If you keep it separated from text
strings properly, it should not be upgraded and thus treated as latin1.
I'm very curious as to "sudden upgrades" that aren't related to mixing
with text strings. Should you encounter them, please let me know.
Indeed, some functions and operations will not work properly on koi8r,
with regards to character properties. For example, the regex engine has
no idea which characters are word characters, and which are cyrillic. It
can only assume it's either ascii or latin1. For full functionality, you
must decode the string.
If your program is just a gateway in between other things, and doesn't
do any text processing, just keep the thing a byte string.
Just like $jpeg_image is a byte string that contains JPEG data, and this
can be safely used, $koi8r_string can be a byte string that contains
koi8r text data.
> especially this random third party module you pulled from CPAN last
> night. :)
Well, yes, modules sometimes have bugs. That's something we have to
learn to live with.
> As you said, all is well as long as you can keep these two beasts seperate,
> but the slightest problem might mangle your data.
That is true. Programming can be a delicate job. Has always been like
that :)
> Hm, maybe one could write a module that always tackles the encoding to an SV
> via magic. (...) so that if you ever try to fuse two strings together
> where one of them is tagged binary, you get an exception (but only
> then!).
That would be neat. You'd effectively have strong typing. I don't think
you can do this in a module, though. It requires checks all over the
place. Maybe Scott Walters' typesafety module can be of help or
inspiration: http://search.cpan.org/~swalters/typesafety-0.05/
> Yeah, I am not a genius :/ (Sometimes I wish I could upgrade my brain :)
But then, it would be much slower! ;)
> > Codepoints 0..256 in latin1 map to byte values 0..256. That makes it
> > special.
> Erm, I don't buy this because:
> Codepoints 0..256 in KOI8-R (to pick one) map to byte values 0.256. That
> would make it special, too.
I should have said "unicode codepoints 0..255 in latin1 map ...".
The interesting thing about latin1 is that 0..255 overlap with unicode.
The 0..255 (not 256 btw, silly mistake) in koi8-r can all be found in
unicode somewhere, but they're not all in exactly the same places.
Marc Lehmann
> Marc Lehmann skribis 2007-03-31 2:12 (+0200):
> > Yes, and the exact same is true for unicode (both have a 1-1 mapping
> > between 0..255 and octets), trivially, of course, as unicode explicitly is
> > a superset of latin1.
>
> Unicode is a character set, not a character encoding.
As is latin1.
> A unicode string is a sequence of codepoints, not octets.
Nope. You can encode unicode codepoints into UTF-8 and still end up with a
unicode string. Encoding doesn't change the fact that it is unicode that
your are storing.
Since it seems hard to grasp, here is an example:
my $s = "Hello, World!";
$s = Encode::encode_utf8 $s;
$s contains the famous greeting before and after the encoding. It is still
an ASCII string, iso-8859-15 string, and a unicode string, and a text
string, regardless of wether it is encoded or not, that does not change
the fact that that string contaisn the message "Hello, World!".
If you drop ASCII, the same is true for "HallΓΆchen!", which looks
differently in UTF-8 then in an unencoded string, but it is still the same
message. And it is till using unicode to represent the characters.
The fact that you encode something does not change the something that you
encode. Making an arbitrary difference only confuses the issue.
> They don't map 1:1 to octets either. To express a unicode string
> in octects, you need to encode it. For this, there are several
> possibilities, including UTF-8, UTF-16, ...
Sure. Octets are just things that store numbers between 0 and 255. The
most compact way to do that in Perl is using a string. Thats also the most
natural way to represent bytes in Perl, closely followed by integers for
single bytes.
You do not store octets in latin1, or unicode, or whatever else in that
string. You are just using the most natural way to represent octets. And that
just happens to work, because Perl was designed to work that way.
The mapping between perl bytes and octets is 1:1.. ord and chr do it for
you, for example, and unpack "n" does it for you in case you encode/decode
two byte entities. unpack "C", however, does not map to octets in
perl. Thats the bug.
> Unicode is a superset of the latin1 character set, not the latin1
> character encoding. We'd need bigger bytes for the latter :)
Right. And Perl has those bigger bytes.
Juerd Waalboer
> But in fact, if you try it, nothing will get corrupted unless you use
> unpack "C" to get the first byte of your KOI8-string. Then you might get
> surprised (current perl) or an exception (Juerd's idea).
Sigh...
It'll work just fine if the string is still a byte string, which it will
be if you cared to keep it separated.
Marc Lehmann
> > places where your KOI8 string might get suddenly upgraded to UTF-8 (and
> > thus get corrupted because Perl treats it is ISO-8859-1). Or did I get this
> > wrong?
>
> A koi8r string is a byte string. If you keep it separated from text
Your definiton is completely useless in the real world. Obviously, a KOI8-R
string is a text string. It contains text characters. End of story.
> Just like $jpeg_image is a byte string that contains JPEG data, and this
And it is actually an octet string (it makes no difference to C, but it
does make a difference in current Perls, or on the wire).
I will not reply to your mails anymore, as you made your point quite clear
to me: you want behaviour to change dependingon the UTF-X flag, but you do
not want the programmer to know about that. You also have very weird ideas
of what programmers should and should not do the defy reality. I find all
that contradictory, but as you ignore the evidence I presented and the
question I asked you (JSON::XS example), I see no point in continuing
talking to you.
(Note: this is not frustrated *plonk*. I don't hate you, I just think it
is pointless to argue about contradictory statements, and I think you are
mildly abusive, too, in assuming you know everything and therefore ingoring
inconvinient questions. Feels to much like a waste of time).
I also might stay out of this discussion, as I think I made my points
clear. If Perl wants to stay broken w.r.t. Unicode abstraction, it is not
my fault, I tried very hard over the last years to report bugs, and so
far, all of my bug reports w.r..t unicode were right, so I just assume I
am not misinformed about how things should work.
Be good, be well!
Marc Lehmann
> I've said it on p5p at least a dozen times, but I'll say it again:
>
> If the UTF8 flag is set, you can be sure that you have a text string.
Repeating wrong statements does not make them true.
> If you have a text string, the UTF8 flag may or not be set. If you have
> a byte string, the UTF8 string is not set (or it was set because you
> treated the byte string as a text string).
No, please look at my example of JSON.
> > The problem with your approach is that you have to expose the UTF-X flag
> > to users. Which comes with a lot of problems.
>
> Again: you're kidding, right?
>
> I'm constantly very explicitly and verbosely telling people to NOT look
> at the flag, NOT set it manually, etcetera.
So why do you propose that people have to make sure that they never put a
binary string with the UTF-X flag set into unpack?
How are users supposed to do that, unless they know about he flag in the
first place?
No, I am not kidding. You are part of the crowd who wants to expose the
UTF-X flag to the perl level, despite your claims that you do not want to.
> Heck, I've even explained that I think you should try to (pretend to) be
> ignorant about the internals, in response to your message even!
Right, and then you want perl functions to die depending on the setting of
that flag, even though you also claim Perl users should not need to know
about it.
So you tell users when they get that error message that they did somethign
wrong that they should not care about?
No, I am certainly not kidding.
> I do not understand how you are able to misinterpret this message even
> after this many posts in this thread alone. Have you ever read
> perlunitut, even?
As I said, I have no such manpage, and even if I had, it has nothing to do
with this. I am not misinterpreting your message at all.
You want perl functions to behave different depending on wether that flag is
set or not. I want perl functions to behave the same, regardless of the fact.
You expose the UTF-X flag that way. I don't.
You *are* contradicting yourself, but that has nothing to do with me not
reading that document or not. Thats alone your problem.
Either you do expose the UTF-X flag by making perl functions behave
differently, or you don't.
No matter of claiming you donot want to expose it can fix that: You do,
wether you want or not, if you change Perl semantics to make a difference.
> That's not what I said, nor what I meant. In fact, quite the opposite.
So then unpack should not croak when it sees the UTF-X flag?
> If you're just spending this evening just to get on my nerves, then
> congratulations!
No, I am trying to make you understand the typeless nature of Perl, and
that your proposals expose the UTF-X flag, no matter what you *want*.
You could just understand that for a change, then maybe you wouldn't need to
accuse me of just trying to get on your nerves.
I do understand that you said you do not want to expose that flag. But as
long as the changes you propose do that, it is being exposed.
I am sorry that I can't say it any clearer.
> > Because "internal format" strings can store binary data just as well,
> > and often does.
>
> Yes, and when you use such a byte string as a text string, its bytes are
> considered to be codepoints, just like in latin1.
Yeah, sure. Mind you: no mention of UTF-X.
> > I am talking purely about the perl level strings. If perlunitut confused
> > the issue by talking about internal encoding it completely failed its
> > mission, imho.
>
> I strongly suggest that you READ the document before whining about its
> supposed failure.
Well, I trust that you don't misquote its contents. Did you?
> > The problem is that some parts of perl make a difference bewteen the
> > very same string, depending on how it is encoded internally, _even if
> > the encoding is the same on the Perl level_.
>
> Those are bugs. Report them, and they might get fixed.
I did. Thats the whole point of this thread. I reported them a number of
times. How could you miss that?
> > > utf8::encode is a text operation. It will assume that whatever you give
> > > it, is a text string. Its characters are considered Unicode codepoints.
> > Where does it say so?
>
> Well, you have already denied that "encoding is going from characters to
> bytes" is a real world fact, so I guess there's little point in pointing
> out the places where exactly the same thing is explained.
If it is wrong, its wrong. No matter how often you try to explain
it. People do store octets in UTF-8. Even perl extends UTF-8 to UTF-X
to make interesting usages possible. So yes, if thats broken, then Pelr
is already broken, fundamentally, by allowing non-unicode-codepoints in
strings.
Choose two: your claims are wrong, or Perl is wrong. Either way suits me,
although I personally think the current model makes much more sense then
your user-has-to-care-for-UTF-X flag explicitly model.
> > > you need to know some internals.
> > Wrong. I need know no internals
>
> A certain Marc Lehmann once said:
>
> "I would love if that were the case, but the powers to be decided that
> every perl progarmmer has to know those internals, and needs to be able
> to deal with them."
Yes. Any problems with that?
As you like to quote with misleading context, let me add that the context
was unpack and perl modules using it or XS, not utf8::encode.
You make a classical logical fallacy: just because some parts of Perl do
not force you to know internals this does not mean that all of Perl does
not force you.
> > > That makes no sense, because UTF-8 is a means of representing
> > > characters. Byte strings consist of bytes, not characters.
> > Not in C, which is what the documentation constantly refers to, mind
> > you.
>
> And that is bad, I agree. Perl programmers should not be expected to
> speak C in order to understand Perl documentation. This is a big
> problem in Perl's documentation, but who's going to fix it?
I donot suffer from it. I just want sane behaviour in Perl, which doesn't
force me to think about wether my UTF-X flag could be set and my program
could die because of that, but where I get the correct and expected
results.
Marc Lehmann
I do talk about the *Perl* level, while you often talk about the
*implementation*. When I say byte or octet string below, I mean on the
Perl level. For example, on the Perl level, upgrading a string does not
change its semantics anywhere except w.r.t. to bugs and unpack: It still
stays an octet string if it was an octet string before.
(Thats of course all in line with me not wanting to expose the UTF-X
flag).
Marc Lehmann
On Sat, Mar 31, 2007 at 01:08:21AM +0200, Juerd Waalboer <ju...@convolution.nl> wrote:
> Marc Lehmann skribis 2007-03-31 0:25 (+0200):
> > If you send a compressed string over the network using JSON and decompress
> > it, you need to know that.
>
> Does JSON compress arbitrary data?
no.
> If so, then the user must do the decoding and encoding,
No, compression is something completely orthogonal from encoding. Neither
forces me to do the other.
> because arbitrary data only exists in byte form
Thats eems completely wrong to me.
> Once you dictate any specific encoding, it's no longer arbitrary.
JSON dictates unicode for the JSON text, and strongly hints at the use of
UTF-8 for interchange purposes.
> On the other hand, if JSON does text data only,
No, it does support binary data just as well. It is used a lot, too.
It works just like perl without the bugs: You have a string type that can
store bytes. It is up to the user to interpret them as she wants.
> it can just use any UTF encoding on both sides, and document it like
> that.
It is a bit complicated, but you can safely assume that 99% of all JSON
is UTF-8 encoded. In fact, you can recode all JSON documents into ASCII,
too. JSON::XS offers that, and JSON::XS by default encodes to/decodes
from UTF-8, but allows the user to decode/encode himself. JSON text is
composed of unicode characters, and in Perl some JSON modules store them
as a simple Perl string.
All that is not well-supported by most JSON modules, though, for example
JSON::XS is the only module for perl that correctly decodes escaped
surrogate pairs.
> Unless both sides are exactly the same platform (e.g. both Perl), you
> need to establish a protocol for sending data anyway. And that protocol
> should also describe encoding. If sender and receiver don't agree, you
> have a problem.
No, it doesn't have anything to do with the platform. Even when both sides
use Perl I need to decide on a common encoding. Thats strictly outside the
JSON definition, though.
> > I am really frustrated at that. It makes perl as a whole rather
> > questionable for unicode use, as you constantly have to think about
> > the internals. And yes, that simply shouldn't be the case.
>
> I maintain that it isn't the case, for almost any programming job,
> unless you're indeed doing things with internals.
Well, the JSON::XS module certainly does things with the internals, it
has to flag some strings as UTF-X, and in fact flags all strings that
way unless you enable the shrink option, which is documented to try to
shrink the memory used in various ways (one way is to try to downgrade the
scalar).
Certainly, the user who reported the bug also didn't look at the
internals. Compress::Zlib called unpack "CCCV" or somesuch, though, which
unfortunately treats V very different from C, by looking at the internals
with "C", and not doing that and treating the string as an octte string
with "V".
The user suggested that JSON::XS corrupts binary data because it happens to
be returned upgraded unless you set the shrink option.
However, Perl does not expose the internals elsewhere, the upgraded
version is semantically equivalent to the downgraded one unless you use
an XS module using SvPV directly or indirectly (considered a bug in Perl
when I understood nick correctly), or when using unpack "C", as that has
a different meaning in perl 5.6 than in perl 5.005, and has confusing
documentation.
The right thing for Compress::Zlib is not to use unpack "CCCV" but unpack
"UUUV", which seems completely weird to me, as no unicode was ever
involved *on the perl level*.
Juerd Waalboer
> > A koi8r string is a byte string. If you keep it separated from text
> Your definiton is completely useless in the real world. Obviously, a KOI8-R
> string is a text string. It contains text characters. End of story.
This is a logical thing to say, but unfortunately not very useful.
The distinction between a text string, and a byte string representing
text, is actually useful.
> You also have very weird ideas of what programmers should and should
> not do the defy reality.
Weird ideas, maybe, but at least weird ideas that help dozens of people
write working and maintainable code.
You don't believe in my weird ideas, fine. But I find it very
interesting that you run into all these problems with Perl's unicode
support, while the people who stick to my weird ideas write lots of code
without that.
> I find all that contradictory, but as you ignore the evidence I
> presented and the question I asked you (JSON::XS example), I see no
> point in continuing talking to you.
Unfortunately, I understand very little of the JSON example. I don't
know JSON and would have to learn about it first.
Juerd Waalboer
> Repeating wrong statements does not make them true.
I'll refrain from the obvious response.
> No, please look at my example of JSON.
JSON is pretty big to just quickly examine. I have nothing set up for
testing it.
> > I'm constantly very explicitly and verbosely telling people to NOT look
> > at the flag, NOT set it manually, etcetera.
> So why do you propose that people have to make sure that they never put a
> binary string with the UTF-X flag set into unpack?
Not unpack in general, but unpack "C".
Because "C" is explicitly catered for byte data, which strings with the
UTF8 flag aren't. It won't always catch mistakes, because indeed lack of
the flag says nothing, but it can help catch some of them.
Perl already has a similar warning in many places, for example when you
print such a "wide character" on a filehandle that has no encoding or
utf8 layer. Some modules, like MIME::Base64, provide the same
functionality.
> How are users supposed to do that, unless they know about he flag in the
> first place?
By keeping byte strings and text string separate. Please either accept
this, or stop asking me questions that will lead to this answer.
> Right, and then you want perl functions to die depending on the setting of
> that flag, even though you also claim Perl users should not need to know
> about it.
The warning would not be a new feature, but an existing feature applied
in more places. "die" is probably too harsh indeed.
> So you tell users when they get that error message that they did somethign
> wrong that they should not care about?
When they get the error message, they can read the following in
perldiag:
Wide character in %s
(W utf8) Perl met a wide character (>255) when it wasnβt expecting one. This warning is by default on for I/O
(like print). The easiest way to quiet this warning is simply to add the ":utf8" layer to the output, e.g.
"binmode STDOUT, β:utf8β". Another way to turn off the warning is to add "no warnings βutf8β;" but that is
often closer to cheating. In general, you are supposed to explicitly mark the filehandle with an encoding,
see open and "binmode" in perlfunc.
Changing the order of these sentences is on my to-do list.
Note how this clear explanation doesn't mention the UTF8 flag!
> As I said, I have no such manpage
See bleadperl or Google.
> You want perl functions to behave different depending on wether that flag is
> set or not. I want perl functions to behave the same, regardless of the fact.
I want Perl to warn about certain mistakes when it can.
> > That's not what I said, nor what I meant. In fact, quite the opposite.
> So then unpack should not croak when it sees the UTF-X flag?
No, it should warn instead. From now on, I no longer think it should die. It
should warn, and people who want it to die can do so with "use warnings FATAL".
> > > The problem is that some parts of perl make a difference bewteen the
> > > very same string, depending on how it is encoded internally, _even if
> > > the encoding is the same on the Perl level_.
> > Those are bugs. Report them, and they might get fixed.
> I did. Thats the whole point of this thread. I reported them a number of
> times. How could you miss that?
I don't usually read bug reports, and never claimed to have done so.
But in this special case, I will make an exception, and read the Unicode
related bug reports that you have submitted.
Juerd Waalboer
> > Unicode is a character set, not a character encoding.
> As is latin1.
For all intents and purposes, latin1 is a character encoding as well as
a character set. If not officially, then certainly for Perl. It can be
used with the :encoding layer, with Encode'decode, etcetera. "Unicode"
cannot.
I don't know where your terminology comes from, but I try to stick to
whatever is common in Perl land. Sorry if that differs from other
communities.
> > Unicode is a superset of the latin1 character set, not the latin1
> > character encoding. We'd need bigger bytes for the latter :)
> Right. And Perl has those bigger bytes.
A byte, in Perl jargon at least, is an octet. An octet can hold any
single value in the rande 0..255, and is exactly 8 bits in size. Every
byte is exactly as large as any other byte.
Juerd Waalboer
> Oh, maybe I know the reason for the confusion.
> I do talk about the *Perl* level, while you often talk about the
> *implementation*. When I say byte or octet string below, I mean on the
> Perl level.
This is not the reason for confusion, because I also discuss the Perl
level. For my terminology, I use whatever is common in the Perl
reference documentation.
> For example, on the Perl level, upgrading a string does not
> change its semantics anywhere except w.r.t. to bugs and unpack: It still
> stays an octet string if it was an octet string before.
s/octet string/character string/ and you're entirely right. "Octets" are
a bit harder, because of the definition of an octet:
octet
<jargon, networking> Eight bits. This term is used in
networking, in preference to byte, because some systems use the
term "byte" for things that are not 8 bits long.
There's no easy way to fit numbers greater than 255 into 8 bits without
sacrificing support for 0 thru 255 inclusive. It may even be impossible.
Who knows. The person who invents a way of storing more than 255
distinct numbers in unique single octets, will probably get famous very
quickly :)
Juerd Waalboer
> Personally, I think that unpack with a byte-specific signature should
> die, or at least warn, when its operand has the UTF8 flag set.
I've since this post changed my mind, and think it should only warn if
there are wide characters after attempting to downgrade first. Just like
the existing "wide character in %s" warning.
juerd@lanova:~$ perl -wle'$a = "foo\x{ff}"; utf8::upgrade($a); print $a' | hexdump -C
00000000 66 6f 6f ff 0a |foo..|
00000005
juerd@lanova:~$ perl -wle'$a = "foo\x{20ac}"; utf8::upgrade($a); print $a' | hexdump -C
Wide character in print at -e line 1.
00000000 66 6f 6f e2 82 ac 0a |foo....|
00000007
Marc Lehmann
> Juerd Waalboer skribis 2007-03-30 21:53 (+0200):
> > Personally, I think that unpack with a byte-specific signature should
> > die, or at least warn, when its operand has the UTF8 flag set.
>
> I've since this post changed my mind, and think it should only warn if
We are making progress, and I would actually be content with that
solution, but it does break "U". The solution, really, is to treat C like
an octet in the same way "n" is treated like two octets. That does not
break existing code and is what many perl programmers find naturally.
Since so many people are confused about why the unpack change breaks code, I
will explain it differently:
my $k = "\x10\x00";
die unpack "n", $k;
this gives me 4096. "n" is documented to take exactly 16 bits, two octets.
I get 4096 regardless of how perl chooses to represent it internally: If
perl goes to using UCS-4 (something that won't happen for sure, but has
been stated before to remind people that internal encoding can change), it
would still work.
Same thing for "L", which is documented to be exactly 32 bit.
Now, when people want an 8 bit value followed by a 16 bit big endian value,
they used "Cn" in the old times. In fact, they still use that, as "C"
always has been the octet companion to the 16 bit and 32 bit sSlLnNvV etc.
However, in a weird stroke, somebody decided that "C" no longer gives
you a single octet of your string, but, depending on internal encoding,
depending on an internal flag, part of that octet or the octet.
Now, what has been unpack "CCV" in perl 5.005 must be written as unpack
"UUV" in perl 5.8, as "U" has the right semantics for decoding a single
octet out of a binary string.
Thats weird, because now code that _doesn't_ want to deal with unicode at
all, but in fact only deals with binary data must use this unicode thingy
"U", even though the documentation for "C" clearly says its an octet, and
even says its an octet in C, which is exactly what those people decoding
structures or network packets want.
That is the problem.
Now, I don't mind at all if I get a die when trying "C" on a
byte=character that is >255 (i.e. not representable as an object). Or a die
when attempting that on a two byte=character string with "n".
I personally dislike the warning, because the warning only ever comes up
when there is a bug. It doesn't matter much to me persoanlly, though.
What matters to me is that binary-only code now needs to use "U" when
formerly "C" as meant to get correct behaviour. This *needs* to be fixed.
Marc Lehmann
> Marc Lehmann skribis 2007-03-31 2:48 (+0200):
> > > A koi8r string is a byte string. If you keep it separated from text
> > Your definiton is completely useless in the real world. Obviously, a KOI8-R
> > string is a text string. It contains text characters. End of story.
>
> This is a logical thing to say, but unfortunately not very useful.
Thanks, I'll take logical over subjective opinions any day.
> The distinction between a text string, and a byte string representing
> text, is actually useful.
It is useful, but making it the mandatory is stupid, because you lose the
ability to handle real-world situations, for example JSON, which simply does
not make the distinction. Ther same is true for Pelr, which also does not
make the distinction.
> > You also have very weird ideas of what programmers should and should
> > not do the defy reality.
>
> Weird ideas, maybe, but at least weird ideas that help dozens of people
> write working and maintainable code.
Likely, but its still your personal opinion, your personal coding style.
Forcing that on everybody else by calling everything that doesn't fit
(such as JSON) "broken" does not convince _me_ that it is a good coding
style.
> You don't believe in my weird ideas, fine. But I find it very
> interesting that you run into all these problems with Perl's unicode
> support, while the people who stick to my weird ideas write lots of code
> without that.
Goddamnit, I more than once told you that I am not running into those
problems because I know most perl bugs regarding unicode inside and out. I am
doing unicode programming for far longer than Perl easily supports it, and I
would be grateful if you would stop bullshitting me and spreading lies.
I *explicitly* said that it is other users who hit problems, and that I
can cope with them quite well.
> > I find all that contradictory, but as you ignore the evidence I
> > presented and the question I asked you (JSON::XS example), I see no
> > point in continuing talking to you.
>
> Unfortunately, I understand very little of the JSON example. I don't
> know JSON and would have to learn about it first.
Well, its one of that reality things where your coding style blankly breaks
down: JSON makes no difference between binary and text, except that binary
only uses character indices 0..255. You do not know wether a json string is
binary or text. Usage decides.
One such usage is unpack, and I find it weird that I have to use "U" to get
binary semantics in unpack. Or you have to downgrade explicitly.
Anyways, that clashes with your notion that the programmer made a bug when
binary data happens to be UTF-X encoded internally. Reality hits, you
lose, simply because calling usage of JSON broken according to your coding
standards will not have any effect on JSON.
And the way JSON handles binary is extremely common in the real world. And
it is exactly how perl handles it, modulo bugs and, well, unpack (and the
unfortunate decision to give old XS code sometimes bytes encoded in UTF-X,
sometimes not).
Perl simply does _not_ work like you want it to. Instead, it is much simpler
because in the majority of cases it just works without having to track wether
my binary string came in contact with something that upgraded it. I simply do
not have to care in Perl, except for the cases above.
And thats the good thing. Teaching people to avoid upgrading by your text vs.
binary string technique is confusing. It is backwards. People should not have
the need to be concerned about upgrading, because it is an internal thing.
And yes, I said I would not answer you, but what prompted it was your
continuous abusive behaviour of putting words into my mouth I have
*explicitly* said to not have said, and explaine dit in detail.
Marc Lehmann
> > For example, on the Perl level, upgrading a string does not
> > change its semantics anywhere except w.r.t. to bugs and unpack: It still
> > stays an octet string if it was an octet string before.
>
> s/octet string/character string/ and you're entirely right. "Octets" are
> a bit harder, because of the definition of an octet:
Please stop correcting completely correct statements. I am entirely right
when I talk about octet strings above. It is a trivial fact. It is said
when you think it isn't entirely correct, but that doesn't give you the
right to your current behaviour.
> <jargon, networking> Eight bits. This term is used in
And stop lecturing me about basic stuff. I quite well know what an
octet is, and I am quite certain when I chose "octet" over "byte" or
"character".
I meant "octet string" above, and my statement is entirely correct with
"octet string".
If you think you need to correct me, please state why the above isn't
entirely right in its original form. In fact, I now assume you are
very confused about that byte/octet/character stuff if you cannot even
understand the correctness of simple facts like the sentence on top of
this mail.
And I am full of your ridicule and belittlement. I am not impressed by
people who make empty claims and miscorrect completely correct statements
because they have difficulties understanding them. If you want to be
taken seriously, try it with logics (which you find not useful at times),
not abusive behaviour (which certainly isn't useful anytime). Certainly I am
not impressed by illogical arguments, or even non-arguments such as personal
coding style preferences, which I happily tolerate as opinions, but should
never be presented as the only true way without sound arguments.
Marc Lehmann
> JSON is pretty big to just quickly examine. I have nothing set up for
> testing it.
Not my problem. Your coding style cnanot handle it, though, so in your own
interest you should try to examine it some day.
> > > I'm constantly very explicitly and verbosely telling people to NOT look
> > > at the flag, NOT set it manually, etcetera.
> > So why do you propose that people have to make sure that they never put a
> > binary string with the UTF-X flag set into unpack?
>
> Not unpack in general, but unpack "C".
>
> Because "C" is explicitly catered for byte data, which strings with the
> UTF8 flag aren't.
Well, you are not tlaking of Perl here.
> It won't always catch mistakes, because indeed lack of
> the flag says nothing, but it can help catch some of them.
Having the flag means nothing, either.
> Perl already has a similar warning in many places, for example when you
> print such a "wide character" on a filehandle that has no encoding or
> utf8 layer. Some modules, like MIME::Base64, provide the same
> functionality.
It is similar, but it works completely different: It only warns if you pass
something into a function/filehandle that knows that it is expecting binary
data.
Unlike unpack, the UTF-X flag has nothing to do with the warning: the warning
tells you that the data you pass in is not binary data because it contains at
least one character >255. Thats completely fine. But when I do pass in a
string only consisting of octets (in the perl level), then it gets passed
into the funciton as binary, as one would expect.
And that, again, has nothing to do with the UTF-X flag. Data passed into
such a function gets properly downgraded (that process is what actually
generates the warning, btw).
> > How are users supposed to do that, unless they know about he flag in the
> > first place?
>
> By keeping byte strings and text string separate. Please either accept
> this, or stop asking me questions that will lead to this answer.
I am asking about how users do that, I am not askign what you think they
should do. I am asking specifically _how_ your idea should be put into
practise. I gave you an example where the only currently known way to do that
is by knowing and manipulating the internal UTF-X flag.
And since you have not given an answer to that question, it stays a valid
question.
The problem is that your coding style cannot resolve this situation, as
the module in question (JSON::XS) does not know wether the given piece of
data is binary or text. Only the user knows, but by ghen it is already
upgraded.
> > Right, and then you want perl functions to die depending on the setting of
> > that flag, even though you also claim Perl users should not need to know
> > about it.
>
> The warning would not be a new feature, but an existing feature applied
> in more places. "die" is probably too harsh indeed.
No part in perl acts like that, see above, the parts that generate that
warning are all downrading properly, ensuring the perl promises of string
handling are kept.
> When they get the error message, they can read the following in
> perldiag:
>
> Wide character in %s
> (W utf8) Perl met a wide character (>255) when it wasnβt expecting one. This warning is by default on for I/O
> (like print). The easiest way to quiet this warning is simply to add the ":utf8" layer to the output, e.g.
> "binmode STDOUT, β:utf8β". Another way to turn off the warning is to add "no warnings βutf8β;" but that is
> often closer to cheating. In general, you are supposed to explicitly mark the filehandle with an encoding,
> see open and "binmode" in perlfunc.
>
> Changing the order of these sentences is on my to-do list.
You are completely confused. I am talking about octet strings (or byte
strings in your parlance). That string _never_ triggers that warning,
regardless of how it is encoded internally, because octte strings nver
contain wide characters.
Thats how the abstraction should work.
Your change of warning when the UTF-X bit is set would break that
abstraction, because users suddenly would get that warning for strings that
do not contain wide characters *at all*.
Thats I can only call very misleading to users.
> Note how this clear explanation doesn't mention the UTF8 flag!
Exactly: because you didn't understand the mechanics of that warning
because it doesn't do what you claim it does, namely warn if the UTF-X
flag is set but instead does the right thing and warns when there *is* a wide
character in the string, regardless of how it was encoded.
Do you finally understand? Please!
> > You want perl functions to behave different depending on wether that flag is
> > set or not. I want perl functions to behave the same, regardless of the fact.
>
> I want Perl to warn about certain mistakes when it can.
No, you want Perl to warn even when no mistakes happened because you
equate UTF-X flag with "contains no (binary) octets/bytes".
But thats not how Perl works. Thats where you misunderstand how the UTF-X
flag works. Perl warns on real problems (and probably should die), not
because the UTF-X flag happens to be set, which is misleading.
Do you finally understand how Perl works?
> > > That's not what I said, nor what I meant. In fact, quite the opposite.
> > So then unpack should not croak when it sees the UTF-X flag?
>
> No, it should warn instead. From now on, I no longer think it should die. It
> should warn, and people who want it to die can do so with "use warnings FATAL".
Of course it should not warn. That *exposes* the UTF-X flag to the
user. And the warning you quote would simply be wrong, because users would
get that warning even when no wide character is in the string at all.
> I don't usually read bug reports, and never claimed to have done so.
>
> But in this special case, I will make an exception, and read the Unicode
> related bug reports that you have submitted.
Maybe you learn what the UTF-X flag does, and why it shouldn't be exposed
in the way you think it should be or is currently exposed.
The UTF-X flag is *no* indication of a wide character whatsoever. In Perl.
I think its obvious by know that you are do not know very much about
unicode handling vs. the UTF-X flag in Perl. At least your knowledge is
mostly wrong it seems.
And thats sad, because it could be very simple, and for the most part already
is very simple: Often used modules will simply be improved to use SvPVbyte
explicitly, even if there is no default typemap support for it. And Modules
requiring binary data will eventually be fixed to use "U" instead of "C" for
decoding single octets. And the rest of perl works relatively fine, and the
remaining issues will be fixed, too.
I just think it would be much better for Perl if those changes were not
required and things would just continue to work by providing backwards
compatibility.
Juerd Waalboer
> > > Personally, I think that unpack with a byte-specific signature should
> > > die, or at least warn, when its operand has the UTF8 flag set.
> > I've since this post changed my mind, and think it should only warn if
> We are making progress, and I would actually be content with that
> solution, but it does break "U".
No, breaking U does not occur, because it's not in my list of
byte-specific (un)pack templates. U is for unicode characters.
> The solution, really, is to treat C like
> an octet in the same way "n" is treated like two octets.
It does that, but we're having a very different understanding of the
word "octet", and my hands hurt, so I'm not going through it all again.
> Since so many people are confused about why the unpack change breaks code, I
> will explain it differently:
> my $k = "\x10\x00";
> die unpack "n", $k;
> this gives me 4096. "n" is documented to take exactly 16 bits, two octets.
juerd@lanova:~$ perl -le'print unpack "n", "\x{20ac}"'
57986
"\x{20ac}" is one character, but "n" works on octets, not characters.
This uses the internal buffer without warning, and picks the first two
octets of the three-octet secuence e2 82 ac. This octet sequence should
be hidden from the programmer, but it is too late for that. So instead,
let's warn the programmer that what's going on is very probably not what
they intended.
juerd@lanova:~$ perl -le'print unpack "n", "\xe2\x82"'
57986
The annoying thing for people who don't know when Perl upgrades strings,
is when you started with a nice 2-octet byte string, and it got upgraded
somewhere. Here, forced for illustration, and using the same 2-octet
sequence so the difference in results is obvious:
juerd@lanova:~$ perl -le'$foo = "\xe2\x82"; utf8::upgrade($foo); print
unpack "n", $foo'
50082
A warning about the wide characters here would be in order and save
people's butts.
> I get 4096 regardless of how perl chooses to represent it internally
Because Perl always uses latin1 or utf8 internally, in both of which
\x10 and \x00 are octets 0x10 and 0x00 respectively.
> If perl goes to using UCS-4 (something that won't happen for sure, but
> has been stated before to remind people that internal encoding can
> change), it would still work.
Not as far as I can tell, because Perl uses the raw octets of the
internal encoding whenever you do byte-specific operations, and the
internal encoding for U+0010 and U+0000 changes when you go from UTF-8
to UCS-4.
That's why it's so darn useful to use latin1 when possible, because you
can then be pretty sure that "\x10\x00" will be the two octets you
expect. (Note that breaking this is the main breakage caused by
encoding.pm.)
> However, in a weird stroke, somebody decided that "C" no longer gives
> you a single octet of your string, but, depending on internal encoding,
> depending on an internal flag, part of that octet or the octet.
What you call "octet", I call "character". And I'll never call that
"octet" or "byte" because then none of the documentation about all this
would still be right, and Perl would suddenly indeed be broken.
If you insist on calling the value of "\x{20ac}" a single octet, then
indeed pack/unpack will not do what you want, because what you want is
just not how it works.
"\x{20ac}" is one character. Internally, represented by three octets.
The internal representation is used, if you unpack with byte-specific
templates like "C" or "n".
Byte strings, i.e. strings with no character values >255 that have never
been in contact with UTF-8 encoded strings, may be interpreted as latin1
and internally converted to UTF-8 when you join them with text strings.
This causes unpack to see very different values, and that's one of the
reasons one should avoid mixing byte strings and text strings.
Note that my definition of "text string" excludes byte encoded strings,
such as the results of encode() or utf8::encode().
> Now, what has been unpack "CCV" in perl 5.005 must be written as unpack
> "UUV" in perl 5.8, as "U" has the right semantics for decoding a single
> octet out of a binary string.
> Thats weird
Weird only because you choose to use a different meaning of the word
"octet" than much of the rest of the world.
> Now, I don't mind at all if I get a die when trying "C" on a
> byte=character that is >255 (i.e. not representable as an object).
Just so other people know: since Perl has had Unicode support, there has
been a consistent effort to teach people that character != byte, and
that a single character may consist of several bytes.
In fact, this effort has been present in larger parts of computing than
just Perl, but for clarity's sake, I'm sticking to Perl because
sometimes Perl's definitions differ. (For example, in Perl, a character
is a single code point, while in Unicode, a character can be composed
out of several combining code points.)
Also, values greater than 255 do not fit in a single byte, according to
computer science that decided that byte==octet==8 bits. 8 bits simply
simply hold only 2**8==256 values. Hence the need for a distinction
between bytes, and things that *are* able to hold other values.
> I personally dislike the warning, because the warning only ever comes up
> when there is a bug.
I love warnings that only ever come up when I have a bug. In fact, I
generally dislike warnings that don't follow that pattern.
Tels
Hash: SHA1
Moin,
On Saturday 31 March 2007 00:20:52 Marc Lehmann wrote:
> On Sat, Mar 31, 2007 at 01:39:06AM +0000, Tels
<nospam...@bloodgate.com> wrote:
> > My question was posed because I wanted to know how to *keep* a KOI8 (or
> > any other random binary) string in Perl without converting it to
> > Unicode. It seems to me this is not easily possible because there are
> > literally dozend places where your KOI8 string might get suddenly
> > upgraded to UTF-8 (and thus get corrupted because Perl treats it is
> > ISO-8859-1). Or did I get this wrong?
>
> Yes, you did get that wrong, liekly because Juerd wants users to care
> about that. But in fact, if you try it, nothing will get corrupted unless
> you use unpack "C" to get the first byte of your KOI8-string. Then you
> might get surprised (current perl) or an exception (Juerd's idea).
I should have said "random binary data" not "KOI8". "KOI8" implies the data
is some sort of text that can be "upgraded" to utf-8.
Now, you can *always* treat random binary datas f.i. ISO-8859-1, upgrade it
to UTF-8 and then downgrade it again, since this is a lossless
transformation. But that doesn't mean it is a good idea because:
* speed - useless transcodings
* memory (utf-8 needs more memory, and the transcoding, too)
* pack/unpack or any other "peeking" at the data might leak the fact that
Perl suddenly converted "\xfc" to "\xc3\xbc" underneath (as Marcs bugreport
showed).
So, yes, if Perl works perfectly in every place, converting you data always
on the fly whenever you look at it, you could stuff "KOI8" or any other
random binary data in, have it (maybe) converted to utf-8, and on
output/looking at converted back to the exact bytes you stuffed in.
However, as you demonstrated yourself, Perl doesn't work perfectly :)
What I was trying to get at is there are different types of data. Before any
encoding or data examination goes on you have:
** random binary data (see notes above why you do not want this treated as
ISO-8859-1 and "text"). Basically, you never want Perl to encode/decode it,
and any attempt in doing so should result in an warning/exception. (utf-8
flag off)
** ascii 7 bit data (utf-8 flag off)
** 8bit data with an encoding (assumed is ISO-8859-1, but user can specify
other types of encoding during a call to "decode") (utf-8 flag off)
** utf-8 data (utf-8 flag on)
As you can see, there are four different types of data, but Perl has only
one bit flag to distiguish them.
So whenever you have data without the utf-8 flag, Perl needs to decide
between the three cases mentioned above. And since it cannot store the
decision of "already seen 7bit ASCII", it needs to do this again sometime
later.
This is costly (scanning for hight bit characters to distiguish between 7bit
ascii and 8bit "something else"), and it overly simple, because Perl cannot
distiguish between "text data in ISO-8859-1 or whatever encoding is in
effect" and "binary data which shouldn't be treated as text".
As an author who inherited software that deals with random binary data (e.g.
JPEGs), this deficency concerns me.
Unfortunately, I am in no position to do anything about it except bitch on
some random mailing list :( Wheere is a time-machine whenever you need one?
[snip]
> > As you said, the current warnings::encode can't decide between the case
> > of "BINARY + UTF_8" and "ISO-8859-1 + UTF_8" as Perl makes no
> > distinction between binary data and ISO-8859-1. And this missing
> > distinction is certainly a bother :)
>
> Only when you hit bugs, or unpack.
<sarcasm> and you never hit bugs, or use unpack </sarcasm> :)
All the best,
Tels
- --
Signed on Sat Mar 31 11:28:57 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
"Duke Nukem Forever will come out before Unreal 2."
-- George Broussard, 2001 (http://tinyurl.com/6m8nh)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg5J2ncLPEOTuEwVAQJTzwf/TH9JUUnoTOq8+sRpROPhb17oWRjLmNs4
+S+vuSldaCk0qxG6LB8NvoJW8BEX7ldz+4zTaEn0/WKi3e+v9YmWFMqblqnRLm5H
lEH7FbVCY+TAINJfVj24JJNaBtZc6ptqqYNzStuVD0T2aNutv5vIVgTdKtkgdYHM
gLuG53iqN70zqwOSnn/Acq91zC56/LvEkGRZzdBwwj+qWbC7UXLJhRtc3ZuCCI9m
DblbMiKoGzorDF7dQVeguBnyohvdCEvKqMPOvs6Wp/ZVReN/DDXhlsGh7kJ3Pjl2
9C9Nmds9KuFkmvsleXZEy5KPmGIKyJVX33llQKPj9woe0g2Iyjeaeg==
=4lLh
-----END PGP SIGNATURE-----
Juerd Waalboer
> I should have said "random binary data" not "KOI8". "KOI8" implies the data
> is some sort of text that can be "upgraded" to utf-8.
Not if "upgrading" refers to the process that Perl has when it goes from
latin1 to utf-8, because this doesn't handle arbitrary encodings like
koi8r.
Your best bet is to treat koi8r encoded data as binary data. (And as
such, not mix it with text data.) If this is confusing, you may want to
gzip it first, and ungzip it afterwards ;)
> Now, you can *always* treat random binary datas f.i. ISO-8859-1,
> upgrade it to UTF-8 and then downgrade it again, since this is a
> lossless transformation. But that doesn't mean it is a good idea
Exactly.
> * speed - useless transcodings
> * memory (utf-8 needs more memory, and the transcoding, too)
> * pack/unpack or any other "peeking" at the data might leak the fact that
> Perl suddenly converted "\xfc" to "\xc3\xbc" underneath
Good summary. Also, if you output it to an encodingless filehandle
before downgrading it again, the value may contain characters greater
than 127, and you'll get output that you probably did not intend.
> ** random binary data (see notes above why you do not want this treated as
> ISO-8859-1 and "text"). Basically, you never want Perl to encode/decode it,
> and any attempt in doing so should result in an warning/exception. (utf-8
> flag off)
Yep.
> ** ascii 7 bit data (utf-8 flag off)
The UTF8 flag can also off for 8 bit data. For ASCII data it will
typically be off, but it wouldn't matter if it were on. (That is, if you
treat ASCII data like text. You don't want to treat UTF8 carrying data
as binary, though, because you will want to mix binary data with other
binary data, without having it upgraded.)
> As you can see, there are four different types of data, but Perl has only
> one bit flag to distiguish them.
I'd say it has two types of data, and indeed that one bit.
With the bit on, it's unicode data that internally is encoded as UTF-8.
You're not supposed to access the UTF-8 encoded octet buffer. This
string should never be used with octet operations like vec or unpack "C"
or "n".
With the bit off, it's either unicode data that internally is encoded as
ISO-8859-1, or it is binary data. This string can safely be used for
octet operations (but of course, that doesn't make sense if the sting
was intended as text, with the exception of some ancient 8bit things
crypt()).
> So whenever you have data without the utf-8 flag, Perl needs to decide
> between the three cases mentioned above.
It doesn't do that. Every UTF8less string is treated the same.
> This is costly (scanning for hight bit characters to distiguish between 7bit
> ascii and 8bit "something else")
I'm not aware of Perl scanning for high bit characters in UTF8less
strings, or any performance loss caused by that.
> As an author who inherited software that deals with random binary data (e.g.
> JPEGs), this deficency concerns me.
I'm not aware of such a deficiency, and my Perl handles JPEG data just
fine as long as I don't let it touch unicode text data.
Dr.Ruud
> Juerd:
> Perl only supported bytes 0..255 in earlier versiosn, and
> now the perl byte can be up to 64 bits (or maybe a bit less, I
forgot).
Aren't you mixing up bytes and words here?
A byte is a unit of memory addressing. One or more bytes make a word. A
word is a unit of transporting memory data.
> In C, a single byte is a character, even if it happens to have a value
> higher than 255 (although very few compilers allow that, usually, a
> byte is an octet, although it is common on DSPs to have 32 bit bytes).
C chars don't have to be machine bytes. The DSPs you mention don't have
to have 32 bit machine bytes, to be used best with 32 bit C chars.
C has effectively 3 types of char: signed, unsigned and default.
--
Affijn, Ruud
"Gewoon is een tijger."
Ben Carter
> In C, a single byte is a character, even if it happens to have a value
> higher than 255 (although very few compilers allow that, usually, a byte
> is an octet, although it is common on DSPs to have 32 bit bytes).
>
> does not mean its more than a single character.
>
> The documentation is completely contradictory when it comes to "C" and can
> easily be interpreted to mean a single character in the C sense.
>
> Fact is "even under Unicode" it doesn't work as advertised, becasue Unicode
> can be internally represented in multiple ways in Perl.
>
> > I think that "char value" should be either removed from perlfunc, or
> > explained in more detail. It's NOT OBVIOUS to those who don't know C.
>
> To those who do know C it has perfectly clear meaning, namely a single
> character.
http://www.parashift.com/c++-faq-lite/intrinsic-types.html#faq-26.3
But that is not really relevant to the discussion.
Communication is difficult if you cannot express clearly what you are
trying to say. Terminology is important to get correct, and it is easy
to confuse others or yourself if you are not precise when you need to
be.
Unicode does not even HAVE characters, it has codepoints. This did not
happen by accident and is an important distinction to make.
$x = "ABCD";
$x = "\x41\x42\x43\x44";
$x = chr(65) . chr(66) . chr(67) . chr(68);
$x = pack("C*", 65, 66, 67, 68);
All of these put the same data into $x. [1] We can reasonably assume
that $x contains a sequence of 4 bytes, each 8 bits wide. We do not
know anything about what $x is, if it has an encoding, if it is actually
the output of pack "V", or maybe it came after "HTTP/1.1 GET ". The
only reasonable thing to assume is that it is just a sequence of octets,
aka binary data.
Now consider the case of
$y = chr(1000);
Clearly whatever is in $y cannot be a single octet. The way Perl
currently works (and this is my limited understanding here - someone
with more knowledge can feel free to step in and correct my errors)
is that now $y is considered to be a string of Unicode codepoints. So
$y contains a single codepoint, U+03E8. The internal flag is used to
indicate that the internal data pointer points to something that is a
"Unicode codepoint string".
What can we do with such a string? We can try to print it, but if we
have not converted it we get a message like
Wide character in print at - line 1.
and we get the bytes "cf a8" as output because that is the internal
encoding.
print unpack("H*", $y);
produces "cfa8" as output, again because we have been given access to
the string as it exists upgraded.
On the other hand,
print unpack("H*", pack("C", 1000));
produces "e8".
So consider again:
unpack("C*", $y);
This currently produces the list (207, 168) which is again the internal
encoding. What else should it do? If you expect values over 255, then
you should not use "C". If you don't have values over 255, then why is
your string not just a sequence of bytes? Something must have occurred
to upgrade it to "sequence of unicode codepoints".
Of course if you have values over 255 you have to use "U" in unpack,
that only makes sense! On the other hand, if you are agnostic to your
string and just treat it as "data" then it will never get upgraded. So
where is the issue?
It sounds to be that what you are trying to suggest is something along
the lines of another type of Sv for the case of "unicode codepoint
sequence", so that SvPV implicitly means "This scalar is not upgraded
and is just data" and SvP_UnicodeArrayValue_ would contain the upgraded
value. Then for anything that wanted a SvPV (XS code, unpack "C") the
only sensible thing would be to try to downgrade the string at that
point and then emit a warning in the case of "wide characters" being
present.
This is the point at which someone more familiar with internals chimes
in and says "This has problems [backwards compatibility, tuits, other]."
And of course this would preclude being able to inspect Perl's internal
Unicode representation using unpack "C". :)
--
-Ben Carter
Human beings, who are almost unique in having the ability to learn from
the experience of others, are also remarkable for their apparent
disinclination to do so. - Douglas Adams, "Last Chance to See"
[1] I am deliberately ignoring the box in the corner labeled "EBCDIC".
Abigail
>
> Now consider the case of
>
> $y = chr(1000);
>
> Clearly whatever is in $y cannot be a single octet. The way Perl
> currently works (and this is my limited understanding here - someone
> with more knowledge can feel free to step in and correct my errors)
> is that now $y is considered to be a string of Unicode codepoints. So
> $y contains a single codepoint, U+03E8. The internal flag is used to
> indicate that the internal data pointer points to something that is a
> "Unicode codepoint string".
No.
"ABCD" also contains 4 Unicode code points.
Perl strings only contain Unicode code points. Always.
The issue is not whether or not a string is a "Unicode" string or not, the
point is the *encoding* of the Unicode code points. That can be in UTF-8
(variable number of bytes/code point), or Latin-1 (one byte/character).
Unicode does not imply UTF-8.
Abigail
Tels
Hash: SHA1
Moin,
Especially since Perl itself doesn't have any way to distinguish "a"
(UNKNOWN ENCODING) from "a" (ASCII) from "a" (ISI-8859-1) from "a"
(UTF-8) - except one bit :)
All the best,
Tels
- --
Signed on Sat Mar 31 12:24:31 2007 with key 0x93B84C15.
Get one of my photo posters: http://bloodgate.com/posters
PGP key on http://bloodgate.com/tels.asc or per email.
"Most people, I think, don't even know what a rootkit is, so why should
they care about it?"
-- Thomas Hesse, President of Sony BMG's global digital business
division, 2005.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEUAwUBRg5TjXcLPEOTuEwVAQIrGAf417/05df4c3hIzTnFoidS3fAKWPHm9Ots
5BNa8n3PJci4cGQ2Sz7LzRf4BjD6+seW8Zq6fKNMIlCpmwCJYh/M+Ol8BBGefjhU
tJxebJs1O2K+ZEd9cJTP/PP2bnqg9Z1CwiBNn8xT/cT8tbF6rR9kujaHooSkHnPV
snDog7uLrk117tof8ORcybml0bDfhWzh4UfYOyue37RyrqAWnIXNOu24uYUjMiDT
US3vym0LX+LUO4aBS9Ur/tX6FSBX/5mXDn0fPR016ESbzWA6TMMurSIjWYLFTw9R
rRK0KSAb/z93Z6ZhHvyaKOz8Tt9ma44adu6WgTXrK5dcrpih8xbX
=Q94f
-----END PGP SIGNATURE-----
Tels
Hash: SHA1
Moin,
On Saturday 31 March 2007 10:03:12 Juerd Waalboer wrote:
> Tels skribis 2007-03-31 11:45 (+0000):
> > I should have said "random binary data" not "KOI8". "KOI8" implies the
> > data is some sort of text that can be "upgraded" to utf-8.
>
> Not if "upgrading" refers to the process that Perl has when it goes from
> latin1 to utf-8, because this doesn't handle arbitrary encodings like
> koi8r.
>
> Your best bet is to treat koi8r encoded data as binary data. (And as
> such, not mix it with text data.)
The "do not mix it" is the part where I am currently having problems with.
As far as I can see, there is nothing in Perl that prevents this from
happening, nor can I enable a warning when it happens. All you get is at
some point corrupted data, or very inefficient code (since Perl internally
uses UTF-8 while it could use just the raw bytes).
> If this is confusing, you may want to
> gzip it first, and ungzip it afterwards ;)
It is not confusing to me, but gzip wouldn't actually help when Perl
helpfully upgrades the gzippd data to utf-8 :)
I know what you mean, but the problem is that you are also proposing that
the UTF-8 flag should be hidden from the user. So, how can I "not access
the UTF-8 encoded" buffer when I don't know if the buffer I access is UTF-8
or not?
I think this is also the problem Marc is having with your POV. You can't
hide the internal encoding from the user, then telling him "do not mix
these two different things even tho you do not know which one is which".
That's a bit, er, unrealistic.
> With the bit off, it's either unicode data that internally is encoded as
> ISO-8859-1, or it is binary data. This string can safely be used for
> octet operations (but of course, that doesn't make sense if the sting
> was intended as text, with the exception of some ancient 8bit things
> crypt()).
>
> > So whenever you have data without the utf-8 flag, Perl needs to decide
> > between the three cases mentioned above.
>
> It doesn't do that. Every UTF8less string is treated the same.
And that is in efficient :)
> > This is costly (scanning for hight bit characters to distiguish between
> > 7bit ascii and 8bit "something else")
>
> I'm not aware of Perl scanning for high bit characters in UTF8less
> strings, or any performance loss caused by that.
use Benchmark;
use Encode qw/decode/;
my $a = 'a' x 100_000_000; # 7bit utf-8 off
my $b = 'b' x 100_000_000; # 7bit utf-8 off
my $c = 'c' x 100_000_000; # 7bit utf-8 flag on
$c = decode('ISO-8859-1', $c);
timethese (-3, {
'a eq b' => sub { $a eq $b; },
'a eq c' => sub { $a eq $c; },
} );
Benchmark: running a eq b, a eq c for at least 3 CPU seconds...
a eq b: 4s (4.72 usr + -0.02 sys = 4.70 CPU) @7218655.96/s (n=33927683)
a eq c: 3s (2.80 usr + 0.46 sys = 3.26 CPU) @ 2.76/s (n=9)
I rest my case. :)
All the best,
Tels
- --
Signed on Sat Mar 31 12:28:15 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
"Blogebrity: Wow, guess what this one stands for? Too easy. Hey, anyone
can do it: take a blogger who's a chef, and you get: BLEF. A blogger
who's a dentist? BENTIST. A female blogger with an itch? You guessed it:
a BITCH."
-- maddox from xmission
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg5Wv3cLPEOTuEwVAQKVMQf9G1RLUfo+fY+H8dn4Qa+ggbL/IRnOz3wi
sR4KAw32xrCvHPZYkQRPm1xVJiDwpMDgEgdVSEo6Ot9qA3TLXGadF4F9PMzPQRWM
4509df7yoEulvKsKNiqHFJSbxO8KlVaX4CO8Zr/8aCnM4IIajBuISRQUtLARRl/d
VQacgTOJwHCkaRqB8T+9kdP3U9OV72xXoYDHRXRbJOiav7QVGmmVib5M2ZQWj5zv
H8r1daSG7mFg3qCUE/KKYLAC2hmMMvC31zhMzWveAxlFE5hWg+EyYFzxbPk9sisT
69seb4XaXXrpM/jn7C3Gq2GKeEggeRDrAhw3DvlPrO0r1VZYvmFDwQ==
=YMp0
-----END PGP SIGNATURE-----
Tels
Hash: SHA1
Moin,
On Saturday 31 March 2007 00:33:55 Juerd Waalboer wrote:
> Tels skribis 2007-03-31 1:39 (+0000):
> > My question was posed because I wanted to know how to *keep* a KOI8 (or
> > any other random binary) string in Perl without converting it to
> > Unicode. It seems to me this is not easily possible because there are
> > literally dozend places where your KOI8 string might get suddenly
> > upgraded to UTF-8 (and thus get corrupted because Perl treats it is
> > ISO-8859-1). Or did I get this wrong?
>
> A koi8r string is a byte string. If you keep it separated from text
> strings properly, it should not be upgraded and thus treated as latin1.
> I'm very curious as to "sudden upgrades" that aren't related to mixing
> with text strings. Should you encounter them, please let me know.
"Keeping things seperate" is not working in the Real World[tm]. As far as I
can see so:
#!/usr/bin/perl -w
use Encode qw/decode/;
my $random = "\xc3\xc3"; # some random bytes
my $ascii = "a"; # some 7bit data
# Somebody "helpfull" decodes the ascii string:
# The encoding doesn't actually matter, since it is 7bit anyway.
# This step happens out of my control (e.g. in third party code)
$string = decode('ISO-8859-1', $ascii);
# now take our random binary data and a 7bit ascii string and do:
print join (" ", unpack("CCC", "$random$string")), "\n";
print join (" ", unpack("CCC", "$random$ascii")), "\n";
Now explain to me why this prints different things even tho $random is the
same string in both cases, and $string and $ascii should be the same,
too. :) Bonus points if you manage to not mention the uhh -- ut - utf --
uhm -- er The Flag[tm].
So far, I can see the ways to handle this are:
* replace C with U (lots of code review work, plus it still means you
200Mbyte TIFF file might make a trip to UTF-8 land and back)
* always forcefully downgrade stuff in 7bit ASCII (wastefull) and just hope
your 8bit data never get's in contact with anything with The Flag[tm]
* never mix fire and water er dogs and cats er I mean text and bytes, and
pray that every piece of code out there to adheres to this, too.
I think the Pray and Hope[tm] strategy doesn't really work, tho.
All the best,
Tels
- --
Signed on Sat Mar 31 12:09:53 2007 with key 0x93B84C15.
Get one of my photo posters: http://bloodgate.com/posters
PGP key on http://bloodgate.com/tels.asc or per email.
"Sundials don't work, the one I've had in my basement hasn't changed
time since I installed it." grub (11606) on 2004-12-03 on /.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg5Sz3cLPEOTuEwVAQJvegf+OVl0Ha2tJ3QIXmkUs+XHXWdYIqtu9xJe
VeBwrelub65lfgIfD8FnNmft+KgZDE8S8QU3sjFo5NArtVT56tFsAeIwtdtC23au
BcobxZxkI9iHWJtkJYlxKHEdSPbWSgJiWfJ7J3fc4zprme3/Zlxgpcd3pyiRee0m
AhpnZ6dui033dNakhZCHu1L/YeUyP72OmGmtWOAJLHGIQ/w0nUrUJrx5kg3WuV88
ATfl7EFVZOxqavSSWJCgBHXvU8iRUg4mmqpoVPY4S9uqMi9IYCZBPZNAc++MSjbn
b0e8+qPTB43zah6EfNSc5Xq22EDEjx7mu0n62FQhajV1lOIoc0kV7g==
=CfKu
-----END PGP SIGNATURE-----
Dr.Ruud
> unicode explicitly is a superset of latin1.
But there are defined differences:
$ perl -wle '(chr().chr(255)) =~ /^\s/ and $n++ and print for 0..255;
print "[$n]"'
10
12
13
32
[5]
$ perl -wle '(chr().chr(256)) =~ /^\s/ and $n++ and print for 0..255;
print "[$n]"'
10
12
13
32
133
160
[7]
perl -wle '(chr().chr(255)) =~ /^\w/ and $n++ and print for 0..255;
print "[$n]"'
...
[63]
perl -wle '(chr().chr(256)) =~ /^\w/ and $n++ and print for 0..255;
print "[$n]"'
...
[134]
Tels
Hash: SHA1
Moin,
On Saturday 31 March 2007 12:00:55 Dr.Ruud wrote:
> Marc Lehmann schreef:
> > unicode explicitly is a superset of latin1.
>
> But there are defined differences:
Uh. The silently upgrading a latin1 string to utf-8 is even more worrysome
as it wouldn't be transparent as far as the regexp go. Is this a bug in the
regexp engine that can be fixed?
All the best,
Tels
- --
Signed on Sat Mar 31 15:29:46 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
Mediawiki graph-extension: http://bloodgate.com/perl/graph/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg5+v3cLPEOTuEwVAQISDQf+Iqm25gexSqhjhFXAwrgwzA3Aq7nAUE9K
z4vMGSoJ8Rv9WNofa5ENsiH0vnU96hhT9l2HQHZVOF8VeCRf8biYdI+IaQ3joDhU
q5uTfWTtAAWs+STTBjV4hnxeTwG5LkMM4HOddj3ZuPvjM2CdXIDpooNSiOrRZLZi
8bvsEcCF9ZI7ib24DjCmR+fwPJ+yBBbm34Gyjf3iVnrDQY8Uc2swZbTf+yhDHCSP
kY/r1fZRS+eBYxqe28XwwI9paoi0Is3f9MCMwIxWwa9DmFunonr4YEve3jMC5ZQo
XDRE1LNYOVB8H1wIblcYzXgGCv3vbVj/52ySNVP0aOEcd8IL2eO62g==
=XfDd
-----END PGP SIGNATURE-----
Juerd Waalboer
> > unicode explicitly is a superset of latin1.
> But there are defined differences:
> $ perl -wle '(chr().chr(255)) =~ /^\s/ and $n++ and print for 0..255;
That's a bug; see also the subthread "fixing the regex engine wrt
unicode" for an overview of why it cannot be fixed, and how it could be
:)
Juerd Waalboer
> The "do not mix it" is the part where I am currently having problems with.
> As far as I can see, there is nothing in Perl that prevents this from
> happening, nor can I enable a warning when it happens.
This is true, but no different from other things that you should keep
track of yourself. Some operations can change the type of a variable,
not just inside, but also conceptually.
* references
$ref++, and it's no longer a ref.
* strings
$string++, and it's no longer a string.
* numbers
"x" on a number very rarely makes sense.
Though this is all visible in your code, because there are different
operators, and they are known to force their type upon the values
(simplified explanation).
Text strings and byte strings share a single type, but also a single set
of operators. Indeed, that makes it harder to cope with keeping them
apart.
Some people may like a hungarian notation for it.
> All you get is at some point corrupted data, or very inefficient code
> (since Perl internally uses UTF-8 while it could use just the raw
> bytes).
If you accidentally mix them, yes. But if you don't, the byte string
won't be upgraded to utf8 (when it is, that is probably a bug that
should be fixed), and your bytestring just lives on exactly like it
would have in Perl 5.005, or 4, or perhaps 1 even.
> It is not confusing to me, but gzip wouldn't actually help when Perl
> helpfully upgrades the gzippd data to utf-8 :)
Perl is helpful when it sees you're using the string as a text string.
It them assumes that it had been latin1 all the time.
It would be useful to have magic on a string that enforced
non-upgrading, but only for strings that you want it on.
This would be the bondage part, for when discipline was broken.
> I know what you mean, but the problem is that you are also proposing that
> the UTF-8 flag should be hidden from the user. So, how can I "not access
> the UTF-8 encoded" buffer when I don't know if the buffer I access is UTF-8
> or not?
Accessing the buffer directly is something that byte operators do, e.g.
vec and unpack("C"). If you never mix your byte strings with text
strings, and use these operators only with byte strings, you can be sure
that the variables won't be UTF8 internally.
Note that if you refactor this guideline, the "UTF8" part disappears.
> > > This is costly (scanning for hight bit characters to distiguish between
> > > 7bit ascii and 8bit "something else")
> > I'm not aware of Perl scanning for high bit characters in UTF8less
> > strings, or any performance loss caused by that.
> use Benchmark;
> use Encode qw/decode/;
> my $a = 'a' x 100_000_000; # 7bit utf-8 off
> my $b = 'b' x 100_000_000; # 7bit utf-8 off
> my $c = 'c' x 100_000_000; # 7bit utf-8 flag on
> $c = decode('ISO-8859-1', $c);
> timethese (-3, {
> 'a eq b' => sub { $a eq $b; },
> 'a eq c' => sub { $a eq $c; },
> } );
> Benchmark: running a eq b, a eq c for at least 3 CPU seconds...
> a eq b: 4s (4.72 usr + -0.02 sys = 4.70 CPU) @7218655.96/s (n=33927683)
> a eq c: 3s (2.80 usr + 0.46 sys = 3.26 CPU) @ 2.76/s (n=9)
Ah, good to know there are more people who don't mind using 100 MB
strings.
I thought you meant implicit scanning, i.e. not caused by manual
decoding, or automatic upgrading.
decode might optimize latin1 or ascii some day. The documentation
already claims that it does that, but it doesn't.
When optimizing, knowledge of the internals can help a great deal. I
stress that you don't need this knowledge for a working program, and
that working with 100 MB strings and then comparing them in a tight loop
is not common. But anyway, a nice optimization is to do utf8::downgrade
on a string that you just decoded from latin1. Then you pay only a
one-time price. Depending on your data, however, a better optimization
may be to utf8::upgrade the other two.
Juerd Waalboer
> #!/usr/bin/perl -w
> use Encode qw/decode/;
> my $random = "\xc3\xc3"; # some random bytes
> my $ascii = "a"; # some 7bit data
>
> # Somebody "helpfull" decodes the ascii string:
> # The encoding doesn't actually matter, since it is 7bit anyway.
> # This step happens out of my control (e.g. in third party code)
> $string = decode('ISO-8859-1', $ascii);
$string is a text string, now. Remember, decoding is going from byte
string to text string.
Using unpack "C" on a text string makes no sense if you consider that
this "C" doesn't stand for "character" in the sense that the
documentation for chr, ord, length, split, etcetera use. It stands for
"char", which is a C datatype that contains one byte.
As such, unpack "C" is a byte operation and makes sense on byte strings
only. $string is a text string, and you can tell by looking at the
decode() step.
> # now take our random binary data and a 7bit ascii string and do:
> print join (" ", unpack("CCC", "$random$string")), "\n";
Dangerous, and that's why I suggested adding a "wide character in..."
warning earlier in this thread.
> Now explain to me why this prints different things even tho $random is the
> same string in both cases, and $string and $ascii should be the same,
> too. :) Bonus points if you manage to not mention the uhh -- ut - utf --
> uhm -- er The Flag[tm].
I get the bonus points! Hurrah! :)
The only explanation that I used is the separation between text strings
and binary strings. It's also the only thing you need to know. You'll
benefit from knowing more, certainly, but I see red flags in your code.
> So far, I can see the ways to handle this are:
> (..)
> * never mix fire and water er dogs and cats er I mean text and bytes, and
> pray that every piece of code out there to adheres to this, too.
Exactly.
> I think the Pray and Hope[tm] strategy doesn't really work, tho.
It doesn't always work, because people can't be trusted to do the right
thing, but it can always be fixed.
Juerd Waalboer
> Unicode does not even HAVE characters, it has codepoints.
Very good point, but Perl's documentation refers to codepoints as
"characters", and does that rather consistently.
I'm considering sweeping through the docs and changing it all, but it
would be a lot of work and a huge patch. I wonder if it's worth that.
> Now consider the case of
> $y = chr(1000);
> Clearly whatever is in $y cannot be a single octet. The way Perl
> currently works is that now $y is considered to be a string of Unicode
> codepoints.
Yes.
But to go into a bit more detail for the more interesting case of
chr(233): this is either a byte string with only one byte, or a text
string with only one cha^Wcodepoint. Perl doesn't know, or care, so the
programmer has to.
> So $y contains a single codepoint, U+03E8. The internal flag is used
> to indicate that the internal data pointer points to something that is
> a "Unicode codepoint string".
No, see Abigail's response for clarification.
> print unpack("H*", pack("C", 1000));
Feeding 1000 to C has undefined behaviour: the C type can only handle
values 0..255, and there's no documentation defining what happens if you
feed it something <0 or >255. A similar thing occurs with floating point
numbers, like 64.5. The current implementation truncates that to 64,
without warning.
> If you expect values over 255, then you should not use "C".
Indeed!
> Of course if you have values over 255 you have to use "U" in unpack,
> that only makes sense!
If these values are codepoints, yes. But if they're just numbers, other
unpack templates, like perhaps N or V are better.
> [1] I am deliberately ignoring the box in the corner labeled "EBCDIC".
Oh, so am I. In fact, I've probably never even seen such a box in my
short life so far.
Tels
Hash: SHA1
Moin,
On Saturday 31 March 2007 16:09:18 Juerd Waalboer wrote:
> Tels skribis 2007-03-31 12:23 (+0000):
> > #!/usr/bin/perl -w
> > use Encode qw/decode/;
> > my $random = "\xc3\xc3"; # some random bytes
> > my $ascii = "a"; # some 7bit data
> >
> > # Somebody "helpfull" decodes the ascii string:
> > # The encoding doesn't actually matter, since it is 7bit anyway.
> > # This step happens out of my control (e.g. in third party code)
> > $string = decode('ISO-8859-1', $ascii);
>
> $string is a text string, now. Remember, decoding is going from byte
> string to text string.
Yes, but my point was that I:
* might not be the one who "decoded" $string or produced it even.
* do not know if I am passed a "text" string as there is only the
flag-you-should-not-know-about to distinguish these two.
> Using unpack "C" on a text string makes no sense if you consider that
> this "C" doesn't stand for "character" in the sense that the
> documentation for chr, ord, length, split, etcetera use. It stands for
> "char", which is a C datatype that contains one byte.
>
> As such, unpack "C" is a byte operation and makes sense on byte strings
> only. $string is a text string, and you can tell by looking at the
> decode() step.
>
> > # now take our random binary data and a 7bit ascii string and do:
> > print join (" ", unpack("CCC", "$random$string")), "\n";
>
> Dangerous, and that's why I suggested adding a "wide character in..."
> warning earlier in this thread.
>
> > Now explain to me why this prints different things even tho $random is
> > the same string in both cases, and $string and $ascii should be the
> > same, too. :) Bonus points if you manage to not mention the uhh -- ut -
> > utf -- uhm -- er The Flag[tm].
>
> I get the bonus points! Hurrah! :)
Not really, as you didn't explain the difference, you merely told me "there
is a difference" (where me personally don't expect to be a difference)
> The only explanation that I used is the separation between text strings
> and binary strings. It's also the only thing you need to know. You'll
> benefit from knowing more, certainly, but I see red flags in your code.
Ok, and how am I supposed know that in:
sub dosomething {
my $a = shift;
}
$a is a text string or a binary string? :)
> > So far, I can see the ways to handle this are:
> > (..)
> > * never mix fire and water er dogs and cats er I mean text and bytes,
> > and pray that every piece of code out there to adheres to this, too.
>
> Exactly.
This is not a working strategy.
> > I think the Pray and Hope[tm] strategy doesn't really work, tho.
>
> It doesn't always work, because people can't be trusted to do the right
> thing, but it can always be fixed.
Only if you consider your own code. But data is sometimes processed by other
code (Perl itself, some module etc.).
All the best,
Tels
- --
Signed on Sat Mar 31 18:33:51 2007 with key 0x93B84C15.
Get one of my photo posters: http://bloodgate.com/posters
PGP key on http://bloodgate.com/tels.asc or per email.
"We're looking at a future where only the very largest companies will be
able to implement software, and it will technically be illegal for other
people to do so."
-- Bruce Perens, 2004-01-23
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRg6qqXcLPEOTuEwVAQINCAf/QWq653liE6ZUnR5sUrO8YFVXU0Gi5s/m
wm4teby4dypHRuyjKov7a2XeheRCZU+iYXnlNFk8Tioqd3ZOwlZC5uGbufX1QnpO
H9lYRtDTG14BHH2D+QsMgSrPcAXwsnvSdlePAmy4m9TJ3xQTtzcPLTWt2p8tgiul
URl0lgMHv7I9ASJusYwPa00YRFDexpdVuYpclTtnzzVPoGkuMxAKIDhhAuKp9uSl
gWJXGiha9hvGEZOh2k6mGZ/bkstEMhp3vrqU1ccp11jfahsaAwvU9EVS7254t22R
KqXh3Ca4/lMxs+2+1xW0j518Asq0sB/L6gkyGr0tHdFgQwX7S71yoA==
=K82l
-----END PGP SIGNATURE-----
Juerd Waalboer
> * might not be the one who "decoded" $string or produced it even.
> * do not know if I am passed a "text" string as there is only the
> flag-you-should-not-know-about to distinguish these two.
> Ok, and how am I supposed know that in:
> sub dosomething {my $a = shift; }
> $a is a text string or a binary string? :)
No, not even the flag-you-should-not-know-about doesn't distinguish
between the two.
When you're writing a library function to handle arbitrary data, you'll
have to pick sides, either text or binary. Fortunately, the choice is
often very simple.
When you can't choose between these two, you could write two functions:
one for text data, one for binary data. Often you can write the text
function simply by using the binary thing underneath, with a specified
UTF encoding.
If you're just serializing data, you could opt for storing the literal
internal buffer along with the state of the UTF8 flag, or (exactly like
the previous paragraph) pick any specific encoding and stick to that.
If you happen to have a function in a current API (i.e. not a contrived
one) for which you find it hard to decide, please let me know the
details. I'll help you offlist.
> Only if you consider your own code. But data is sometimes processed by other
> code (Perl itself, some module etc.).
Yes, indeed. This can be troublesome. Especially many, many modules
still don't correctly support Unicode. I'm slowly but surely compiling a
list at http://juerd.nl/perluniadvice. Wanna help?
Juerd Waalboer
> Juerd says two, but describes 3 types of data.
Well observed! :)
And in general, your post is a good summary of how things work.
> Juerd says two, but describes 3 types of data.
Binary and latin1 are hybrid in Perl. Only the programmer knows (can
know) the difference, Perl doesn't (can't). That's why to Perl, there
are only 2.
> As a developer that hopes to be implementing Unicode character support
> in a perl application soon (tuits, always tuits), I have the following
> questions and comments.
I invite you to read perlunitut and perlunifaq. You'll have to look for
them (e.g. Google) or get them from bleadperl.
> 1) What operations can safely be used on bytes stored in a string
> without causing implicit upgrades to multi-bytes?
All operations are safe, except:
1. operations that add characters greater than 255
2. joining text strings with byte strings (because
the text string may already internally be prepared for handling
characters greater than 255, and forces the byte string to be prepared
in a similar way, breaking its binaryness) and byte operations (because
they cannot handle characters greater than 255).
If you read carefully, you'll notice that "1" is just a different way of
having "2", and come to the conclusion that there's only one simple, yet
important, guideline: keep binary and text separate, and only bridge
between the two by means of decoding and encoding.
> My perception from following all this discussion is that you can do any
> operation, as long as all the data involved is bytes data that has never
> been upgraded, except for decode, which always assumes a bytes parameter.
Your perception is correct.
> 2) What operations create multi-bytes data?
1. operations that add characters greater than 255
2. joining text strings with byte strings or byte operations, if the
text string is internally prepared for handling characters greater than
255 ("is internally encoded as UTF8", "has the UTF8 flag set").
This is, in essence, the same list as before :)
> 3) What operations create bytes data?
In general, everything that creates a new string value with only
characters in the 0..255 range, with the possible exception of
operations that are designed to create text strings only (like decode).
Some examples:
"\xdf\xff\xa0\x00\xa1" # binary (but can be used as text)
"\x{df}\x{ff}\x{a0}\x{00}\x{a1}" # binary (but can be used as text)
"\x{100}..." # text (should not be used as bin)
chr 255 # binary (but can be used as text)
chr 256 # text (should not be used as bin)
readline $fh # either, depends on io layers
Note that "use encoding" will turn almost everything into a text-only
creating thing, and makes using binary data very hard. My advise is to
avoid the module altogether.
> 4) What operations implicitly upgrade data from binary, assuming that
> because of context it must be ISO-8859-1 encoded data?
In the core, only concatenating with an internally-UTF8 string. All
other operations that require UTF8 only upgrade temporarily.
There are modules that carelessly upgrade strings; this causes no
problem if your string is a text string and you decode/encode properly
and keep the string properly separated from byte strings. But it might
otherwise.
> My perception is _any operation_ that also includes another operand that
> is UTF8 already.
When the "other operand" becomes part of the target string at some
point, yes.
> 5) It seems that there should be documented lists of operations and core
> modules that a) never upgrade b) never downgrade c) always upgrade d)
> always downgrade e) may upgrade f) may downgrade and the conditions
> under which it may happen.
Instead of compiling lists, isn't adding this information to the
existing documentation a better idea?
Lists like this would suggest that you'd have to learn them by heart in
order to write safe code, while in fact it's only needed to keep text
from byte values and byte operations. The latter is a lot easier to
learn and universally applicable.
> Juerd's forthcoming perlunitut document seems to imply that the rules
> are indeed common to all operations, but this discussion seems to
> indicate that there might be a few exceptions to that...
It dose seem to indicate so, but I've yet to see proof of these other
supposed implicit upgrades...
> A) pack -- it is not clear to me how this operation could produce
> anything except bytes for the packed buffer parameter, regardless of
> other parameters supplied.
pack "U" works like chr. I'd strongly advise against using U with other
letters, because U makes text strings, and the other letters are byte
operations (so using them together would break the text/byte rule).
pack "U*", LIST is useful because it is more convenient than writing
join "", map chr, LIST.
> B) unpack -- it is not clear to me how this operation could successfully
> process a multi-bytes buffer parameter, except by first downgrading it,
> if it contains no values > 255, since all the operations on it are
> defined in terms of unpacking bytes.
Indeed. Even downgrading is questionable, because its operand should
never have been upgraded in the first place.
Binary strings don't have any encoding, as far as Perl is concerned.
When it gets a string of which it is certain that it does have an
encoding, it can't possibly be binary.
non-U unpacking might as well return undef or random values, when it
gets a string that has the UTF8 flag set :)
Again, here's the special case for U, that works like "ord", but again
supports lists in a nicer way. And with unpack too, I think it's wrong
to mix U with other letters.
> C) use bytes; -- clearly this impacts lots of other operations.
I advise against "use bytes", and "use encodings".
> D) Data::Dumper -- someone made the claim that Data::Dumper simply
> ignores the UTF-8 flag, and functions properly. Could someone elucidate
> how that happens?
It's because the text/byte distinction exists in a programmer's mind,
not in the string value. There is a latin1/utf8 distinction in the
string value, internally, but the representation of the codepoints
doesn't change the value of the codepoints, so effectively, even with a
different internal encoding, you maintain the same string value.
If you use $Data::Dumper::Useqq, Dumper uses \ notation for non-ASCII,
so dumped binary strings even survive encoding layers. (Okay, they
should be ASCII compatible, but most are).
Without Useqq, D::D works fine on unicode strings and binary strings,
but your binary strings might be re-encoded in an inconvenient way.
> G) regular expressions -- lots of reference is made to regular
> expressions being broken, or at least different, for multi-byte stuff.
> I fail to see why regular expressions are so hard to deal with.
I guess that it is not particularly hard to deal with the bug, because
both kinds of semantics are already present. It's dealing with all the
code out in the wild that depends on the current buggy semantics that is
hard. To remain backwards compatible, new syntax has to be introduced
(but may be implied with "use 5.10", for example). See my thread "fixing
the regex engine wrt unicode".
> Firstly, regular expressions deal in "characters", not bytes, or
> multi-byte sequences.
Some people like to use regular expressions on their binary data, and I
think they should be able to keep doing it. Of course, things like /i or
the predefined character classes don't make sense there, and those are
the broken things.
Dr.Ruud
> $ perl -wle '(chr().chr(255)) =~ /^\s/ and $n++ and print for 0..255;
s/\$n\+\+/++\$n/
(to get the line for tab printed, thx for whispering)
Juerd Waalboer
> You sidestepped comment on what the range is for data stored in
> multi-bytes format, though... is it [0:2^30-1] or [0:2^32-1], or what?
I didn't know, but used the following to find out by brute force:
juerd@lanova:~$ perl -le'$a = 1; while ($a *= 2) { ord(chr $a) == $a or die $a }'
4294967296 at -e line 1.
juerd@lanova:~$ perl -le'print ord(chr(4294967295))'
4294967295
So 32 bits, or in that cryptic format of yours: [0:2^32-1] ;)
> I've read perlunifaq several times trying to figure things out. I read
> perlunitut yesterday, when it came up in this discussion.
Great!
> I found perlunifaq quite opaque the first several times I read it.
It's meant to be read after perlunitut, where the basics are outlined.
The FAQ assumes basic knowledge.
(In an earlier version, they were both in one document.)
> perlunitut seems easier to follow, but didn't answer all my questions
> either.
The questions that you have asked here may be useful additions to
perlunifaq.
> Again, you say two, but describe 3.... :) Maybe that is a habit of yours?
You're on to me!
> 1. operations that add characters greater than 255
> 2. joining text strings with byte strings
> 3. byte operations
It's actually still just 1, not 2 or 3: "operations that add characters
greater than 255" is equal to "joining text strings with byte strings",
because anything with characters >255 is a definitely text string.
"joining text strings with byte strings" is again equal to "using text
strings in byte operations", because concatenating with a byte string
can be seen as a byte operation.
> [perlfunc/"pack"]
All of the documentation for pack is, in my humble opinion, a canditate
for a rewrite.
I didn't touch pack's documentation when I updated unicode documentation
recently, because I never thought there would be someone convinced that
Perl should treat entire multibyte characters as single bytes/octets.
But, it appears now that bleadperl does do that for other pack template
letters, just not for "C". I think this change is a bad one and should be
reversed, but if it's not reversed, then the special case for "C" is
indeed bad and should be removed.
Personally, I think it's better to warn when unpack is used on a
multibyte string, and promise no specific return value.
An alternative would be the possibility for having a string type that is
explicitly only for byte buffers.
Let's call that, hypothetically, a "blob". There could be a "blob"
operator that adds this protecting magic to an existing string.
You could say: "blob my $foo". Sounds dwimmy enough.
Of course, it's attractive to add an optional parameter for the encoding
of the binary string. All non-blobs added to the string would be
automatically encoded. This doesn't reliably work in the other
direction, because the bytes may not be part of the text. An
encodingless blob can't ever have UTF8 things added to it, but doesn't
mind accepting latin1 (UTF8less) data.
This all isn't needed for writing stable programs, but it would help
those who feel insecure without a tool to enforce separation.
> I could imagine something like the following being invented as a
> communication protocol...
> $x = $text_string . "\0" . pack( "template", @params);
No.
A communication protocol needs to define the required byte encoding for
text. There are many encodings, and defaulting to any of them, without
making that part of your specification, is a huge mistake. You think
UTF-8 is standard? Guess again, because many applications use UTF-16 or
UCS-2 instead. And of course, there are still many things that don't
even handle unicode.
You must encode text strings, and you can safely use the result of that
in a byte string or stream. The world doesn't quite DWYM enough to guess
what you meant.
(You already found out, and replied to your own post, but I felt it
would be good if I responded here anyway.)
> # send somewhere, that does the following
> ( $retrieve_text_string, $unpack_me ) = split( "\0", $x );
> @retrieve_params = unpack( "template", $unpack_me );
And then decode.
> It seems that would violate your recommendation of keeping things
> separate, but one needs to avoid two separate communications, for
> efficiency, eh?
Absolutely!
> So do you have a recommended practice for this sort of action?
decoding and encoding.
Remember, "decoding" converts byte strings to text strings, "encoding"
converts text strings to byte strings. Whenever you do such a
conversion, you need to know which byte encoding was used, or is to be
used on the byte side.
It's okay to use text strings together, and it's okay to use byte
strings together. Just don't mix text strings and byte strings within
the same filehandle or concatenating operator.
And realise that only you, the programmer, can know the difference. Perl
can't help you here.
> Not sure how it would return undef??? The values are hardly random,
> either, they come out of the buffer it is handed, right?
Current stable Perl (5.8.8) uses the internal buffer directly.
Current bleadperl (5.9.5 to be) uses the codepoints, except for the "C"
letter.
I believe that neither is "right", because it simply does not make any
sense to do byte packing or unpacking on text strings. Hence: a warning,
and then return any value you like. I would prefer if the internal
buffer was used, just like in current stable Perl, because otherwise
existing deficient code may start falling apart. But I'm okay with
codepoints (and indeed, then "C" should probably be changed to also use
codepoints), or undef, or random values.
> But if a multi-bytes string is passed to a bytes-expecting template,
> then clearly it won't produce the results you expect...
Exactly.
> OK. What do you recommend when needing to store a UTF-8 string into a
> struct?
My Perl language doesn't have structs. I store strings in scalars, and
those handle both byte data (which could be UTF-8 encoded text) and
unicode data just fine.
> so it truly wants to stay bytes. But I want to store UTF-8 encoded
> strings into it.
Then add UTF-8 encoded strings to it! That's okay, because *encoded*
strings are byte strings.
my $encoded_string = encode("UTF-8", $text_string);
$byte_string .= $encoded_string; # ok!
You can use "utf8" (no hyphen) instead of "utf-8" (with hyphen) if you
don't care about codepoints that don't exist in Unicode (yet), or
"utf-8" (with hyphen) if you want strictness. For utf8 (no hyphen),
there is the shortcut function encode_utf8.
> Seems like "use bytes;" is a perfect match for the operations that
> work on the simulated memory.
> Maybe this would be a place where you would agree to make an exception
> to your above advice?
No. Making an exception here will only hurt in the long run, because the
internal byte buffer that bytes:: accesses may change encoding over
time, or because of its contents.
> So you are saying that Data::Dumper treats strings as text, whether they
> are text or binary.
No, it uses strings and really doesn't know or care if 8 bit strings are
internally-latin1 text strings, or byte strings. However, if you pull
Dumper's output through an :encoding layer, or through encode(), the
bytes will be assumed to be latin1 text.
Useqq avoids this by outputting the bytes as escapes rather than literal
bytes.
> The problem is that there are two (<- :) *) kinds of data that regexp's
> can operate on:
> 1) Unicode multi-byte
> 1) ASCII byte
> 1) ASCII multi-byte
> 2) Latin-1 byte
> 1) Latin-1 multi-byte
It's a bit different.
Regexes, or actually, case independency and predefined character
classes, work on characters (note: that's Perl jargon for
"codepoints"!).
Codepoints stay the same, regardless of internal re-encoding. The
semantics should also stay the same.
But they don't. Characters in the non-ascii latin1 range are treated
differently, based on their internal encoding. That sucks, because the
programmer doesn't know the internal encoding, and because the internal
encoding depends on the history of the string.
The easiest examples are \s and \w.
\s matches space, form feed, tab, newline, and carriage return. Except
when the internal encoding happens to be UTF8. Then, it also matches non
breaking space (0xA0).
\w matches A-Z, a-z, 0-9, and underscore. Except when the internal
encoding happens to be UTF8. Then, it also matches accented word
characters like ΓΏ, Γ, Γͺ, Γ±, and Γ, and word characters like ΓΎ, Γ¦ and Γ°.
Because of backwards compatibility, it cannot be fixed without adding
new syntax. These semantics can be described as "ASCII mode" and
"Unicode mode". That's where the suggested flags /a and /u come from.
Note that not all predefined character classes work like this. \p{}, for
example, always uses unicode semantics.
> * That's 5 kinds of data
The world (not just Perl!) has two kinds of string data, byte strings
and text strings.
Perl has two kinds of string representation: 8 bit octets, and utf8
There is no direct mapping between them, but an overlap.
Your data is: binary data text data
Perl uses: 8 bit 8 bit or utf8.
or, the other way around:
Perl uses: 8 bit utf8
Your data is: binary or text text
Whenever you notice that your byte string got the UTF8 flag somehow, you
found a bug in your code (you didn't properly separate text from binary)
or you found a bug in perl (or a module you used).
Note that Perl has no special treatment for ASCII data! I just call the
"pre-unicode" regex semantics "ASCII mode" because the character classes
only match ASCII with it.
> So the "unicode regexp" problem is really a "Latin-1 bytes regexp"
> problem? Yes, your /u feature would seem to cure that, then, if that
> is the only problem.
Basically, but it would be nice to have /a too, because the old ASCII \w
was so incredibly widely used, that even with unicode text data, you may
still want to match it. I have, for example, used it for security
reasons: \w was the whitelist for characters in page names. I have now
replaced it with [A-Za-z0-9_] explicitly, because I have the page name
itself is a text string and for security reasons I don't *want to*
support other characters.
Juerd Waalboer
> I read perlunifaq again... so one can "encode_utf8", which presumably
> causes a binary byte stream that contains the UTF-8 style of octets
> sequences.
> $x = encode_utf8( $text_string ) . "\0" . pack( "template", @params);
> $retrieve_text_string = decode_utf8( $retrieve_text_string );
Yes!
> And so encode_utf8 is probably the solution to my question just above,
> allowing "length in bytes" to be obtained for a UTF-8 encoded string.
> And pulling it back out, one would decode_utf8 it...
Absolutely!
Demerphq
> A) pack -- it is not clear to me how this operation could produce
> anything except bytes for the packed buffer parameter, regardless of
> other parameters supplied.
This sounds reasonable to me.
> B) unpack -- it is not clear to me how this operation could successfully
> process a multi-bytes buffer parameter, except by first downgrading it,
> if it contains no values > 255, since all the operations on it are
> defined in terms of unpacking bytes.
This is sortof what happens with pack *in blead*. Except that instead
of downgrading it treats codepoints as being bytes by doing mod 256 on
their values. This makes a certain amount of sense if you assume that
strings can (apparenly) randomly change from octect encoding to utf8
encoding. For instance:
my $s=pack 'N',12345678;
$s.=chr(256); # upgrade $s to utf8 by catting on a unicode codepoint
chop $s; # lose the catted codepoint, encoding remains utf8
print unpack 'N',$s; # prints 12345678
So 'N' works with codepoints, not with bytes. Apparently this holds
true for most of the pack template formats. HOWEVER, it doesnt apply
to the pattern 'C' (and if i understand his recent posts this is what
Marc was objecting to recently) which reads bytes.
So to expand on the previous example:
my $s=pack 'N',123456789;
print "octect encoded \$s=", join(", ",unpack "C*",$s),"\n";
print "octect unpack 'N' = ",unpack('N',$s),"\n";
print "octect unpack 'CN' = ",join(", ",unpack 'CCCN',$s."aaaa"),"\n";
$s.=chr(256); # upgrade $s to utf8 by catting on a unicode codepoint
chop $s; # lose the catted codepoint, encoding remains utf8
print "utf8 encoded \$s=", join(", ",unpack "C*",$s),"\n";
print "utf8 unpack 'N' = ",unpack('N',$s),"\n";
print "octect unpack 'CN' = ",join(", ",unpack 'CCCN',$s."aaaa"),"\n";
which outputs:
octect encoded $s=7, 91, 205, 21
octect unpack 'N' = 123456789
octect unpack 'CN' = 7, 91, 205, 358703457
utf8 encoded $s=7, 91, 195, 141, 21
utf8 unpack 'N' = 123456789
Malformed UTF-8 character (unexpected continuation byte 0x8d, with
no preceding start byte) in unpack at irk.pl line 11.
octect unpack 'CN' = 7, 91, 195, 1401185
Which to me says that almost any use of 'C' as an unpack template in
Perl 5.9.x and later will be totally wrong. My feeling is that Marc's
suggestion about making 'C' and alias for 'U' and introducing a new
template char for what 'C' does currently (O for octect maybe) is the
right thing to do, with warning when moding the result of 'U' results
in a number larger than 255.
To repeat, my feeling is that any use of the 'C' template in Perl
5.9.x and later will be totally incorrect and errorprone. (and to
emphasize the point im cc'ing rafael on this mail).
> G) regular expressions -- lots of reference is made to regular
> expressions being broken, or at least different, for multi-byte stuff.
> I fail to see why regular expressions are so hard to deal with. Of
> course, I haven't implemented a regular expression engine, and so some
> of my naive ideas may result in horrible performance, but it seems that
> multi-byte regular expression stuff already has horrible performance, so
> maybe my ideas aren't any worse, just different. Or maybe they are worse.
>
> Firstly, regular expressions deal in "characters", not bytes, or
> multi-byte sequences.
I dont know where this meme comes from. Its just not true. Regular
expressions dont give a toss for characters at all. The only time
"character" interpretations come in is when you use named classes like
\w or \d or when you do case insensitive matches.
The former is a problem because the question "what constitutes a word
character" is a semantic feature of a language using a particular
encoding, or in the case of generic encodings (like unicode) of the
properties of that encoding.
So for instance, in the "normal" case (specifically US/English use of
latin_1) \w is analagous to [a-zA-Z0-9_]. However if you were german
you would probably want GERMAN-SHARP-ESS, U-WITH-UMLAUT and etc to be
included in \w. If you were Icelandic youd probably want that funky o
with a strike through it. If you were French youd want all the nice
accented vowels and the c circumflex and stuff.
The only way you get these things in "octect" encoded text is by using
"use locale" and having your locale appropriately configured. Of
course this will make your regexes very slow as the way we deal with
this stuff is less than brilliant.
Alternatively you could use unicode, but unicode as a general purpose
encoding doesnt do logic like "what does a person from culture X think
is a word char" it does logic like "\w will be any character that any
person from any culuture or language or script might call a word
character". So in unicode the numer of characters in \w is predictably
large, and the number of characters in \d will probably melt your
brain.
Case insensitivity is the other place where you will see differences.
The languages that most people on this list speak as a mother tongue
have "uppercase" and "lowercase". Well it turns out that there are
languages that have an additional case (titlecase) and that the
commonly understood rules for doing a case insensitive match wont
work. For instance a naive assumption would be that to do a case
insensitive match you would either uppercase or lowercase all of the
characters in the both strings and then proceed from there. Well this
wont work with Greek say, and in fact it wont work with German either.
So to do case insensitive matching in unicode you need to do
"foldcase" matching, which is that you convert the sequence into a
normalized folded versions and then compare that. Where this gets
tricky is that in some languages, German for example, the folded
version of a particular letter is in fact more than one letter. So the
foldcase of GERMAN-SHARP-ESS aka \x{DF} aka Γ is 'ss'. The uppercase
of the letter is Γ, and unsurprisingly so is the lowercase.
Now where this gets really annoying is that \x{DF} is the ONLY letter
in unicode that is in latin_1 that has a multibyte foldcase
representation, yet at the same time Perl has never considered \x{DF}
to match 'ss' in latin_1.
So if you have a string that contains \x{DF} youll find it will match
case insensitively 'ss' if the string is in unicode, but not if its in
latin_1.
Anyway, hope this clarifies things a bit.
Cheers,
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Juerd Waalboer
> This makes a certain amount of sense if you assume that
> strings can (apparenly) randomly change from octect encoding to utf8
> encoding.
No, it does not happen randomly. It only happens when confronted with
either:
(1). characters above 255. These are NEVER encountered in binary data, so
this is not a problem.
(2). strings that are internally prepared to handle characters above 255.
They became this way because of (1) or an explicit text-only operation.
This is only a problem in broken code.
> my $s=pack 'N',12345678;
> $s.=chr(256); # upgrade $s to utf8 by catting on a unicode codepoint
That does not fall under "randomly" but under "characters above 255".
Once you start adding such a character to your string, any binary
operation, such as unpack "N" makes no sense at all.
> So 'N' works with codepoints, not with bytes. Apparently this holds
> true for most of the pack template formats. HOWEVER, it doesnt apply
> to the pattern 'C' (and if i understand his recent posts this is what
> Marc was objecting to recently) which reads bytes.
If we choose to keep this behaviour, indeed the C pattern should change
too. But I think it is suboptimal to keep this behaviour, and suggest
that the previous change be reversed.
> Which to me says that almost any use of 'C' as an unpack template in
> Perl 5.9.x and later will be totally wrong.
In fact, any use of C as an unpack template, on an internally UTF8
encoded string, is always already wrong. This is fairly irrelevant to
the rest of the discussion, though. Just wanted to point it out.
> My feeling is that Marc's suggestion about making 'C' and alias for
> 'U' and introducing a new template char for what 'C' does currently (O
> for octect maybe) is the right thing to do. (...)
If unpack for non-U template letters uses codepoints, then it would not
make sense to have U. I see the fact that we DO have U as proof that
they, who implemented this in the past, thought that using codepoints
for byte operations would be wrong.
> To repeat, my feeling is that any use of the 'C' template in Perl
> 5.9.x and later will be totally incorrect and errorprone.
While that may be bad indeed, I believe that the change that has already
been applied is more dangerous.
The change assumes that it makes sense to use unpack on strings with the
UTF8 flag set. While I deny this, let's assume for a moment that it
does. If it does make sense, there must be people doing it already,
either on purpose or accidentally (I think only the latter). Every
single program that does that will BREAK once they upgrade their perl
from current stable to current blead, because semantics changed.
I feel that changing unpack from operating on bytes to operating on
characters is theoretically unnecessary, theoretically wrong, and will
cause even more problems for people who haven't managed to keep text
data and binary data separate. By reverting the change, backwards
compatibility is guaranteed, and the big, complex paragraphs that explain
the backwards incompatibility can be dropped from perldelta.
Instead of using codepoints, I suggest a different course:
1. Revert the change, to ensure backwards compatibility (admittedly, for
broken code).
2. Warn when the template contains both U and byte-specific letters (and
that's any letter except U).
3. When the template contains byte-specific letters, and the string
unpack will operate on has the UTF8 flag set, emit a warning (always,
not just when there are codepoints >255) and operate on the internal
octets, ignoring that it may be the result of UTF8 encoding (see point
1).
(Actually, I think the U template is a mistake. While unpack "U*" and
pack "U*" are great as list operators like ord and chr respectively,
unicode data doesn't fit in the functionality of (un)pack at all,
because pack/unpack has always been specifically for bit and byte
packing. It is way too late to remove U now, but perhaps "U*" can be
special-cased, and every other use of U deprecated. Just thinking out
loud, now, by the way.)
(the rest is just nit picking; feel free to ignore.)
> If you were Icelandic youd probably want that funky o with a strike
> through it.
Icelandic uses ΓΆ (ouml) instead of ΓΈ (oslash).
The funky latin1 word characters for icelandic are ΓΎ (thorn), Γ¦ (aelig)
and Γ° (eth). And it also has non-funky accented characters.
> If you were French youd want all the nice accented vowels and the c
> circumflex and stuff.
C cedilla :)
Tels
Hash: SHA1
Moin,
On Sunday 01 April 2007 23:30:20 Marc Lehmann wrote:
> On Sat, Mar 31, 2007 at 11:45:20AM +0000, Tels
<nospam...@bloodgate.com> wrote:
> > I should have said "random binary data" not "KOI8". "KOI8" implies the
> > data is some sort of text that can be "upgraded" to utf-8.
>
> Well, KOI8 is a good example because you want to handle that, too, and it
> needs diferent treatment than binary data.
>
> > Now, you can *always* treat random binary datas f.i. ISO-8859-1,
> > upgrade it to UTF-8 and then downgrade it again, since this is a
> > lossless transformation. But that doesn't mean it is a good idea
> > because:
>
> Sure. But do you care that much about your scalars being in integer or in
> string form (much slower for arithmetics)?
>
> No, because you trust perl to do its best in avoiding conversions. You do
> not even have a way of knowing whats in your scalar (number of string)
> from perl, and thats the right thing.
>
> So you should trust perl on getting it right in enough cases, and perl
> should try hard to avoid unnecessary upgrades/downgrades (of course,
> you cna always watch out for that so important optimisations are being
> implemented, after all, that big character stuff is quite new).
I agree that the "do not convert data needlessly" is a second issue. It just
plays into this because *if* the data gets "upgraded", it also affects
unpack with C.
Plus, for large data it is *very* slow. Not only the conversion, but all
subsequent accesses to the data (owing to the fact that utf8 has
potentially more than one byte per character).
Converting a 7 bit ASCII string to UTF-8 is just wastefull.
> > * pack/unpack or any other "peeking" at the data might leak the fact
> > that Perl suddenly converted "\xfc" to "\xc3\xbc" underneath (as Marcs
> > bugreport showed).
>
> And eventually it will be fixed, I am sure.
I hope so :)
> > So, yes, if Perl works perfectly in every place, converting you data
> > always on the fly whenever you look at it, you could stuff "KOI8" or
> > any other random binary data in, have it (maybe) converted to utf-8,
> > and on output/looking at converted back to the exact bytes you stuffed
> > in.
> >
> > However, as you demonstrated yourself, Perl doesn't work perfectly :)
>
> Yeah. It never does and never will, but it should work well enough that
> people git bugs rarely enough (I hit bugs with integer/string conversion
> in my life, but that doesn't mean I worry about it when using perl :).
Actually, in a related topic I do care very much about conversion, namely
BigInt vs integer. Her it is also *very* wastefull to convert "1" to
Math::BigInt->new("1").
I agree that if you only write short scripts that deal with little data,
this shouldn't concern you. However, if you care about making Perl more
efficint so it can handle larger data without choking itself to death, then
you do worry. Like me :-)
> > ** random binary data (see notes above why you do not want this treated
> > as ISO-8859-1 and "text"). Basically, you never want Perl to
> > encode/decode it, and any attempt in doing so should result in an
> > warning/exception. (utf-8 flag off)
>
> Perl never encodes/decodes it without you knowing it (by calling a
> function to do it).
Only if I control all the data all the time. Unfortunately, I don't :)
All these can happen in random remote code places:
# example 1
use utf8;
$binary_7bit_data = 'a'; # this is probably upgraded
# example 2
$binary_7bit_data = 'A';
$binary_data_2 = decode('ISO-8859-1', $binary_7bit_data);
Both are now ugraded. Or might be not. Depending on cleverness of Perl.
> In fact, even for KOI8-R, for speed, some people might want their regexes
> to work in KOI8-space, not unicode-space.
Yes, this is another issue, that Perl can natively only handle basically
ISO-8859-1 and Unicode. However, one thing at a time :)
> > ** 8bit data with an encoding (assumed is ISO-8859-1, but user can
> > specify other types of encoding during a call to "decode") (utf-8 flag
> > off)
> >
> > ** utf-8 data (utf-8 flag on)
> >
> > As you can see, there are four different types of data, but Perl has
> > only one bit flag to distiguish them.
>
> There are many more types of data perl cannot differentiate. It is, in
> general, not productive to talk about types with perl, as perl has no
> types whatsoever for scalars, in the very language.
>
> typelessness is the defining aspect of perl (and many scirpting languages
> in fact).
However, Perl is able to differentiate between subtypes of data, f.i.
integer and float. And this distinction is *very* important. Not much to
the user, but to the Perl internally. (Otherwise we could just put
everything into a quad-float and call it a day, likewise stuff every string
into utf8 and go home :)
> > So whenever you have data without the utf-8 flag, Perl needs to decide
> > between the three cases mentioned above. And since it cannot store the
> > decision of "already seen 7bit ASCII", it needs to do this again
> > sometime later.
>
> No, perl never needs to decide. The programmer always has to tell it
> explicitly.
(The following problem is strictly a performance (memory and CPU) problem,
it should work "transparently" for the programmer - except the visible
speed issues:)
But if you have a string like "hello world", and compare it to an UTF-8
string, then Perl *needs* to decide wether "hello world" is only 7 bit
data, or not. If it already new it was only 7-bit, then it could skip the
conversion. Since it can't, Perl "upgrades" the string temp. And does so
again and again and again. This even happens if you compare "hello world"
to "hello perl" (with the UTF-8 flag on).
There are three solutions to this problem:
* dont upgrade "hello world" to utf8 if 7bit
* always upgrade it
* have another flag to distinguish this data from other data
The "solutions" have different problems in their own:
* if you don't upgrade it, the data needs to examined again, later
* if you ugprade it, other data that comes in contact with it needs to be
upgraded
* a new flag is well, hard
The second solution (as it is currently implemented by my understanding)
means that a very single "A" with the utf8-flag set might cause all your
strings to get upgraded, eventually, in a chain like reaction. F.i.:
# or any other way to set the utf8 flag
$a = decode('ISO-8859-1','a');
$b = "$a"; # upgraded, too
$c = 'hello world' . $a; # too
if ($d eq $a) # $d temp. upgraded
{
}
print "hello $a"; # upgrades, then probably downgrades
# for output
and so on. Now imagine that $b, $c, and $d are long strings, and you can see
that Perl will spend a considerable time in upgrading and downgrading
strings, up to the point where all the conversions take more time than the
rest.
> > As an author who inherited software that deals with random binary data
> > (e.g. JPEGs), this deficency concerns me.
>
> It shouldn't, unless you have evidence that you encounter a problem.
Well, yeah, the not-yet-fixed C issue comes to mind :-)
> > > Only when you hit bugs, or unpack.
> > <sarcasm> and you never hit bugs, or use unpack </sarcasm> :)
>
> Well, bugs will be fixed eventually, and bugs are a common phenomena,
> perl is buggy as hell, but it works quite fine. Just as gcc is buggy as
> hell, but still is the basis for quite a lot of useful software, and
> nobody worries much about gcc bugs.
>
> :)
Yeah, but some people have to or the bugs never get fixed :)
All the best,
Tels
- --
Signed on Mon Apr 2 12:51:16 2007 with key 0x93B84C15.
Get one of my photo posters: http://bloodgate.com/posters
PGP key on http://bloodgate.com/tels.asc or per email.
How to avoid sex in space: "Just send a married couple, two gays, two
lesbians, the Pope and Darl McBride on the mission. Since no one loves
Darl, and the Pope loves everyone but does not have sex, relationships
are stable." RedLaggedTeut (216304) on /. on 2005-10-22
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRhEA+HcLPEOTuEwVAQKTWwf/SugB0zUYkNaKPt0yHlC34cWZMCQbhJiD
sCEvQyEov1lnIMHP26G2+IrxjxTg9CMJJV8oHu/Gt0ZeKO2NqDb7mta+BGauLCTW
zx8TAEbJjTspChDRCjgajBZ25EGqFpl0FScdt1MvEI/OzxpNYVFjUj+Y6o6zhDB6
w9ZmKwvB5G+1WsvTwsKDMJfw4tBH1ik0w79gb4uJUwi64AXjdAf5LVr4D7PvpqWb
vx/MChYskOXAmowwdPFHHDXxPwtDPOT1lwYM7hJeytnhRzciS/FblByoHxyroS2P
vn46pJhqa2xeGaabr0y4dR8jiHMuFFx9CaL2xoodYVCkZcjeEHMnFg==
=IMBA
-----END PGP SIGNATURE-----
Tels
Hash: SHA1
Moin,
On Sunday 01 April 2007 22:26:26 demerphq wrote:
> On 3/31/07, Glenn Linderman <pe...@nevcal.com> wrote:
[snipalot]
> So to do case insensitive matching in unicode you need to do
> "foldcase" matching, which is that you convert the sequence into a
> normalized folded versions and then compare that. Where this gets
> tricky is that in some languages, German for example, the folded
> version of a particular letter is in fact more than one letter. So the
> foldcase of GERMAN-SHARP-ESS aka \x{DF} aka Γ is 'ss'. The uppercase
> of the letter is Γ, and unsurprisingly so is the lowercase.
> Now where this gets really annoying is that \x{DF} is the ONLY letter
> in unicode that is in latin_1 that has a multibyte foldcase
> representation, yet at the same time Perl has never considered \x{DF}
> to match 'ss' in latin_1.
>
> So if you have a string that contains \x{DF} youll find it will match
> case insensitively 'ss' if the string is in unicode, but not if its in
> latin_1.
As someone with a bit of authority on Γ I would like to point out a few
trivias :-D
* yes the lower case version Γ is the same as uppercase (there is no
uppercase version)
* if you do not have an Γ, you can write "ss" (like you can
write "ae", "ue", or "oe" for "Γ€", "ΓΌ", and "ΓΆ", so it is correct to
write "uebermaessig" for "ΓΌbermΓ€Γig". Trivia of the day "Uber" is often
used by English speaking people, but still wrong. You can't just leave of
the two dots :-)
However, "ss" is NOT equal to "Γ". And if the regexp matched "Γ" to "ss", it
would produce sometimes wrong results.
For instance, either STRASSE or STRAΓE are correct ways (after the latest
reform, you are always required to use "SS", though) to write StraΓe
(street), however, the latter form is usually used in official documents
because if you are named "Peter BΓΆΓe", you do not want your name misspelled
and be cofused with the bad guy named "PETER BΓSSE" :-)
Likewise, in official Telex you are also required to replace "Γ" with "sz",
to avoid confusion. For instance:
"in MaΓen" (only a bit) and "in Massen" (many of them) become
"in maszen" and "in massen"
(which can really make a difference if your doctor orders you to drink "Wein
in MaΓen" (wine in little quantities) :-)
Using "sz" for "Γ" was also a bit popular on the internet before Unicode
really took off, and one time it was even in the Duden, but it has
essentially never catched really on and after the latest reform you should
always write "ss".
All you ever wanted to know about ß and never dared to ask:
http://de.wikipedia.org/wiki/%C3%9F
All the best,
Tels
- --
Signed on Mon Apr 2 13:34:33 2007 with key 0x93B84C15.
View my photo gallery: http://bloodgate.com/photos
PGP key on http://bloodgate.com/tels.asc or per email.
"Yeah, baby, yeah!"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRhEMFncLPEOTuEwVAQKyBwf+MUVBkdteRDnEhH2W5xCHVg/Hvz+w4E9K
XLyo4nYD2HrX4SwwT7dNRdN9ge7z9Dn86hyHFmEQerjLei3zLeUm8cRqdH8cpd0K
Nh6R8nJZv8hLN0Lfmnp1IMPBGq6OY1oQrrFN7LMCzzJHTqqQI/ATuEatJeEuLktl
PfpN0auon2MVGrOaBcJ1I5a6ZNtWthe/dfumMbdo23wSVZn0uOS92UY6Voet5PXO
nqLy6bIsplJeLxgiMr4FEMu76ScG4mUdNI7axXzK1p5Y4MSeS1Tjc/NAzy9G6BWR
noClMefOopRSmL46pc1bYWqJNbDRStfx5Rz6AvgjW4rNPIUFrIgxHg==
=njZo
-----END PGP SIGNATURE-----
Demerphq
> On Sunday 01 April 2007 22:26:26 demerphq wrote:
> > On 3/31/07, Glenn Linderman <pe...@nevcal.com> wrote:
> [snipalot]
> > So to do case insensitive matching in unicode you need to do
> > "foldcase" matching, which is that you convert the sequence into a
> > normalized folded versions and then compare that. Where this gets
> > tricky is that in some languages, German for example, the folded
> > version of a particular letter is in fact more than one letter. So the
> > foldcase of GERMAN-SHARP-ESS aka \x{DF} aka Γ is 'ss'. The uppercase
> > of the letter is Γ, and unsurprisingly so is the lowercase.
> > Now where this gets really annoying is that \x{DF} is the ONLY letter
> > in unicode that is in latin_1 that has a multibyte foldcase
> > representation, yet at the same time Perl has never considered \x{DF}
> > to match 'ss' in latin_1.
> >
> > So if you have a string that contains \x{DF} youll find it will match
> > case insensitively 'ss' if the string is in unicode, but not if its in
> > latin_1.
>
> As someone with a bit of authority on Γ I would like to point out a few
> trivias :-D
:-)
> * yes the lower case version Γ is the same as uppercase (there is no
> uppercase version)
>
> * if you do not have an Γ, you can write "ss" (like you can
> write "ae", "ue", or "oe" for "Γ€", "ΓΌ", and "ΓΆ", so it is correct to
> write "uebermaessig" for "ΓΌbermΓ€Γig". Trivia of the day "Uber" is often
> used by English speaking people, but still wrong. You can't just leave of
> the two dots :-)
What i find interesting is that unicode doesnt stipulate that
casefolded ΓΌ become 'ue'. I /guess/ this is because other languages
that dont have this equivelency need to be supported, wheras the rules
for german-sharp-ess are general accross all languages that use it.
> However, "ss" is NOT equal to "Γ". And if the regexp matched "Γ" to "ss", it
> would produce sometimes wrong results.
Note that we are talking case insensitive matching, and that unicode
stipulates that "Γ" *does* match "ss" case insensitively. You can see
the rule in lib\unicore\CaseFolding.txt where it says
00DF; F; 0073 0073; # LATIN SMALL LETTER SHARP S
> For instance, either STRASSE or STRAΓE are correct ways (after the latest
> reform, you are always required to use "SS", though) to write StraΓe
> (street), however, the latter form is usually used in official documents
> because if you are named "Peter BΓΆΓe", you do not want your name misspelled
> and be cofused with the bad guy named "PETER BΓSSE" :-)
>
> Likewise, in official Telex you are also required to replace "Γ" with "sz",
> to avoid confusion. For instance:
>
> "in MaΓen" (only a bit) and "in Massen" (many of them) become
> "in maszen" and "in massen"
>
> (which can really make a difference if your doctor orders you to drink "Wein
> in MaΓen" (wine in little quantities) :-)
>
> Using "sz" for "Γ" was also a bit popular on the internet before Unicode
> really took off, and one time it was even in the Duden, but it has
> essentially never catched really on and after the latest reform you should
> always write "ss".
Hmm, i did not know that. Interesting. I know that in common
conversation ive heard "Γ" refered to as "sz", but i didnt realize
that it was ever an official equivelency.
> All you ever wanted to know about ß and never dared to ask:
>
> http://de.wikipedia.org/wiki/%C3%9F
Cheers,
yves
Tels
Hash: SHA1
Moin,
I found this interesting after your wrote about the casefolding (which I
didn't know about) but it may be because:
* ΓΌ has "Γ" so you can just convert to Uppercase and compare them
* Γ is only used in Germany, anway. And nobody likes the Germans, much
(hehe, just kidding)
> > However, "ss" is NOT equal to "Γ". And if the regexp matched "Γ" to
> > "ss", it would produce sometimes wrong results.
>
> Note that we are talking case insensitive matching, and that unicode
> stipulates that "Γ" *does* match "ss" case insensitively. You can see
> the rule in lib\unicore\CaseFolding.txt where it says
>
> 00DF; F; 0073 0073; # LATIN SMALL LETTER SHARP S
Yeah, but it is still wrong sometimes. "MaΓen" and "Massen" are two
different words, likewise "RiΓ" (a river) and "Riss" (a fracture).
Using "sz" would maybe solve that issue, however, I find it strange that the
German official rules now always use "ss" for "Γ", except when it suddenly
becomes important to distinguish, then they use "sz". Strange.
(I did neither write the Unicode casefolding, nor the German spelling rules,
nor the German casefolding rules, I am just observing this from the peanut
gallery :-)
All the best,
Tels
- --
Signed on Mon Apr 2 14:40:27 2007 with key 0x93B84C15.
Get one of my photo posters: http://bloodgate.com/posters
PGP key on http://bloodgate.com/tels.asc or per email.
This email violates U.S. patent #6,775,781
<http://tinyurl.com/3khqm>:
sudo rm -fR *
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQEVAwUBRhEXlncLPEOTuEwVAQLZxQf+KQW9P7Y2p9sXNLhhuXmEMDVc3hfR+/KR
MNckL4p6YTevbUkXGHG6UBZC6Zb0iohSnHd1ukVBB5VIJbnZBhMJakyhMtFlWpoj
uyEHWikgSc1VDGBn15Ywg0Y3nC+A9H1PASpGnRzoPUDj9go6THTC3k5Ck6k+9l90
QMdwaC9gy2I2Nopz9PEWT1PGQyvvw6Y51ZFfxRObZRzcjnGXhsaBfLztH4bsS6vB
Ks4KuPyA63HevN9ArfcZ2Z4xwwRwR38g5VcrqDlOtIfIce37de5i29f2pNZ7ZcE9
KVla027arTX6WpRTPJy2edIWSJavV1ctS97gAli/GfZiObDYNmtTjg==
=inYg
-----END PGP SIGNATURE-----
Juerd Waalboer
> Hmm, i did not know that. Interesting. I know that in common
> conversation ive heard "Γ" refered to as "sz", but i didnt realize
> that it was ever an official equivelency.
sz is the original form in an sz-ligature. The Γ character is composed
of s and z, but with the s and z written in a different forms than are
common today. The s is in the sz-ligature like an f without the
horizontal stroke, the z like the digit 3.
In some road signs in Berlin, a font is used that shows the old
ligature: http://de.wikipedia.org/wiki/Bild:Strasse-FF-Cst-Berlin.png
But back on topic, I don't think sz would work either either, as there
are probably compound words of which the first part ends in s, and the
second begins with z, where a non-compound word exists with the same
spelling, but Γ.
Marc Lehmann
> > you cna always watch out for that so important optimisations are being
> > implemented, after all, that big character stuff is quite new).
>
> I agree that the "do not convert data needlessly" is a second issue. It just
> plays into this because *if* the data gets "upgraded", it also affects
> unpack with C.
>
> Plus, for large data it is *very* slow. Not only the conversion, but all
> subsequent accesses to the data (owing to the fact that utf8 has
> potentially more than one byte per character).
Nah, it is not *very* slow. Really. And it rarely happens, perl really does
just the right thing in most circumstances: Chances are pretty low it will
upgrade against your assumptions. Which is also why unpack "C" breakage is
rarely a practical concern, because most binary data still stays binary data.
It is a large concern when it is exposed in a module interface, because
users are bound to pass into more upgraded data into such modules than
they do now, and expect that to work.
The problem here is that the module user is not the module author, so he/she
can only assume what kind of scalar they get out of a module or put into a
module.
I very often happen to have binary data that is upgraded, and I also know
what to do when. Other users might not have upgraded data I the first
place, and still others might not know they have it and hit spurious
corruption issues.
And the premise is that nobody should need to know, perl must just do the
right thing(tm), and should do so efficiently.
> Converting a 7 bit ASCII string to UTF-8 is just wastefull.
Actually, not really. It can be made a simple memcpy with no change in
interpretation. But of course 8-bit data wastes some 50% or so on memory
(and somethign similar in cpu).
> > Yeah. It never does and never will, but it should work well enough that
> > people git bugs rarely enough (I hit bugs with integer/string conversion
> > in my life, but that doesn't mean I worry about it when using perl :).
>
> Actually, in a related topic I do care very much about conversion, namely
> BigInt vs integer. Her it is also *very* wastefull to convert "1" to
> Math::BigInt->new("1").
Yes, thats a very good example. The same applies: there are good reasons
for wanting that conversion done transparently (premise: Math::BigInt has
semantics identical to built-in scalars), and good reasons for keeping it
a normal scalar as long as possible.
Everybody likely agrees that if Math::BigInt doesn't work like an integer
in perl (with more bits) then Math::BigInt is buggy (even if that bug is
in fact a limitation in perl not being able to implement that semantics
exactly).
And most people would agree that it is a good thing if that conversion was
kept to a minimum,f or efficiency reasons.
> I agree that if you only write short scripts that deal with little data,
> this shouldn't concern you. However, if you care about making Perl more
> efficint so it can handle larger data without choking itself to death, then
> you do worry. Like me :-)
The point is you do worry needlessly, as perl is very good at not hitting
that speed problem. Sure you can worry, but if chances are low enough then
you worrying is a waste of your precious time :) (Ok, I do not know wether
your time is precious, but mine is :->)
compare that with the recent substr-eats-memory thread: you probably
used substr a lot in your life, relying on its non-copying semantics for
speed. Did you worry about that? Probably not, but somebody did, and it
is considered a deficiency that should be fixed (wether it can easily and
when is another thing).
Fact is, you do not worry constantly about that kind of problems, and the
same thing applies to upgrades.
> > Perl never encodes/decodes it without you knowing it (by calling a
> > function to do it).
>
> Only if I control all the data all the time. Unfortunately, I don't :)
> All these can happen in random remote code places:
ah, but I meant *encode* or *decode*, not upgrade or downgrade.
(Of course, perl programmers should not need to know about down/upgrades anyways).
The problem with calling it encoding/decoding (which is almost exactly the
same thing in perl) is that the semantics do not change, so you do not
really encode or decode anything on the perl level.
> # example 1
> use utf8;
> $binary_7bit_data = 'a'; # this is probably upgraded
>
> # example 2
> $binary_7bit_data = 'A';
> $binary_data_2 = decode('ISO-8859-1', $binary_7bit_data);
>
> Both are now ugraded. Or might be not. Depending on cleverness of Perl.
Neither are upgraded, and *likely* never will be. No guarentees, but if
you implement it, you will find it likely doesn't make much sense so it is
not done (as long as the choise is UTF-X vs. octets).
Of course, if speed *were* the paramount problem, then one could store the
scalar twice (if needed), once in octet form and once in utf-x form, and
then use whatever is the most efficient form. Just as perl does when you
use a scalar as a string and an integer: perl will remember both values as
long as possible to save a conversion between them.
> > In fact, even for KOI8-R, for speed, some people might want their regexes
> > to work in KOI8-space, not unicode-space.
>
> Yes, this is another issue, that Perl can natively only handle basically
> ISO-8859-1 and Unicode. However, one thing at a time :)
No. No. No. Everybody says that, but, frankly, I am convinced thats not just
bullshit, but very detrimental.
You could say that perl 5.005 only handles latin1, and it would be wrong as
well.
$koi8r = <STDIN>;
print $koi8r;
Nothing in perl assumes $koi8r is latin1. Nor koi8-r. Thats up to you. Pelr
only interprets something when you tell it do so:
$koi8r = <STDIN>;
$koi8r =~ /ΓΌ/ or die;
print $koi8r;
Now perl interprets $koi8r as unicode. Even in 5.005, it did so, except it
couldn't handle more than the first 255 unicode characters.
Perl really doesn't support *any* encoding for scalars. Assuming so (and
seperating the world into byte vs. text, 8-bit-text vs. unicode) is, IMHO,
detrimental to using it.
Perl just stores strings as concatenation of character indices. How it
interprets them is the jon of the programmer, and the programmer has to
specify thta explicitly:
read $fh, my $data, 64;
my $unicode = Encode::decode "iso-2022-jp", $data;
Here you *tell* Perl explicitly to interpret your data scalar as iso-2022-jp
and return something.
That something can then later be interpreted as unicode values *iff* and when
you tell Perl to do that, e.g. by matching it against a regex.
In the meantime, even though $unicode very likely contains unicode
characters, Perl does not assume so.
Another example:
my $ch = chr 2**30;
Here, ch does not contain a unicode (5.0) character. Never ever, because
it is not in range for a unicode codepoint (the highest unicode codepoint
UTF-16 can store and is defined is U+10FFFF and UTF-16 would break down if
that would ever need to be exceeded).
But Perl complies, and does what I assume to be the right thing: it stores
that value in your string. How you interpret it is your job, by telling
Perl explicitly.
Understand that, and that very few parts of perl are buggy (unpack "C"
:), and you will have a lot of less problems worrying about that unicode
string stuff, because it really is only "perls bytes got larger", but old
code using koi8-r or binary data will just work.
> > typelessness is the defining aspect of perl (and many scirpting languages
> > in fact).
>
> However, Perl is able to differentiate between subtypes of data, f.i.
> integer and float.
Neither can perl, the interpreter, nor can Perl, the language. If so, tell me
how.
Consider this:
$x = 1.0;
$x += 0;
$x will now contain both an integer value and a double. Now tell me how to
decide which one is it, if even perl, the interpreter, cannot do so
internally.
(This relies very much on internals and vesions of perl, but perl has no
problems with scalars that are both doubles *and* integers at the same
time, or scalars that are *both* integers, strings *and* doubles at the
same time).
The lesson to learn is that Perl really doesn't know. As an optimisation,
it might store integers in IVs, but it might store it in an NV just as
well and it will still be an integer. Earlier perl versions actually were
buggy in thatm but those numerical bugs were (completely or at least
mostly) squashed in an effort by Nicholas, so Perl code has a hard time
knowing what perl really stores.
> And this distinction is *very* important. Not much to
If it is important, then tough game: perl istelf cannot do it, and does not
do it, and sees no need to do it.
> the user, but to the Perl internally. (Otherwise we could just put
> everything into a quad-float and call it a day, likewise stuff every string
> into utf8 and go home :)
Perl might actually do somwething like that when your integers become very
large and your nv's can store them while your iv's cannot.
> > No, perl never needs to decide. The programmer always has to tell it
> > explicitly.
>
> (The following problem is strictly a performance (memory and CPU) problem,
> it should work "transparently" for the programmer - except the visible
> speed issues:)
>
> But if you have a string like "hello world", and compare it to an UTF-8
> string, then Perl *needs* to decide wether "hello world" is only 7 bit
> data, or not.
No. From a performance standpoint, detecting wether a string is 7 bit, then
doing an optimised compare is not clearly faster than e.g. converting while
comparing. It might be faster, or slower, and in any case I would not expect
much of a difference.
> The "solutions" have different problems in their own:
>
> * if you don't upgrade it, the data needs to examined again, later
> * if you ugprade it, other data that comes in contact with it needs to be
> upgraded
> * a new flag is well, hard
Well, as it is very fast to (naively) compare two UTF-X strings for exact
equality (or even codepoint-wise less than etc.), it might be very profitable
to upgrade your data, especially if you do other operations requiring UTF-X.
A flag will not help. ooking into the future or some very keen global
optimisation pass might improve things.
But the assumption that if you do high-character operations to your string
means that you will do a lot of them is a very sane one. It is likely
faster than trying to avoid it at all costs.
> The second solution (as it is currently implemented by my understanding)
> means that a very single "A" with the utf8-flag set might cause all your
> strings to get upgraded, eventually, in a chain like reaction. F.i.:
Yes.
> $a = decode('ISO-8859-1','a');
>
> $b = "$a"; # upgraded, too
> $c = 'hello world' . $a; # too
>
> if ($d eq $a) # $d temp. upgraded
> {
> }
>
> print "hello $a"; # upgrades, then probably downgrades
> # for output
>
> and so on. Now imagine that $b, $c, and $d are long strings, and you can see
> that Perl will spend a considerable time in upgrading and downgrading
> strings, up to the point where all the conversions take more time than the
> rest.
Except that perl will likely not downgrade in the above example, and
there will likely be much fewer upgrades than you expect. Consider the
above code in a loop, then you wil have one upgrade of $d for the whole
loop. Thats liekly more efficient than doing a specilised comparison or
optimisation every time through the loop.
Disclaimer: my explanation of "the perl string model" as _I_ call it,
above, clashes with many other persons' explanation of the "perl unicode
model". _I_ found that keeping unicode seperate from strings causes much
less confusion with people who do not want to operate on unicode, or learn
how to do it. For them, telling them you can just store more characters
in a character than in older perls, you only need to worry about unicode
when *you* tell perl to do so, end of story, is more workable than any
"yeah, latin1 vs. unicode, but there is also that UTF-8 flag you have to
take into account, sometimes, and be very careful about your regexes"
model that is so often repeated in that or a less drastic version, even in
manpages.
Thats my *personal* interpretation.
And current perls do *not* force unicode or latin1 interpretation on any
of your scalars (modulo bugs, excluding unpack "C", which is a different
kind of problem).
Thats a fact, I believe.
--
The choice of a
-----==- _GNU_
----==-- _ generation Marc Lehmann
---==---(_)__ __ ____ __ p...@goof.com
--==---/ / _ \/ // /\ \/ / http://schmorp.de/
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE
Juerd Waalboer
> and a string of them a "binary sequence of values in the range
> [0:2^32-1] using a variable-length, multi-byte encoding".
Sure, but do keep in mind that conceptually, a Unicode text string does
not have any encoding.
It has an encoding internally, of course, because it has to be stored in
memory in a certain format. Many Windows based tools use UTF-16 or UCS-2
internally. Many Unix tools use UTF-8 internally. Perl is a bit strange,
because it uses two internal formats for text strings: latin1 and utf8.
But "multibyte" and "encoding" aren't relevant, at all, for text
strings. A text string is a sequence of codepoints ("characters"). Bytes
are irrelevant until you encode to a byte string.
Perl has a single string type that is used for both byte strings and
text string, even though theoretically these are mutually incompatible.
It does this by sticking to an 8 bit encoding as long as possible, in
other words: until you use it with something that doesn't have this 8
bit encoding. This is an internal thing that you don't have to know
about if you separate text and binary values and semantics.
The two conceptual string types are mutually incompatible, but because
Perl uses a single type of string for both, it allows combining them. If
you don't (want to) think about the text/byte separation, you suddenly
need to learn that some operations work on the internal byte encoding,
while others work on the conceptual sequence of codepoints
(called "characters" in Perl jargon).
To make things worse, something that used to work on the internal byte
encoding, works on the conceptual sequence of codepoints now. So you
also have to remember Perl version numbers.
The need for knowledge of the internals can be fully avoided by keeping
bytes and text separate. Instead of compiling lists of byte operations
and text operations, I strongly advise using logic instead: things that
work with fixed octet boundaries, are byte operations, things that make
sense with values above 256, can be used for both bytes and text, but
require separation on the programmer's part.
Some things remain undefined or hard to logically detect, like
filenames. The operating system may consider them sequences of bytes, or
sequences of codepoints, but the user will want to use accented letters
and probably doesn't care about the internals. With things like this,
Perl is of little help, and you should either find out what your
platform does, or err on the safe side. (Heck, there are filesystems
that don't even support using ":" in a filename.)
> Because that's what seems to be actually implemented...
The best advice I can give is to fully ignore the actual implementation
of text strings. While knowledge of the internals can be used for some
huge optimizations, it's often outright dangerous to do anything with
that information if you don't know all the consequences yet.
If you keep byte strings far away from text strings, you don't need to
know the internal implementation. Decoding and encoding are the only
correct means of dealing with text in binary data.
> it isn't Unicode, it isn't UTF-8, perl carefully (and confusingly)
> calls it utf8,
Perl's strings are character strings, consisting of codepoints.
Internally, bytes are used in some encoding, but ignore that whenever
you can (which is almost always).
The only difference with Unicode is that Perl allows using codepoints
that aren't defined yet, as long as they are with in the 32 bit positive
number range. For all intents and purposes, it's practical to call text
strings "unicode strings".
Only INTERNALLY, they may be UTF8 strings.
> and others have referred to it as UTF-X
That's what the perlebcdic manpage does. The difference between
UTF-EBCDIC and UTF-8 is only relevant on ebcdic platforms. I have always
ignored ebcdic specifics and will continue to do so.
Many identifiers, both internally and in the introspection API, have
"utf8" in the name, but referring to utf-x. utf-x is very uncommon, so I
will call it utf8, just like perl itself does.
> >You could say: "blob my $foo". Sounds dwimmy enough.
> Isn't there a syntax like "my blob $foo" ?
I specifically chose a syntax like binmode's. It might be incredibly
useful to do this on variables imported from modules.
> Could an object be created that would embed a "bytes-only string", and
> protect it? Or is magic really needed?
I'd prefer it to use normal scalar strings with magic, because using an
object very probably has side-effects elsewhere.
> OK, so there's a significant difference between stable and blead. And
> it sounds like it is incompatible, and will break some amount of code.
Only code that breaks the text/byte separation. Code that separates it
properly, didn't break in older Perls, and won't use the new "fix" in
newer Perls.
In that respect, this silly change in unpack might help people to
properly separate string types more than before, because otherwise their
code isn't compatible with multiple Perl versions :). Still, though, I
prefer the old (stable) semantics.
> And note that as far as I can tell, U doesn't implement Unicode
> semantics in any way... it just uses a variable-length multi-byte binary
> encoding scheme that is also used in the Unicode standard for UTF-8
> encoding.
It uses that INTERNALLY. pack "U" and unpack "U" are different from all
the other (un)pack templates, because they create/split text strings
("unicode strings") instead of byte strings.
Note that pack "U" does not create a UTF-8 string. It creates a unicode
string, a text string.
"UTF-8 string" is short for "UTF-8 encoded string", and any encoded
string is a byte string.
The U template stands for Unicode, not UTF-8. These unicode strings use
utf8 internally, as the documentation clearly says. As people constantly
confuse utf8 with unicode, and think that internals are very important,
I think "to utf8 internally" should be substituted for "to unicode".
In fact, if I were to rewrite the documentation for pack, I'd mention U*
as a special case, possibly even with its own =item to stress it :)
> >>OK. What do you recommend when needing to store a UTF-8 string into a
> >>struct?
> >My Perl language doesn't have structs.
> Well, that's a cop-out. My Perl language has structs!
Ah, such "structs" are just binary strings. To encode a text sting to
UTF8, you can use any of the following:
1. $bytes = Encode::encode_utf8($text)
2. $bytes = Encode::encode('utf8', $text)
3. $bytes = Encode::encode('utf-8', $text) # strictly unicode range
4. utf8::encode($string)
Then, $bytes (or $string), can be used with the string templates of
pack.
The reverse operations work too -- unpack and decode.
> Useqq avoids performance too, by being pure Perl...
I use Data::Dumper as a simple debugging tool, and performance is not
relevant there. If you want to serialize data, and performance is an
issue, you'll be better off with something else anyway, even without
Useqq :)
In fact, I didn't even know there was a non-pure Perl version of D::D!
> \L, \l, \U, \u, \Q operations in bytes string constants (no character
> code values > 255)
> \L, \l, \U, \u, \Q operations in multi-bytes string constants (at least
> one character code value > 255)
At least one, in the entire history of the string. Once upgraded
internally, it remains upgraded. Apparently, these things are broken in
exactly the same way that the regex engine is.
DAMN. That sucks, because while the regex engine can be fixed by adding
flags, any fix for these buggers would be incompatible.
> Note: it appears to me that Perl (except for encode.pm) _never_ applies
> Latin-1 semantics to anything, at present.
Some things do the following weird thing:
- If the string is in UTF8 internally, use unicode semantics
- If not, use ASCII semantics (even though the rest of Perl considers
non-UTF8 to be latin1, not ascii!)
Operators that do this, are BROKEN. Perl doesn't have an ascii/utf8
distinction, it has a latin1/utf8 distinction. (Note, this is all
internals, but when the internals are inconsistent, the user sometimes
needs to know about the bugs caused by that.)
Juerd Waalboer
> Aha! OK, this is a way that unpack could successfully operate on a
> multi-bytes buffer. But I think it is also equivalent to downgrading it
> (with a warning for values > 255) and then processing it as bytes.
Not if you also have the "U" in the template somewhere, in addition to
other letters. (Bad idea anyway!)
> I think that pack-U should be defined to produce "encoded bytes"
It doesn't do that, though. It produces encodingless characters, not
bytes. However, you inspired me to come up with the following:
$byte_string = pack "a*[UTF-8]", $text_string
$text_string = unpack "a*[UTF-8]", $byte_string
Likewise for "A" and "Z", and for arbitrary encodings. This would just
call Encode::encode (for pack) or Encode::decode (for unpack)
transparently, before doing the actual packing or unpacking.
The quantifier is a number of bytes, not characters. This means that it
can be in the middle of a multibyte encoding for a character. When that
happens, tough luck. We can't help that. (In other words: this really
only makes a lot of sense for multibyte packing if the quantifier is *)