warding against bytes.pm
Dave Mitchell
tasks, the user has no need for bytes.pm, and indeed, that most naive use
of it is likely to be harmful.
Given that, should we add a warning to bytes.pm's pod to the effect that
you shouldn't be using this for new code unless you really know what you're
doing, and that a desire to do so probably indicates a design flaw in your
code or your understanding of perl's Unicode implementation?
--
Counsellor Troi states something other than the blindingly obvious.
-- Things That Never Happen in "Star Trek" #16
Steffen Mueller
Dave Mitchell wrote:
> I think these days there's a general consensus that, for normal day-to-day
> tasks, the user has no need for bytes.pm, and indeed, that most naive use
> of it is likely to be harmful.
>
> Given that, should we add a warning to bytes.pm's pod to the effect that
> you shouldn't be using this for new code unless you really know what you're
> doing, and that a desire to do so probably indicates a design flaw in your
> code or your understanding of perl's Unicode implementation?
I'm usually purposefully ignoring Unicode issues on this list. They make
my head hurt. But in this case, it's about the clueless (like myself).
I'd say such a warning may be useful. But...
1) If we add such a warning, we make certain assumptions* about what the
user was trying to do and ended up wanting to use bytes.pm.
2) If we assume we have a good idea what she really wanted to do, we
should add pointers to the right way to do it next to the warning.
Without pointers to the right solution, such a warning wouldn't be very
helpful.
Best regards,
Steffen
* I'm being vague for lack of Unicode-fu, here.
Aristotle Pagaltzis
> Given that, should we add a warning to bytes.pm's pod to the
> effect that you shouldn't be using this for new code unless you
> really know what you're doing
I’m in favour. I’d word it more strongly than that too.
* Steffen Mueller <smue...@cpan.org> [2010-01-04 09:45]:
> 1) If we add such a warning, we make certain assumptions* about
> what the user was trying to do and ended up wanting to use
> bytes.pm.
Very nearly *everything* they would want to do with bytes.pm
would be a mistake. Using bytes.pm in application code (as
opposed to guts hackery) is almost by definition a bug.
> 2) If we assume we have a good idea what she really wanted to
> do, we should add pointers to the right way to do it next to
> the warning.
About the only useful thing I can think of is to point them to
`utf8::downgrade`, which is sometimes us to cope with
buggy XS code.
> Without pointers to the right solution, such a warning wouldn't
> be very helpful.
The only correct thing you can do with bytes.pm is mess around
with the byte buffer of an upgraded string while circumventing
Perl’s own fixed-width/variable-width character conversion
machinery. I would guess that of everyone in the world, maybe
5 guts-hacking CPAN authors will ever need to do that. And they
probably know what they are doing anyway.
Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>
Zefram
> I would guess that of everyone in the world, maybe
>5 guts-hacking CPAN authors will ever need to do that.
And there are better (more explicit) ways for them to do it.
-zefram
Tom Hukins
> Very nearly *everything* they would want to do with bytes.pm
> would be a mistake. Using bytes.pm in application code (as
> opposed to guts hackery) is almost by definition a bug.
perlfunc contains the following:
To get the length of the internal string in bytes, use
C<bytes::length(EXPR)>, see L<bytes>.
What should it say instead?
Tom
Zefram
> C<bytes::length(EXPR)>, see L<bytes>.
>
>What should it say instead?
It shouldn't be advertising that facility.
-zefram
Aristotle Pagaltzis
> perlfunc contains the following:
> To get the length of the internal string in bytes, use
> C<bytes::length(EXPR)>, see L<bytes>.
>
> What should it say instead?
I agree with Zefram:
Nothing.
The size of the internal byte buffer is only of interest to
people fiddling directly with perl’s data structures. Telling
application programmers about `bytes::length` is at best
irrelevant and at worst an attractive nuisance. This should be
documented in `perlguts` maybe – but not in `perlfunc`.
David Golden
> The size of the internal byte buffer is only of interest to
> people fiddling directly with perl’s data structures. Telling
> application programmers about `bytes::length` is at best
> irrelevant and at worst an attractive nuisance. This should be
> documented in `perlguts` maybe – but not in `perlfunc`.
Um, just to play devil's advocate here -- let's say I have a really
big chunk of utf8 data in a scalar. I want to know how big it will be
if I write it to disk or send it across the net. Wouldn't I want to
know the size of the internal byte buffer in such a case and not just
the number of characters?
-- David
Zefram
>Um, just to play devil's advocate here -- let's say I have a really
>big chunk of utf8 data in a scalar. I want to know how big it will be
>if I write it to disk or send it across the net. Wouldn't I want to
>know the size of the internal byte buffer in such a case and not just
>the number of characters?
No. Assuming you would write/send the UTF-8 encoding, you want to know
the length of the UTF-8 encoding of the characters. This *might* be
equal to the length of the internal representation, but might be longer
(if the string is internally encoded in Latin-1 and contains non-ASCII
characters). We currently don't have a specialised encoded_length()
function, so you'll have to do length(encode(...)) yourself.
-zefram
David Golden
Got it. *That* is what needs to be explained, then, since a "naive"
programmer like me who doesn't really deal with utf8 very often won't
understand the difference.
David
Eric Brine
A recent discussion on PerlMonks shows that while C<< if (bytes::length($s))
>> is faster than C<< if (length($s)) >>, C<< eq "" >> is faster yet.
Eric Brine
> Um, just to play devil's advocate here -- let's say I have a really
> big chunk of utf8 data in a scalar. I want to know how big it will be
> if I write it to disk or send it across the net. Wouldn't I want to
> know the size of the internal byte buffer in such a case and not just
> the number of characters?
>
>
Change
my $length = bytes::length($s);
utf8::encode $s;
print pack('N', $length), $s;
to
utf8::encode $s;
my $length = length($s);
print pack('N', $length), $s;
Karl Williamson
People on this list have said that bytes.pm is legitimate when
manipulating binary data.
That implies to me that many operations in its scope should warn instead
of the pragma itself.
Aristotle Pagaltzis
I don’t see how. There is no real reason why strings with binary
data should get the UTF8 flag set. But they legitimately might
(eg. some XS code might unconditionally produce upgraded strings,
for ease of implementation). And I cannot think of any case where
*ignoring* the state of the UTF8 flag is the correct thing to do
in application code.
my $img = slurp 'foo.png';
my $copy = $img;
utf8::upgrade($copy);
Now the UTF8 flag on $copy is set but on $img is not.
If you `eq` these strings, you get true. If you print them, both
will produce the same output.
There is no reason to care about whether either of them has the
UTF8 flag set. They are the same string. Internally they may use
different storage formats, but why would you care?
Yet `bytes::length` and other bytes.pm operations will yield
different results for them. Quite simply the `bytes` pragma does
nothing more than force a solid abstraction to leak.
Dr.Ruud
> Very nearly *everything* they would want to do with bytes.pm
> would be a mistake. Using bytes.pm in application code (as
> opposed to guts hackery) is almost by definition a bug.
It is not.
A byte count can be very useful,
for example if you need to stay
within MySQL's max_allowed_packet
when creating big SQL statements.
sub bytes {
require bytes;
bytes::length($_[0]);
}
--
Ruud
Zefram
> A byte count can be very useful,
> for example if you need to stay
> within MySQL's max_allowed_packet
> when creating big SQL statements.
For that purpose, what you need to know is the number of bytes that
you'll be sending to MySQL. This depends on the characters and how
you're encoding them for communication with MySQL, *not* on how Perl
is representing them internally. See previous messages about writing
UTF-8-encoded text to a file.
-zefram
John
I think the problem here is that we think of characters in Perl being
in UTF-8. We need to get away from this idea and think of characters as
*CHARACTERS* they only become bytes when encoded into some other format
for transport out of the program.
So if you want to send UTF-8 to MySQL you need to take you string and
encode it to UTF-8 only then can you get a length of bytes from it.
John
PS This means we need to remove any talk of UTF-8 encoding for Perl
characters from all documentation except the Perl Guts docs.
______________________________________________
This email has been scanned by Netintelligence
http://www.netintelligence.com/email
Marvin Humphrey
> PS This means we need to remove any talk of UTF-8 encoding for Perl
> characters from all documentation except the Perl Guts docs.
I think this is idealistic. In my opinion, it's not practical to use Perl's
Unicode tools without understanding the implementation down to the C
representation, including the SVf_UTF8 flag and how to use Devel::Peek to
snoop it. Unicode glitches are just too hard to debug without such expertise.
Getting UTF-8 into Perl scalars was an awesome hack, but the implementation is
prone to silent failure. I don't think it's practical to harden the system
now, for the same backwards compatibility reasons that it wasn't done in the
first place. So, our only defense will continue to be troubleshooting.
If you're going to purge leakage of the abstraction from the primary
documentation, I think it makes sense to arm users with a tutorial, named,
say, "perldebugunicode", that intentionally goes where the rest of the
documentation does not, covering the SVf_UTF8 flag and Devel::Peek::Dump() and
explaining how to diagnose and solve common problems.
Marvin Humphrey
John
> If you're going to purge leakage of the abstraction from the primary
> documentation, I think it makes sense to arm users with a tutorial, named,
> say, "perldebugunicode", that intentionally goes where the rest of the
> documentation does not, covering the SVf_UTF8 flag and Devel::Peek::Dump() and
> explaining how to diagnose and solve common problems.
>
John
Salvador Ortiz Garcia
Salvador.
Nicholas Clark
Sure. The *documentation* is buggy.
Longer answer:
The internal representation of string data within the perl interpreter
*should* be transparent from Perl space. (Not talking about XS here)
The fact is that right now it isn't. The consensus of the maintainers of
the perl interpreter is that that is a *bug*, and we're working to resolve
it. We're not there yet, and it's likely to take 2 more deprecation cycles
to get there.
Hence documentation answering a question with an approach that will become
deprecated and removed, when a functional approach now exists that will not
become deprecated and removed, is unhelpful documentation, and should be
updated.
Nicholas Clark
Karl Williamson
And the documentation fore utf8::encode is unclear. It doesn't say for
example, what happens if the input string is already in utf8.
Eric Brine
> And the documentation fore utf8::encode is unclear. It doesn't say for
> example, what happens if the input string is already in utf8.
>
It says it encodes the input string, and that's what it does. Are you saying
you want to *add* documentation about the UTF8 flag there? We were just
talking about *eliminating* misleading mentions of the UTF8 flag.
Karl Williamson
utf8::encode($string)
Converts in-place the character sequence to the orresponding octet
sequence in UTF-X. The UTF8 flag is turned off, so that after this
operation, the string is a byte string. Returns nothing.
I remember reading that for the first time and remaining clueless about
why one would want to do that, and what format $string is allowed to be
in. Just above it, for utf8::upgrade, it says that "$string already
encoded as characters does no harm." That could be worded better, by
saying, for example, "... is a no-op", but at least it addresses the
issue. As a naive user, I read the entry for encode(), and contrast it
with the one for upgrade(), and think if $string is already in utf8,
then this probably does harm, and hence I should make sure $string is
not in utf8 before using it.
I gather from this thread that this function is used just before
outputting a string as a stream of utf8-encoded characters. And perhaps
if I had ever had written network interface code in Perl I would divined
the usefulness of this immediately. But to a naive user, such as
myself, why would you convert to utf8 and then pretend it's not?
Eric Brine
> I remember reading that for the first time and remaining clueless about why
> one would want to do that, and what format $string is allowed to be in.
> Just above it, for utf8::upgrade, it says that "$string already encoded as
> characters does no harm." That could be worded better, by saying, for
> example, "... is a no-op", but at least it addresses the issue. As a naive
> user, I read the entry for encode(), and contrast it with the one for
> upgrade(), and think if $string is already in utf8, then this probably does
> harm, and hence I should make sure $string is not in utf8 before using it.
>
ok, that makes more sense. I hadn't read the document top-down like you did.
The distinction between up/downgrade and en/decode could be stronger.
Dr.Ruud
> why would you convert to utf8 and then pretend it's not?
Sometimes you just need the bare octets themselves,
without any interfering functionality.
--
Ruud
Marc Lehmann
> > 2) If we assume we have a good idea what she really wanted to
> > do, we should add pointers to the right way to do it next to
> > the warning.
>
> About the only useful thing I can think of is to point them to
> `utf8::downgrade`, which is sometimes us to cope with
> buggy XS code.
Note that most of this so-called "buggy XS" code is buggy because perl 5.6
redefined the meaning of typemap entries. Before 5.6, char * meant "give me
bytes". after perl 5.6, char * became utterly useless, as perl randomly
passes utf-8 encoded bytes, or bytes to the XS sub, with no way for the xs
sub to differentiate between them.
If perl would be fixed to use pre-5.6 semantics for "char *", all those
"buggy XS modules" would magically *just work*.
XS subs needing to take advantage of higher codepoints already need to use
"SV *" and SvPVutf8*.
This change to the "char *" semantics in 5.6 basiclaly broke all of cpan
that involved xs and char * stuff, basically requirign all those modules
to add many additional lines of code to achieve the same as under 5.005
and earlier.
I think calling that xs code buggy is not goal-oriented. perl should be
fixed to make sense instead.
--
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / p...@goof.com
-=====/_/_//_/\_,_/ /_/\_\
Aristotle Pagaltzis
> If perl would be fixed to use pre-5.6 semantics for "char *",
> all those "buggy XS modules" would magically *just work*.
>
> XS subs needing to take advantage of higher codepoints already
> need to use "SV *" and SvPVutf8*.
>
> This change to the "char *" semantics in 5.6 basiclaly broke
> all of cpan that involved xs and char * stuff, basically
> requirign all those modules to add many additional lines of
> code to achieve the same as under 5.005 and earlier.
>
> I think calling that xs code buggy is not goal-oriented. perl
> should be fixed to make sense instead.
Sure.
I didn’t know the implementation specifics of the problem. All
I know is that quite often I an explicit downgrade is necessary
to make XSUBs work, which means there is a bug – somewhere. For
the purposes of saying “you need utf8::downgrade as a workaround
sometimes”, it doesn’t matter *whose* bug it is. I wasn’t going
to make any claims about a subject where I am clueless.
Your argument about fixing `char *` sounds reasonable to me (as
indeed pretty much all of your arguments sound to me, as far as
the technical side is concerned). However, not every string can
be downgraded, so perl would then have to throw an exception when
trying to invoke an XSUB that uses `char *` when called with
a non-downgradeable string. Is that a good idea?
Zefram
> However, not every string can
>be downgraded, so perl would then have to throw an exception when
>trying to invoke an XSUB that uses `char *` when called with
>a non-downgradeable string. Is that a good idea?
Yes, that's a good idea. An xsub using a "char *" arg type isn't
prepared to receive characters above U+ff. This should cleanly die,
not silently do the wrong thing.
-zefram
Dave Mitchell
> I think these days there's a general consensus that, for normal day-to-day
> tasks, the user has no need for bytes.pm, and indeed, that most naive use
> of it is likely to be harmful.
>
> Given that, should we add a warning to bytes.pm's pod to the effect that
> you shouldn't be using this for new code unless you really know what you're
> code or your understanding of perl's Unicode implementation?
Following the discussion, I've committed the following:
commit a515200d681686a605e69620572e93eba2acd0a4
Author: David Mitchell <da...@iabyn.com>
AuthorDate: Mon Feb 22 15:25:16 2010 +0000
Commit: David Mitchell <da...@iabyn.com>
CommitDate: Mon Feb 22 15:25:16 2010 +0000
deprecate bytes.pm
Affected files ...
M lib/bytes.pm
Differences ...
diff --git a/lib/bytes.pm b/lib/bytes.pm
index b7fabbf..31294cb 100644
--- a/lib/bytes.pm
+++ b/lib/bytes.pm
@@ -1,6 +1,6 @@
package bytes;
-our $VERSION = '1.03';
+our $VERSION = '1.04';
$bytes::hint_bits = 0x00000008;
@@ -33,6 +33,18 @@ __END__
bytes - Perl pragma to force byte semantics rather than character semantics
+=head1 DEPRECATED
+
+This pragma reflects early attempts to incorporate Unicode into perl and
+has since been superceded. It breaks encapulation (i.e. it exposes the
+innards of how perl happens to be currently storing a string), and use of
+this module for anything other than debugging purposes is strongly
+discouraged. If you feel that the functions here within might be useful
+for your application, this possibly indicates a mismatch beteen your
+mental model of Perl Unicode and the current reality. In that case, you
+wish to peruse some of the perl Unicode documentation: L<perluniintro>,
+L<perlunitut>, L<perlunifaq> and L<perlunicode>.
+
=head1 SYNOPSIS
use bytes;
--
Please note that ash-trays are provided for the use of smokers,
whereas the floor is provided for the use of all patrons.
-- Bill Royston
Dave Mitchell
I've now committed the following:
commit 836ccc8ea0b1baa3680e911631ffd126ca7b9966
Author: David Mitchell <da...@iabyn.com>
AuthorDate: Mon Feb 22 16:44:59 2010 +0000
Commit: David Mitchell <da...@iabyn.com>
CommitDate: Mon Feb 22 16:44:59 2010 +0000
Better explain utf8::upgrade/downgrade/encode/decode
Try to make it clear what each does, and the fact that upgrade/downgrade
don't change the logical string value while encode/decode do.
Affected files ...
M lib/utf8.pm
Differences ...
diff --git a/lib/utf8.pm b/lib/utf8.pm
index a985021..ed0a4d8 100644
--- a/lib/utf8.pm
+++ b/lib/utf8.pm
@@ -2,7 +2,7 @@ package utf8;
$utf8::hint_bits = 0x00800000;
-our $VERSION = '1.07';
+our $VERSION = '1.08';
sub import {
$^H |= $utf8::hint_bits;
@@ -32,13 +32,16 @@ utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
use utf8;
no utf8;
- # Convert a Perl scalar to/from UTF-8.
+ # Convert the internal representation of a Perl scalar to/from UTF-8.
+
$num_octets = utf8::upgrade($string);
$success = utf8::downgrade($string[, FAIL_OK]);
- # Change the native bytes of a Perl scalar to/from UTF-8 bytes.
- utf8::encode($string);
- utf8::decode($string);
+ # Change each character of a Perl scalar to/from a series of
+ # characters that represent the UTF-8 bytes of each original character.
+
+ utf8::encode($string); # "\x{100}" becomes "\xc4\x80"
+ utf8::decode($string); # "\xc4\x80" becomes "\x{100}"
$flag = utf8::is_utf8(STRING); # since Perl 5.8.1
$flag = utf8::valid(STRING);
@@ -99,9 +102,10 @@ you should not say that unless you really want to have UTF-8 source code.
=item * $num_octets = utf8::upgrade($string)
-Converts in-place the internal octet sequence in the native encoding
-(Latin-1 or EBCDIC) to the equivalent character sequence in I<UTF-X>.
-I<$string> already encoded as characters does no harm. Returns the
+Converts in-place the internal representation of the string from an octet
+sequence in the native encoding (Latin-1 or EBCDIC) to I<UTF-X>. The
+logical character sequence itself is unchanged. If I<$string> is already
+stored as I<UTF-X>, then this is a no-op. Returns the
number of octets necessary to represent the string as I<UTF-X>. Can be
used to make sure that the UTF-8 flag is on, so that C<\w> or C<lc()>
work as Unicode on strings containing characters in the range 0x80-0xFF
@@ -113,9 +117,11 @@ L<Encode>.
=item * $success = utf8::downgrade($string[, FAIL_OK])
-Converts in-place the internal octet sequence in I<UTF-X> to the
-equivalent octet sequence in the native encoding (Latin-1 or EBCDIC).
-I<$string> already encoded as native 8 bit does no harm. Can be used to
+Converts in-place the the internal representation of the string from
+I<UTF-X> to the equivalent octet sequence in the native encoding (Latin-1
+or EBCDIC). The logical character sequence itself is unchanged. If
+I<$string> is already stored as native 8 bit, then this is a no-op. Can
+be used to
make sure that the UTF-8 flag is off, e.g. when you want to make sure
that the substr() or length() function works with the usually faster
byte algorithm.
@@ -133,8 +139,13 @@ L<Encode>.
=item * utf8::encode($string)
Converts in-place the character sequence to the corresponding octet
-sequence in I<UTF-X>. The UTF8 flag is turned off, so that after this
-operation, the string is a byte string. Returns nothing.
+sequence in I<UTF-X>. That is, every (possibly wide) character gets
+replaced with a sequence of one or more characters that represent the
+individual I<UTF-X> bytes of the character. The UTF8 flag is turned off.
+Returns nothing.
+
+ my $a = "\x{100}"; # $a contains one character, with ord 0x100
+ utf8::encode($a); # $a contains two characters, with ords 0xc4 and 0x80
B<Note that this function does not handle arbitrary encodings.>
Therefore Encode is recommended for the general purposes; see also
@@ -143,10 +154,15 @@ L<Encode>.
=item * $success = utf8::decode($string)
Attempts to convert in-place the octet sequence in I<UTF-X> to the
-corresponding character sequence. The UTF-8 flag is turned on only if
-the source string contains multiple-byte I<UTF-X> characters. If
-I<$string> is invalid as I<UTF-X>, returns false; otherwise returns
-true.
+corresponding character sequence. That is, it replaces each sequence of
+characters in the string whose ords represent a valid UTF-X byte
+sequence, with the corresponding single character. The UTF-8 flag is
+turned on only if the source string contains multiple-byte I<UTF-X>
+characters. If I<$string> is invalid as I<UTF-X>, returns false;
+otherwise returns true.
+
+ my $a = "\xc4\x80"; # $a contains two characters, with ords 0xc4 and 0x80
+ utf8::decode($a); # $a contains one character, with ord 0x100
B<Note that this function does not handle arbitrary encodings.>
Therefore Encode is recommended for the general purposes; see also
--
Hofstadter's Law: It always takes longer than you expect, even when you
take into account Hofstadter's Law.
Jesse Vincent
On Mon 22.Feb'10 at 16:47:36 +0000, Dave Mitchell wrote:
> On Sun, Jan 03, 2010 at 11:27:22PM +0000, Dave Mitchell wrote:
> > I think these days there's a general consensus that, for normal day-to-day
> > tasks, the user has no need for bytes.pm, and indeed, that most naive use
> > of it is likely to be harmful.
> >
> > Given that, should we add a warning to bytes.pm's pod to the effect that
> > you shouldn't be using this for new code unless you really know what you're
> > doing, and that a desire to do so probably indicates a design flaw in your
> > code or your understanding of perl's Unicode implementation?
>
> Following the discussion, I've committed the following:
Based on your explanation, "deprecate" feels like a very strong
statement. It's more of a "This library does unspeakable things to Perl's
understanding of strings. It's unlikely that you want to use it. Instead
have a look at ..."
Or do we actually have a better way to do _everything_ bytes does?
-Jesse
Aristotle Pagaltzis
> Based on your explanation, "deprecate" feels like a very strong
> statement. It's more of a "This library does unspeakable things
> to Perl's understanding of strings. It's unlikely that you want
> to use it. Instead have a look at ..."
I’m not making sense of this. Are you saying “deprecate” doesn’t
feel strong enough for the wording, or are you saying that that
the wording is too strong? Or…?
> Or do we actually have a better way to do _everything_ bytes
> does?
Well, no. We do have correct ways for every misguided use of
bytes.pm and I believe we have better ways for all its correct
uses, but I don’t think we have anything to offer to people with
insane uses. :-)
Ben Morrow
> > does?
>
> Well, no. We do have correct ways for every misguided use of
> bytes.pm and I believe we have better ways for all its correct
> uses, but I don’t think we have anything to offer to people with
> insane uses. :-)
OK, so how do I ensure that certain strings will never be upgraded, or
at any rate will never contain characters > 255? How do I make /\d\s\w/
match ASCII-only? How, in general terms, do I say 'This is 8-bit data so
keep your grubby Unicode hands off it'?
I can certainly see that bytes.pm as currently implemented has serious
bugs, not least being the leaking of the internal representation of the
string. However, I also think the core idea of being able to go back to
the pre-5.8 non-Unicode operations is important, and should be preserved
somehow (preferably in a way that is portable between perl versions,
though I can see that may not be possible).
Ben
Jesse Vincent
On Thu 25.Feb'10 at 3:51:20 +0100, Aristotle Pagaltzis wrote:
> * Jesse Vincent <je...@fsck.com> [2010-02-25 01:50]:
> > Based on your explanation, "deprecate" feels like a very strong
> > statement. It's more of a "This library does unspeakable things
> > to Perl's understanding of strings. It's unlikely that you want
> > to use it. Instead have a look at ..."
>
> I’m not making sense of this. Are you saying “deprecate” doesn’t
> feel strong enough for the wording, or are you saying that that
> the wording is too strong? Or…?
What I was trying to say, on far too little sleep, is that our typical
usage of "deprecate" is "this is no longer supported", often with either
an implicit or explicit suggestion that it might go away.
What I believe we intend with bytes.pm is to tell users who don't know
any better that this is a "bad, deep magic" module and that they likely
want to use something else for most regular use cases.
But we're not saying that bytes.pm is being wrapped up and thrown out.
[There's a meta-point here that we use deprecate to mean a few
different things and that we're doing a lot more of it (for some value
of 'it') than we used to, often on relatively short notice. That makes
me a little twitchy.]
Eric Brine
> Quoth paga...@gmx.de (Aristotle Pagaltzis):
> > * Jesse Vincent <je...@fsck.com> [2010-02-25 01:50]:
> >
> > > Or do we actually have a better way to do _everything_ bytes
> > > does?
> >
> > Well, no. We do have correct ways for every misguided use of
> > bytes.pm and I believe we have better ways for all its correct
> > uses, but I don’t think we have anything to offer to people with
> > insane uses. :-)
>
> OK, so how do I ensure that certain strings will never be upgraded
bytes doesn't affect "certain strings", and it doesn't prevent upgrading. It
just downgrades or encodes everything automatically, and you can do that
(with more predictable results) using utf8::downgrade or utf8::encode.
use strict;
use warnings;
use Test::More tests => 2;
use bytes;
my $x = "abcd\x{E9}fghij";
utf8::upgrade($x);
is($x, "abcd\x{E9}fghij", "prevents upgrade 1"); # pass
$x .= chr(0x2660);
chop($x) for 1..3;
is($x, "abcd\x{E9}fghij", "prevents upgrade 2"); # fail
You lose the automatic aspect by using utf8::encode or utf8::decode, though.
I don't know of anything that does it automatically and predictably.
How do I make /\d\s\w/ match ASCII-only?
use bytes only does that if the input string doesn't contains greater than
8-bit values. utf8:downgrade does the same.
use strict;
use warnings;
use utf8;
use Test::More tests => 2;
my $eacute = "\xE9";
utf8::upgrade( $eacute );
ok($eacute =~ /♠|\w/, "Control");
utf8::downgrade( $eacute );
ok($eacute !~ /♠|\w/, "Test");
> How, in general terms, do I say 'This is 8-bit data so
> keep your grubby Unicode hands off it'?
>
use bytes is lexically scoped, so use bytes will be completely ineffectual
at keeping grubby hands away. It just pretends everything is 8-bit data,
corrupting it if it isn't.
Dave Mitchell
>
>
>
> On Mon 22.Feb'10 at 16:47:36 +0000, Dave Mitchell wrote:
> > On Sun, Jan 03, 2010 at 11:27:22PM +0000, Dave Mitchell wrote:
> > > I think these days there's a general consensus that, for normal day-to-day
> > > tasks, the user has no need for bytes.pm, and indeed, that most naive use
> > > of it is likely to be harmful.
> > >
> > > Given that, should we add a warning to bytes.pm's pod to the effect that
> > > you shouldn't be using this for new code unless you really know what you're
> > > doing, and that a desire to do so probably indicates a design flaw in your
> > > code or your understanding of perl's Unicode implementation?
> >
> > Following the discussion, I've committed the following:
>
> Based on your explanation, "deprecate" feels like a very strong
> statement. It's more of a "This library does unspeakable things to Perl's
> understanding of strings. It's unlikely that you want to use it. Instead
> have a look at ..."
>
> Or do we actually have a better way to do _everything_ bytes does?
The earlier discussion in the thread seemed to reach a rough consensus
that any use (apart from debugging etc) of bytes.pm was a mistake, even
bytes::length.
--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.
karl williamson
We were a little too late for 5.12 in getting in a /a modifier to
regexes to do just that.
>
> use bytes only does that if the input string doesn't contains greater than
> 8-bit values. utf8:downgrade does the same.
>
> use strict;
> use warnings;
> use utf8;
> use Test::More tests => 2;
> my $eacute = "\xE9";
> utf8::upgrade( $eacute );
> ok($eacute =~ /♠|\w/, "Control");
> utf8::downgrade( $eacute );
> ok($eacute !~ /♠|\w/, "Test");
>
>
>
>> How, in general terms, do I say 'This is 8-bit data so
>> keep your grubby Unicode hands off it'?
>>
>
> use bytes is lexically scoped, so use bytes will be completely ineffectual
> at keeping grubby hands away. It just pretends everything is 8-bit data,
> corrupting it if it isn't.
>
Once "use feature unicode_strings" is fully implemented, there will be
much less need to upgrade it, so many of those cases of upgrading can be
removed, for efficiency's sake, and this will improve this situation as
a side effect.
Ben Morrow
> On Wed, Feb 24, 2010 at 11:00 PM, Ben Morrow <b...@morrow.me.uk> wrote:
> > Quoth paga...@gmx.de (Aristotle Pagaltzis):
> > > * Jesse Vincent <je...@fsck.com> [2010-02-25 01:50]:
> > >
> > > > Or do we actually have a better way to do _everything_ bytes
> > > > does?
> > >
> > > Well, no. We do have correct ways for every misguided use of
> > > bytes.pm and I believe we have better ways for all its correct
> > > uses, but I don’t think we have anything to offer to people with
> > > insane uses. :-)
> >
> > OK, so how do I ensure that certain strings will never be upgraded
>
> bytes doesn't affect "certain strings", and it doesn't prevent upgrading.
I know. I'm not arguing that bytes.pm isn't broken, I'm arguing that it
needs to be fixed rather than thrown away.
FWIW, if you are writing a short script to process binary data (a not
uncommon occurrence), bytes *will* prevent anything from getting
upgraded automatically. It doesn't protect you against modules upgrading
things behind your back, but that's one of the things that's broken.
> It
> just downgrades or encodes everything automatically, and you can do that
> (with more predictable results) using utf8::downgrade or utf8::encode.
>
> use strict;
> use warnings;
> use Test::More tests => 2;
> use bytes;
> my $x = "abcd\x{E9}fghij";
> utf8::upgrade($x);
> is($x, "abcd\x{E9}fghij", "prevents upgrade 1"); # pass
> $x .= chr(0x2660);
> chop($x) for 1..3;
> is($x, "abcd\x{E9}fghij", "prevents upgrade 2"); # fail
>
> You lose the automatic aspect by using utf8::encode or utf8::decode, though.
> I don't know of anything that does it automatically and predictably.
Yes. That's much too fussy to be useful.
I'm increasingly thinking that's what needed here is blob magic type,
that stops given strings from ever being upgraded. I'm not certain
that's right though, since I think it falls foul of 'polymorphic data
types but not polymorphic operators', which I believe is the reason
bytes was implemented as a lexically-scoped pragma in the first place.
> How do I make /\d\s\w/ match ASCII-only?
>
>
> use bytes only does that if the input string doesn't contains greater than
> 8-bit values. utf8:downgrade does the same.
...but IIUC Karl and Yves are (for good reason) trying to change things
so that isn't the case, and ISO8859-1 letters match as well as ASCII. Of
course, if we get an /a regex flag, and some sort of re::default_flags
pragma, then bytes could use that.
> > How, in general terms, do I say 'This is 8-bit data so
> > keep your grubby Unicode hands off it'?
> >
>
> use bytes is lexically scoped, so use bytes will be completely ineffectual
> at keeping grubby hands away. It just pretends everything is 8-bit data,
> corrupting it if it isn't.
So it needs fixing, perhaps by throwing an exception if any bytes-ified
operator gets passed a string with codepoints >255. At the very least it
should be able to transparently convert SvUTF8 strings that would in
fact fit into bytes.
Ben
Abigail
>
>
> [There's a meta-point here that we use deprecate to mean a few
> different things and that we're doing a lot more of it (for some value
> of 'it') than we used to, often on relatively short notice. That makes
> me a little twitchy.]
It's even worse outside of p5p. The other day, someone was blogging
(triggered by the birth of cpanminus) that it took a few years for
CPANPLUS to enter the core, but that now CPAN was "officially deprecated".
It seems that were "deprecated" first meant "consider this a warning, this
construct may go away without further notice in some future version of Perl",
it slowly gets the meaning "this construct is sooo last week, there's a
different colour of kool-aid out".
Can we deprecate the usage of deprecate? ;-)
Abigail
Nicholas Clark
> On Wed, Feb 24, 2010 at 06:21:22PM -1000, Jesse Vincent wrote:
> >
> >
> > [There's a meta-point here that we use deprecate to mean a few
> > different things and that we're doing a lot more of it (for some value
> > of 'it') than we used to, often on relatively short notice. That makes
> > me a little twitchy.]
>
>
> It's even worse outside of p5p. The other day, someone was blogging
> (triggered by the birth of cpanminus) that it took a few years for
> CPANPLUS to enter the core, but that now CPAN was "officially deprecated".
But what does Wikipedia say about it?
How about now?
> It seems that were "deprecated" first meant "consider this a warning, this
> construct may go away without further notice in some future version of Perl",
> it slowly gets the meaning "this construct is sooo last week, there's a
> different colour of kool-aid out".
> Can we deprecate the usage of deprecate? ;-)
Sadly, no more that we can deprecate stupid.
Nicholas Clark
PS How about now?
jesse
> > Based on your explanation, "deprecate" feels like a very strong
> > statement. It's more of a "This library does unspeakable things to Perl's
> > understanding of strings. It's unlikely that you want to use it. Instead
> > have a look at ..."
> >
> > Or do we actually have a better way to do _everything_ bytes does?
>
> The earlier discussion in the thread seemed to reach a rough consensus
> that any use (apart from debugging etc) of bytes.pm was a mistake, even
> bytes::length.
>
That wasn't quite how I read "day to day use" - regardless, it ain't
being deprecated for 5.12. We're well into code freeze and the time for
picking new things that are no longer supported as of 5.12.
I'm absolutely thrilled with the rest of the prose. It's just the
"deprecated" bit that's got me cranky and needs to get backed out or
softened.
>
> --
> A walk of a thousand miles begins with a single step...
> then continues for another 1,999,999 or so.
>
--
Dave Mitchell
> I'm absolutely thrilled with the rest of the prose. It's just the
> "deprecated" bit that's got me cranky and needs to get backed out or
> softened.
commit 490aa361e2910b6b451cc8bceccd93efa8fbed11
Author: David Mitchell <da...@iabyn.com>
AuthorDate: Fri Feb 26 12:50:21 2010 +0000
Commit: David Mitchell <da...@iabyn.com>
CommitDate: Fri Feb 26 12:50:21 2010 +0000
bytes.pm: downgrade DEPRECATED to NOTICE
Affected files ...
M lib/bytes.pm
Differences ...
diff --git a/lib/bytes.pm b/lib/bytes.pm
index 8065a04..57916a5 100644
--- a/lib/bytes.pm
+++ b/lib/bytes.pm
@@ -33,7 +33,7 @@ __END__
bytes - Perl pragma to force byte semantics rather than character semantics
-=head1 DEPRECATED
+=head1 NOTICE
This pragma reflects early attempts to incorporate Unicode into perl and
has since been superseded. It breaks encapsulation (i.e. it exposes the
--
Spock (or Data) is fired from his high-ranking position for not being able
to understand the most basic nuances of about one in three sentences that
anyone says to him.
-- Things That Never Happen in "Star Trek" #19
Dr.Ruud
> Can we deprecate the usage of deprecate? ;-)
Do I read a nomination?
--
Ruud
Reini Urban
> commit 490aa361e2910b6b451cc8bceccd93efa8fbed11
> Author: David Mitchell <da...@iabyn.com>
> AuthorDate: Fri Feb 26 12:50:21 2010 +0000
> Commit: David Mitchell <da...@iabyn.com>
> CommitDate: Fri Feb 26 12:50:21 2010 +0000
>
> bytes.pm: downgrade DEPRECATED to NOTICE
Thanks!
--
Reini Urban
Jesse Vincent
"Dave Mitchell" <da...@iabyn.com> wrote:
--
Sent from my Android phone with K-9. Please excuse my brevity.
Aristotle Pagaltzis
> If you're going to purge leakage of the abstraction from the
> primary documentation, I think it makes sense to arm users with
> a tutorial, named, say, "perldebugunicode", that intentionally
> goes where the rest of the documentation does not, covering the
> SVf_UTF8 flag and Devel::Peek::Dump() and explaining how to
> diagnose and solve common problems.
Absolutely.
In the grand tradition of perldoc, I’d call it `perluniguts`
maybe – a guide to the gory details of the implementation that
are rarely relevant to day to day business. Of course, not as
rarely as with other guts, unfortunately:
> Getting UTF-8 into Perl scalars was an awesome hack, but the
> implementation is prone to silent failure. I don't think it's
> practical to harden the system now, for the same backwards
> compatibility reasons that it wasn't done in the first place.
> So, our only defense will continue to be troubleshooting.
I think by far the primary reason that “purifying” the docs won’t
be easy now is simply because the documentation itself has been
giving people the wrong ideas for such a long time, resulting in
a lot of broken code whose problems users now need to be able to
troubleshoot and work around. That doesn’t necessitate that the
documentation stay broken, though.
karl williamson
where it is wrong, and I will fix them.
Aristotle Pagaltzis
> I thought I had purified the docs for 5.12. Please point out
> examples where it is wrong, and I will fix them.
I haven’t looked at the 5.12 doc updates yet. It’s such a big job
I’d assume you probably haven’t cleaned up everything. But if you
have, so much the better.