PATCH: partial [perl #58182]: regex case-sensitive matching now utf8ness independent
Karl Williamson
them up.
The patch is both attached, and available at:
git://github.com/khwilliamson/perl.git
branch: matching
This patch makes case-sensitive regex matching give the same results
regardless of whether the string and/or pattern are in utf8, unless "use
legacy 'unicode8bit'" is in effect, in which case it works as before.
Since Yves is incommunicado, I took what he had done before Larry's veto
and extended and modified it, adding an intermediate way. What that
means is that anything that looks like[[:xxx:]] will match only in the
ASCII range, or in the current locale, if set. I never heard any
controversy about that part of the proposal, and it makes sense to me
that a Posix construct should act like the Posix definition says to.
\d, \s, and \w (hence \b) and their complements act as before, except
that when 8-bit unicode mode is on, they also match appropriately in the
128-255 range.
This solves the utf8ness problem, as the Posix never match outside their
locale or ascii, so utf8ness doesn't matter; and the others match the
same whether utf8 or not.
I was surprised at actually how little code was involved. Making Posix
always mean Posix simplified things quite a bit. \d doesn't match
anything in the 128-255 range, so it did not have to be touched.
Essentially, all that had to be done was to create new regnodes for \s,
\w, and \b (and complements) that say to match using unicode semantics.
Everywhere their parallel nodes are in the code, I added these nodes.
When compiling, regcomp checks for being in 8-bit unicode semantics
mode, and if so, uses the new node; if not it uses the old node. In
execution, regexec uses the old definition when matching the old node,
and the new semantics when the match is for the new node. I split
[[:word:]] from \w and [[:digit:]] from \d so that they would match
using Posix semantics regardless of utf8ness.
But that is basically it.
Several .t files depended on the legacy behaviors to test edge cases for
utf8ness. I added a 'use legacy' to those.
Also, several text processing modules can't deal with \s matching a
no-break space. I spent too much time trying to learn them to decide if
this is a bug or not, finding the one or two lines in each that were at
fault. It is a bug if the text can be utf8, which would automatically
cause the \s to suddenly match the no-break space. But I wasn't sure
which ones are claimed to transparently handle utf8. So, I added a 'use
legacy' to the modules, which gives the same behavior as in the past.
Several TODOs were accomplished and removed from some regex .t files
I took advantage of changing regcomp.c to add a croak when the re has
gone insane; I've had it in my development version for some time. It
seems to happen when there are too many /\N{...}/ calls in a program.
Demerphq
> I believe this resolves other bug reports, but haven't had time to look them
> up.
>
> The patch is both attached, and available at:
> git://github.com/khwilliamson/perl.git
> branch: matching
>
> This patch makes case-sensitive regex matching give the same results
> regardless of whether the string and/or pattern are in utf8, unless "use
> legacy 'unicode8bit'" is in effect, in which case it works as before.
>
> Since Yves is incommunicado,
I was heads-uped about this mail, but I've not had any time to respond
yet. Sorry.
> I took what he had done before Larry's veto and
> extended and modified it, adding an intermediate way. What that means is
> that anything that looks like[[:xxx:]] will match only in the ASCII range,
> or in the current locale, if set. I never heard any controversy about that
> part of the proposal, and it makes sense to me that a Posix construct should
> act like the Posix definition says to.
This is good IMO, it will allow us to close a number of open tickets.
I had a quick review of the patch and what you have done.
I have two minor objections, but i dont think they need be seen as roadblocks.
First, the problem of qr// raises its head. You construct a pattern
one context with your new pragma in effect, and then embed it in
another pattern somewhere else and the magicness of the pattern is
lost. This is the same problem as with use locale, and personally
something I think breaks the general modern model of patterns. However
it is better than nothing and modifiers can be leveraged on top of
your patch so that is fine IMO.
Second, and really this is just another facet of the original problem
is that people now need to modify existing code to preserve the
existing semantics. If this was controlled by modifier then this
wouldnt be necessary as we would just make the default modifier behave
as in 5.8.x, also if really necessary we could bifurcate the POSIX
stuff into multiple opcodes (old/new behaviour) and resolve any
objections to fixing the POSIX opcodes.
However my opinion is this is a really good step forward and should be
applied to blead.
cheers,
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Zefram
>First, the problem of qr// raises its head. You construct a pattern
>one context with your new pragma in effect, and then embed it in
>another pattern somewhere else and the magicness of the pattern is
>lost.
Or vice versa, of course. This means that for a pattern to be freely
embeddable you need to avoid all the constructs whose behaviour depend
on the pragma. This is no worse than the current situation, where you
need to avoid those constructs because they're broken, but for that class
of situation it's no better either. And how many people are actually
going to think about the pragma when they write their exportable regexps?
The existence of a pragma with this class of effects is a problem. I also
don't think we should be supporting buggy behaviour by any mechanism.
The old behaviour isn't just legacy, it's a bug, and any code that
invokes it is buggy. Please just fix the bug, don't bless it.
-zefram
Jesse
Thank you very much for all your work on this. I'll admit that the patch
is a bit more extensive than I'd anticipated.
At a basic procedural level, any module that's inside cpan/ really needs
to be patched upstream in the relevant CPAN distribution and then pulled
into blead as they're released to CPAN.
That highlights my base concern here -- As of right now, if those
changes were pushed into the CPAN distributions, they wouldn't run on
any Perl release before 5.11.2.
That issue is fixable if we CPAN a version of "feature.pm" designed for
older releases of Perl, but it gets increasingly hard to keep "legacy"
directives synced between the differing realities of different Perls'
concepts of "legacy".
Talking to Nicholas about my concerns, he suggested that many of these
problems would go away if legacy directives always defaulted to enabled.
I know that a number of folks are eager to jettison historical designs
that are now considered to have been mistakes, but intentionally
breaking backwards-compatibility by inverting default behavior isn't
the right thing for us to doing.
Would you be comfortable with flopping the 'unicode8bit' legacy default
such that users who want the new semantics would use something like:
"no legacy 'unicode8bit'"
or "use feature 'unicode8bit';
or "use feature ':5.12';
Thanks,
Jesse
> From 65f96077a5c64ea2ebaa200194782540c112fd8d Mon Sep 17 00:00:00 2001
> From: Karl Williamson <khw@khw-desktop.(none)>
> Date: Wed, 9 Dec 2009 11:25:36 -0700
> Subject: [PATCH] regex case-sensitive match utf8ness independent
>
> ---
> cpan/Pod-Simple/lib/Pod/Simple/BlackBox.pm | 4 +-
> cpan/Test-Harness/lib/TAP/Parser/YAMLish/Reader.pm | 1 +
> cpan/podlators/lib/Pod/Text.pm | 1 +
> cpan/podlators/lib/Pod/Text/Color.pm | 1 +
> cpan/podlators/lib/Pod/Text/Overstrike.pm | 1 +
> cpan/podlators/lib/Pod/Text/Termcap.pm | 1 +
> dist/Storable/t/downgrade.t | 6 +-
> ext/POSIX/t/time.t | 1 +
> handy.h | 13 +
> lib/legacy.t | 127 +++++++++-
> regcomp.c | 269 +++++++++++++++-----
> regcomp.h | 53 +++--
> regcomp.sym | 19 +-
> regexec.c | 176 ++++++++++----
> regnodes.h | 54 +++-
> t/op/sysio.t | 6 +-
> t/re/pat_special_cc.t | 1 +
> t/re/re_tests | 2 +-
> t/re/reg_posixcc.t | 32 ++--
> 19 files changed, 604 insertions(+), 164 deletions(-)
Nicholas Clark
> Talking to Nicholas about my concerns, he suggested that many of these
> problems would go away if legacy directives always defaulted to enabled.
With a slightly more "exciting" view, that I don't know Jesse's opinion on,
that the intent is that specific legacy behaviours will change, by in the
next major release warning, and the one after that not being the default.
Although thinking more about that, it means that "legacy" would be slightly
hairy, in that the default enabled subset would be different on different
(future) major releases. I don't know if that's too hard to explain and teach.
Nicholas Clark
Demerphq
I am firmly convinced the only sane solution is modifiers.
Jesse
I very much liked your modifier plan when you described it to me. I _can_
see a desire to update Perl's default semantics for this or some other
core feature. There are certainly historical decisions that I'd love to
see rectified, as they are clearly bugs. What I worry about at night[1]
is breaking bugward-compatibility. I don't want 20% of CPAN breaking
needlessly on my watch. If we're going to "fix" default semantics, it is
imperative that users need to declare a desire for the new behavior.
-Jesse
[1] Terrifyingly, the dream I remember having as I woke up this morning
was about release engineering 5.12.0. Really.[2]
[2] Does this job come with health insurance? Does that insurance
include coverage for psychological care or psychoactive medication?
Karl Williamson
> [snip]
>
> These "posix" constructs have for a long time been documented as
> *equivalent* to \d, \s and \w, with two remarks: [[:space:]] also
> includes \cK and [[:word:]] doesn't even exist in POSIX.
>
> Changing them is as bad as changing the metacharacters. Changing them to
> break the equivalency might even be worse.
>
> Also, note that perlre calls this "POSIX character class **syntax**"
> (emphasis mine).
>
> An even stronger argument is that perlre defines equivalence with
> \p{...}, and explicitly mentions that these are Unicode constructs.
Just so everything is exposed, an argument the other way is that Punct
is very different between Posix and Unicode, in the ASCII range.
Karl Williamson
Well clearly there is some controversy now that I had not anticipated.
I can empathize, Jesse. I have been a project manager for a few
projects. Even when I was working with a bunch of programmers who were
getting paid a lot of money to do the "right thing", and I knew them and
their work pretty well, it was nerve wracking, especially as the release
date crept, nay galloped, up. I can only imagine how much worse it is
when you don't really know these volunteer participants.
I've come out of retirement to work on this glaring hole in Perl wrt
Unicode. It was bigger than I imagined. And I would like to see it
fixed, yesterday. Also, anecdotally, I ran into a friend, actually an
ex-coworker of mine, who is a linguist, and when I told him what I was
doing, he got excited. He'd tried before, and given up working with
Perl on Unicode. So now he wants me to tell him as soon as this is
available.
That said, it is more important to not destabilize existing code. I
don't know what the right thing is for 5.12.
Jesse
> Well clearly there is some controversy now that I had not anticipated.
Indeed. I apologize if I was overly enthusiastic before I understood the
destabilizing impact.
> date crept, nay galloped, up. I can only imagine how much worse it is
> when you don't really know these volunteer participants.
No worries. My footnotes were intended as humor rather than as a cry of
pain. (Though I really did dream about 5.12 release engineering last
night.)
> I've come out of retirement to work on this glaring hole in Perl wrt
> Unicode. It was bigger than I imagined. And I would like to see it
> fixed, yesterday.
I'd love to see it fixed ten years ago. That said, I _am_ absolutely
thrilled that it's getting fixed. I just want to make sure that we don't
hurt lots of people as we do it.
> That said, it is more important to not destabilize existing code. I
> don't know what the right thing is for 5.12.
To reiterate my current thinking:
* I like the control that Yves' per-regex modifiers give us.
* I don't want to break users' legacy Perl code.
* I really do like the correct semantics you've gotten working.
Yves modifiers will give us fine-grained control but won't eliminate the
need for good defaults.
It really does sound like the small tweak to the default legacyness
value that Nicholas suggested would eliminate the contention about
default behavior. It would mean that new code that declares it
wants new semantics would get them and code that says nothing is
expecting the traditional behaviour.
How's that sound?
-Jesse
--
Karl Williamson
I'm not sure I follow this. I think what you're saying is that the
original pattern is decompiled or thrown away and then recompiled under
the new scheme?
>
> Second, and really this is just another facet of the original problem
> is that people now need to modify existing code to preserve the
> existing semantics. If this was controlled by modifier then this
> wouldnt be necessary as we would just make the default modifier behave
> as in 5.8.x, also if really necessary we could bifurcate the POSIX
> stuff into multiple opcodes (old/new behaviour) and resolve any
> objections to fixing the POSIX opcodes.
One should be able to change the default modifier, I would hope.
Demerphq
> demerphq wrote:
>> I have two minor objections, but i dont think they need be seen as
>> roadblocks.
>>
>> First, the problem of qr// raises its head. You construct a pattern
>> one context with your new pragma in effect, and then embed it in
>> another pattern somewhere else and the magicness of the pattern is
>> lost. This is the same problem as with use locale, and personally
>> something I think breaks the general modern model of patterns. However
>> it is better than nothing and modifiers can be leveraged on top of
>> your patch so that is fine IMO.
>
> I'm not sure I follow this. I think what you're saying is that the original
> pattern is decompiled or thrown away and then recompiled under the new
> scheme?
Yes. Essentially that is how embedding a qr// object into another pattern works.
Basically its like a C include. When you do:
my $qr1= qr/this is a pattern/;
my $qr2= qr/this is a pattern containing another pattern $qr1/;
the *source* of $qr1 is embedded in $qr2, not the opcodes. So any
behaviour that is controlled that by pragmatta and not by in-regex
modifiers will be lost. This is why the /msix modifiers have (?msix:
... ) forms.
>>
>> Second, and really this is just another facet of the original problem
>> is that people now need to modify existing code to preserve the
>> existing semantics. If this was controlled by modifier then this
>> wouldnt be necessary as we would just make the default modifier behave
>> as in 5.8.x, also if really necessary we could bifurcate the POSIX
>> stuff into multiple opcodes (old/new behaviour) and resolve any
>> objections to fixing the POSIX opcodes.
>
> One should be able to change the default modifier, I would hope.
Yes, i was thinking that a good plan would be to just define a generic
interface for specifying default modifiers. The people that like to
follow PBP recommendation of using /msx always can use the pragma.
Gerard Goossen
>
> That issue is fixable if we CPAN a version of "feature.pm" designed for
> older releases of Perl, but it gets increasingly hard to keep "legacy"
> directives synced between the differing realities of different Perls'
> concepts of "legacy".
Publishing "legacy" shouldn't be too much of a problem. If we add for
each directive, a version number in which it is introduced and
optional a version number where the directive is stopped being
supported, using this to decide what to do would be easy.
Gerard Goossen
Gerard Goossen
would be improved by chaning the semantics, and thus instead of
thinking about possibly breaking 20% of CPAN we are fixing 80% of
CPAN.
If we want this to be the default at any time in the future, we should
do it now, because I don't see how having another release cycle would
change anything.
More specific about the failures caused by the changes:
The pod stuff is breaking because it expect a non-breakable-space to
be matched by \s, as far as I know it is about the only module
expecting this behaviour (which is probably broken because it
currently depends on the utf8-ness of the scalar). I did a similar
change in Perl Kurila and what I remember is that only the pod module
had problems with it. I'll check whether I can find the changes to the
pod module, which make them work without using the "use legacy
'unicode8bit'".
I am suprised at the failure of Test::Harness, if anything I would
expect it to fix it, looking at ...\YAMList\Reader.pm it uses \s to
match space characters, but according to YAML a non-breaking-space
isn't a space (and thus it would be part of 80% of CPAN which
would be fixed by the change).
Karl: could you find out why it fails? I suspect that there is
something having some (unwanted) side effect (which probably isn't
wrong or shouldn't have any effect on code, but might be easily
prevented).
Another class of failures are those that depend on the current
behaviour to test the internals, like the POSIX/t/time.t test, which
uses the current behaviour to test that utf8-flag is not set, this is
simply broken, and it should simply use utf8::is_utf8.
Gerard Goossen
Demerphq
> What I am missing in the dicussion is that on average exists code
> would be improved by chaning the semantics, and thus instead of
> thinking about possibly breaking 20% of CPAN we are fixing 80% of
> CPAN.
>
Yes I agree.
And those crying "yeah but it was documented to behave like X so
changing it is bad" have to accept that it DOESNT work like "X" and
CANT work like "X" without being buggy. Also, in many cases relating
to this subject the docs are just wrong, or misleading or whatever.
However it should be remembered that going from the non-unicode world
to the unicode one involves breaking a lot previous system invariants
(aka laws of the universe). For instance in ascii you can always
assume that you can put a case modified version of a string in the
same storage as its original, this is most definitely NOT safe in
unicode and will/can/could result in buffer overruns as a consequence.
Thats just a start.
Juerd Waalboer
> Since Yves is incommunicado, I took what he had done before Larry's veto
> and extended and modified it, adding an intermediate way. What that
> means is that anything that looks like[[:xxx:]] will match only in the
> ASCII range, or in the current locale, if set. I never heard any
> controversy about that part of the proposal, and it makes sense to me
> that a Posix construct should act like the Posix definition says to.
These "posix" constructs have for a long time been documented as
*equivalent* to \d, \s and \w, with two remarks: [[:space:]] also
includes \cK and [[:word:]] doesn't even exist in POSIX.
Changing them is as bad as changing the metacharacters. Changing them to
break the equivalency might even be worse.
Also, note that perlre calls this "POSIX character class **syntax**"
(emphasis mine).
An even stronger argument is that perlre defines equivalence with
\p{...}, and explicitly mentions that these are Unicode constructs.
--
Met vriendelijke groet, Kind regards, Korajn salutojn,
Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig>
Convolution: ICT solutions and consultancy <sa...@convolution.nl>
Demerphq
> karl williamson skribis 2009-12-09 12:11 (-0700):
>> Since Yves is incommunicado, I took what he had done before Larry's veto
>> and extended and modified it, adding an intermediate way. What that
>> means is that anything that looks like[[:xxx:]] will match only in the
>> ASCII range, or in the current locale, if set. I never heard any
>> controversy about that part of the proposal, and it makes sense to me
>> that a Posix construct should act like the Posix definition says to.
>
> These "posix" constructs have for a long time been documented as
> *equivalent* to \d, \s and \w, with two remarks: [[:space:]] also
> includes \cK and [[:word:]] doesn't even exist in POSIX.
*mis*documented.
And, [[:word:]] is spelled [[:alnum:]].
>
> Changing them is as bad as changing the metacharacters. Changing them to
> break the equivalency might even be worse.
I very very very much doubt it, and consider this to be essentially FUD.
Especially as it fixes a stack of bugs related to their behaviour now.
You cannot have both the current behaviour and non buggy implementation.
Simply put I consider that:
[^STUFF] matching the same code points as [STUFF] to be an irrefutable
and overwhelming reason why the current behavior of POSIX charclass
cannot be preserved.
Essentially for me this bug ends ANY debate on this particular issue.
Had we known of this violation of the rules we NEVER would have
allowed this to escape in the wild.
> Also, note that perlre calls this "POSIX character class **syntax**"
> (emphasis mine).
>
> An even stronger argument is that perlre defines equivalence with
> \p{...}, and explicitly mentions that these are Unicode constructs.
*mis*documented as equivalent.
At least one of the "equivalencies" was *never* true, and the other
equivalencies were by breaking unicode rules to be more perl like.
Demerphq
> demerphq skribis 2009-12-10 13:23 (+0100):
>> And, [[:word:]] is spelled [[:alnum:]].
>
> 1
>
> See perlre
See regexec.c and regcomp.c for the source of our mutual confusion.
>> You cannot have both the current behaviour and non buggy implementation.
>
> Fully agreed. That's certainly not what I'm after, either.
>
>> Simply put I consider that:
>> [^STUFF] matching the same code points as [STUFF] to be an irrefutable
>> and overwhelming reason why the current behavior of POSIX charclass
>> cannot be preserved.
>
> What exactly do you mean by "current behaviour"?
>
> To fix the issue that codepoints 128..255 are included depending on
> internal encoding, there are two options:
>
> - Ignore anything above 127
> - Provide full unicode semantics.
>
> The first, ASCII-only, would be a mistake.
No it wouldnt. There are no "unicode semantics" for POSIX.
It is a fundamental error to speak of there being any.
> Perhaps there is other current behaviour that I am not aware of.
Apparently my hint wasnt strong enough.
Try matching all the legal codepoints against [^POSIX] and against [POSIX]
And note all the cases where you have both matching. Then do it with
the strings in unicode. Note all the errors.
These are fundamental errors.
For me this debate is over, POSIX charclasses are not Unicode
charclasses and any contortion to try to make them so is futile and
doomed to screw stuff over.
Zefram
>And, [[:word:]] is spelled [[:alnum:]].
No, they're different. /\w/ and /[[:word:]]/ match "_", whereas
/[[:alnum:]]/ does not.
-zefram
Demerphq
Yes, mea-culpa. They are both called ALNUM internally, well [[:word:]]
is internally called ALNUM, and [[:alnum:]] is helpfully called
ALNUMC.
Sigh.
yves
John
> karl williamson skribis 2009-12-09 12:11 (-0700):
>
>> Since Yves is incommunicado, I took what he had done before Larry's veto
>> and extended and modified it, adding an intermediate way. What that
>> means is that anything that looks like[[:xxx:]] will match only in the
>> ASCII range, or in the current locale, if set. I never heard any
>> controversy about that part of the proposal, and it makes sense to me
>> that a Posix construct should act like the Posix definition says to.
>>
>
> These "posix" constructs have for a long time been documented as
> *equivalent* to \d, \s and \w, with two remarks: [[:space:]] also
> includes \cK and [[:word:]] doesn't even exist in POSIX.
>
> Changing them is as bad as changing the metacharacters. Changing them to
> break the equivalency might even be worse.
>
> Also, note that perlre calls this "POSIX character class **syntax**"
> (emphasis mine).
>
> An even stronger argument is that perlre defines equivalence with
> \p{...}, and explicitly mentions that these are Unicode constructs.
>
name} and not just limit ourselves to the names list?
______________________________________________
This email has been scanned by Netintelligence
http://www.netintelligence.com/email
Karl Williamson
> demerphq skribis 2009-12-10 14:11 (+0100):
>> See regexec.c and regcomp.c for the source of our mutual confusion.
>
> probably have been much more specific when suggesting fixes, from the
> beginning.
>
>>> The first, ASCII-only, would be a mistake.
>> No it wouldnt. There are no "unicode semantics" for POSIX.
>
>
> However, the question of whether [[:xxx:]] is POSIX-like syntax, or an
> actual POSIX character class, remains unanswered or at least unclear.
>
> Certainly Perl's documentation isn't fully definitive. perlre mentions:
>
> 1 => "POSIX character class syntax"
> 2 => "POSIX character classes"
> 3 => Equivalences to \p{} Unicode constructs
>
> 1 and 3 can both be true, but then 2 is not. This is how I (prefer to)
> think of it.
>
> However, it could also be that 1 and 2 are true, ruling 3 out. If I
> understand correctly, that's how you see the matter.
>
> The mere existence of exceptions to the POSIX standard in [:xxx:], and
> the exclusion of [.xxx.] and [=xxx=] lead me to believe that it's just
> syntax compatibility, and Perl is free to extend the class definitions
> to meet more modern requirements, like acknowledging that � is indeed
> alphanumeric. Even if the people who invented the original POSIX bracket
> expressions failed to notice.
>
>> Try matching all the legal codepoints against [^POSIX] and against [POSIX]
>> And note all the cases where you have both matching. Then do it with
>> the strings in unicode. Note all the errors.
>
> do try this. Would have saved me some time and failure, because I was
> unable to reproduce the errors.
>
> perl -CO -le'(chr($_) =~ /[[:alnum:]]/) and (chr($_) =~ /[^[:alnum:]]/)
> and warn sprintf "U+%04x (%s)\n", $_, chr for 1..65000'
>
> doesn't give me anything. It is likely, however, that I misinterpreted
> your instructions. (Note: I have no idea which codepoints qualify as
> legal for this purpose, so I used the arbitrary limit of 65000.)
>
>> For me this debate is over, POSIX charclasses are not Unicode
>> charclasses and any contortion to try to make them so is futile and
>> doomed to screw stuff over.
>
> almost a decade. You suggest going back, I suggest going forward.
> Unfortunately I don't understand the points you're making, except the
> one about POSIX simply not having any notion of unicode. I'm okay with
> a change that makes Perl's [:x:] charclasses fully POSIX compliant, but
> then it needs to be done rigourously, and all Perl exceptions have to be
> eradicated. This should then not be seen as a fix of any unicode bug,
> but as a design/semantics change.
If you run the attached test file that I believe Yves wrote, you can
find some of the errors. That said, I do believe that given enough
work we could fix things so that these posix-like constructs seamlessly
match above the ASCII range without matching a class and its complement
simultaneously (we've invented quantum particle regexes!) BUT with
several exceptions. One is locale (and EBCDIC machines, should they
ever use Perl again). These constructs are supposed to match in the
given locale. The parallel \p ones do not. The other exceptions are
that the Perl/Unicode definitions don't match the Posix definitions for
two of the constructs. [[:Word:]] is a perl extension, and hence not
relevant here; but the Punct differs significantly, and the Space
differs in one character, as Juerd pointed out.
It was our intention that 5.12 would use strict Posix definitions
rigourously for all these, except the perl made-up extension,
[[:Word:]], which has no Posix definition. So I think you said you were
OK with that. In a sense it is a partial fix the Unicode bug because it
means that these cases will no longer have different semantics if the
internal representation changes.
Juerd Waalboer
juerd@lanova:~$ perl -le'print "foo" =~ /[[:word:]]/'
1
See perlre
> You cannot have both the current behaviour and non buggy implementation.
Fully agreed. That's certainly not what I'm after, either.
> Simply put I consider that:
> [^STUFF] matching the same code points as [STUFF] to be an irrefutable
> and overwhelming reason why the current behavior of POSIX charclass
> cannot be preserved.
What exactly do you mean by "current behaviour"?
To fix the issue that codepoints 128..255 are included depending on
internal encoding, there are two options:
- Ignore anything above 127
- Provide full unicode semantics.
The first, ASCII-only, would be a mistake.
Perhaps there is other current behaviour that I am not aware of.
Juerd Waalboer
Unfortunately I don't speak C. If I understood Perl's source, I would
probably have been much more specific when suggesting fixes, from the
beginning.
> > The first, ASCII-only, would be a mistake.
> No it wouldnt. There are no "unicode semantics" for POSIX.
That would be relevant if Perl had POSIX character classes.
However, the question of whether [[:xxx:]] is POSIX-like syntax, or an
actual POSIX character class, remains unanswered or at least unclear.
Certainly Perl's documentation isn't fully definitive. perlre mentions:
1 => "POSIX character class syntax"
2 => "POSIX character classes"
3 => Equivalences to \p{} Unicode constructs
1 and 3 can both be true, but then 2 is not. This is how I (prefer to)
think of it.
However, it could also be that 1 and 2 are true, ruling 3 out. If I
understand correctly, that's how you see the matter.
The mere existence of exceptions to the POSIX standard in [:xxx:], and
the exclusion of [.xxx.] and [=xxx=] lead me to believe that it's just
syntax compatibility, and Perl is free to extend the class definitions
to meet more modern requirements, like acknowledging that � is indeed
alphanumeric. Even if the people who invented the original POSIX bracket
expressions failed to notice.
> Try matching all the legal codepoints against [^POSIX] and against [POSIX]
> And note all the cases where you have both matching. Then do it with
> the strings in unicode. Note all the errors.
I wish you had spent the same time trying to explain what happens if you
do try this. Would have saved me some time and failure, because I was
unable to reproduce the errors.
perl -CO -le'(chr($_) =~ /[[:alnum:]]/) and (chr($_) =~ /[^[:alnum:]]/)
and warn sprintf "U+%04x (%s)\n", $_, chr for 1..65000'
doesn't give me anything. It is likely, however, that I misinterpreted
your instructions. (Note: I have no idea which codepoints qualify as
legal for this purpose, so I used the arbitrary limit of 65000.)
> For me this debate is over, POSIX charclasses are not Unicode
> charclasses and any contortion to try to make them so is futile and
> doomed to screw stuff over.
The futile, doomed to screw stuf over attempt has been ongoing for
almost a decade. You suggest going back, I suggest going forward.
Unfortunately I don't understand the points you're making, except the
one about POSIX simply not having any notion of unicode. I'm okay with
a change that makes Perl's [:x:] charclasses fully POSIX compliant, but
then it needs to be done rigourously, and all Perl exceptions have to be
eradicated. This should then not be seen as a fix of any unicode bug,
but as a design/semantics change.
Demerphq
> karl williamson skribis 2009-12-10 22:29 (-0700):
>> It was our intention that 5.12 would use strict Posix definitions
>> rigourously for all these,
>
> perldelta documents the change as such.
>
>> except the perl made-up extension, [[:Word:]], which has no Posix
>> definition.
>
> incompatible change. Why keep [:word:]? Not that I really mind, but
> strict interpretation usually doesn't come with exceptions.
if we restrict [:word:] to the ascii definition of \w we historically
have had then we stay true to the spirit of the POSIX definitions
while getting our underbar too and providing a work around for \w's
strange behaviour.
Juerd Waalboer
> rigourously for all these,
In that case I'm entirely fine with the change, provided of course that
perldelta documents the change as such.
> except the perl made-up extension, [[:Word:]], which has no Posix
> definition.
Changing the bracket expressions to strict POSIX semantics is an
incompatible change. Why keep [:word:]? Not that I really mind, but
strict interpretation usually doesn't come with exceptions.
Karl Williamson
> 2009/12/11 Juerd Waalboer <ju...@convolution.nl>:
>> karl williamson skribis 2009-12-10 22:29 (-0700):
>>> It was our intention that 5.12 would use strict Posix definitions
>>> rigourously for all these,
>> In that case I'm entirely fine with the change, provided of course that
>> perldelta documents the change as such.
>>
>>> except the perl made-up extension, [[:Word:]], which has no Posix
>>> definition.
>> Changing the bracket expressions to strict POSIX semantics is an
>> incompatible change. Why keep [:word:]? Not that I really mind, but
>> strict interpretation usually doesn't come with exceptions.
Belaboring the point: But it is strict interpretation. All Posix
defined constructs are rigorously Posix. This is not a Posix construct;
it looks like one, but it isn't one.
>
> if we restrict [:word:] to the ascii definition of \w we historically
> have had then we stay true to the spirit of the POSIX definitions
> while getting our underbar too and providing a work around for \w's
> strange behaviour.
Exactly
>
> Yves
>
karl williamson
> What I am missing in the dicussion is that on average exists code
> would be improved by chaning the semantics, and thus instead of
> thinking about possibly breaking 20% of CPAN we are fixing 80% of
> CPAN.
>
> If we want this to be the default at any time in the future, we should
> do it now, because I don't see how having another release cycle would
> change anything.
I'm thinking that if we make it not the default now, that it would give
people a chance to switch to it if they want; and a chance for module
authors to check their code. If they don't, well, they did have a
chance, as opposed to us springing it on them with no time for reaction.
>
> More specific about the failures caused by the changes:
>
> The pod stuff is breaking because it expect a non-breakable-space to
> be matched by \s, as far as I know it is about the only module
> expecting this behaviour (which is probably broken because it
> currently depends on the utf8-ness of the scalar). I did a similar
> change in Perl Kurila and what I remember is that only the pod module
> had problems with it. I'll check whether I can find the changes to the
> pod module, which make them work without using the "use legacy
> 'unicode8bit'".
How much of CPAN did you actually try on Kurila?
I actually did find the lines that needed changing in all the modules
except Test::Harness. They were in the wrap functions, and in some
cases, another one as well. I was starting to fix them there, but
realized I didn't know enough about what their input character set
domain was supposed to be.
>
> I am suprised at the failure of Test::Harness, if anything I would
> expect it to fix it, looking at ...\YAMList\Reader.pm it uses \s to
> match space characters, but according to YAML a non-breaking-space
> isn't a space (and thus it would be part of 80% of CPAN which
> would be fixed by the change).
>
> Karl: could you find out why it fails? I suspect that there is
> something having some (unwanted) side effect (which probably isn't
> wrong or shouldn't have any effect on code, but might be easily
> prevented).
I actually don't feel I have the time to spend on this. The test that
failed talked about Unprintables, and the failure was with the no break
space.
Karl Williamson
I had some more insight about this. I believe it is a bug in the test.
I changed the order so that the no break space wasn't first on the
line, and it passed. There is probably a s/^\s+// line in the module,
and it is reasonable for that to strip off a leading no-break space.
But the test assumes that it shouldn't.