[perl #26986] $& value does not properly propagate into replacement body of s///

[perlbug-...@perl.org]

# New Ticket Created by perl-...@ton.iguana.be
# Please include the string: [perl #26986]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org:80/rt3/Ticket/Display.html?id=26986 >

This is a bug report for perl from perl-...@ton.iguana.be,
generated with the help of perlbug 1.34 running under perl v5.8.2.

-----------------------------------------------------------------
[Please enter your report here]

perl -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print; print"a=@a"'
Argument "b" isn't numeric in range (or flop) at -e line 1.
Use of uninitialized value in range (or flop) at -e line 1.
Use of uninitialized value in range (or flop) at -e line 1.
<B>
a=0

I expected $& to be "a", so @a should become (a b c ....z), so fetch
element 26 from @F, which should be "Z"

The 'Argument "b" isn't numeric in range (or flop) at -e line 1.' seems to
indicate $& is undef which in my perl still causes @a to become (0)
(this was bug 24735, but even with that fixed I just expect that now
@a will be () and it will output <A>, which is still wrong).

However, mentioning $& in the real string body will make things work
as expected:
perl -wle '@F="A".."BZ"; $_="a"; s/./$&<$F[@a="b"..$&]>/; print; print"a=@a"'
a<Z>
a=b c d e f g h i j k l m n o p q r s t u v w x y z

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=low
---
Site configuration information for perl v5.8.2:

Configured by ton at Sun Jan 4 19:19:06 CET 2004.

Summary of my perl5 (revision 5.0 version 8 subversion 2) configuration:
Platform:
osname=linux, osvers=2.6.0, archname=i686-linux-64int-ld
uname='linux quasar 2.6.0 #3 thu dec 18 18:22:48 cet 2003 i686 gnulinux '
config_args=''
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=undef uselongdouble=define
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -fomit-frame-pointer',
cppflags='-fno-strict-aliasing -I/usr/local/include'
ccversion='', gccversion='3.4.0 20031231 (experimental)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='long double', nvsize=12, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=/lib/libc-2.3.2.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.3.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:

---
@INC for perl v5.8.2:
/usr/lib/perl5/5.8.2/i686-linux-64int-ld
/usr/lib/perl5/5.8.2
/usr/lib/perl5/site_perl/5.8.2/i686-linux-64int-ld
/usr/lib/perl5/site_perl/5.8.2
/usr/lib/perl5/site_perl
.

---
Environment for perl v5.8.2:
HOME=/home/ton
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/home/ton/bin.Linux:/home/ton/bin:/home/ton/bin.SampleSetup:/usr/local/bin:/usr/local/sbin:/usr/local/jre/bin:/home/oracle/product/9.0.1/bin:/usr/local/ar/bin:/usr/games/bin:/usr/X11R6/bin:/usr/share/bin:/usr/bin:/usr/sbin:/bin:/sbin:.
PERL_BADLANG (unset)
SHELL=/bin/bash

[Casey West]

It was Monday, February 23, 2004 when perl-...@ton.iguana.be (via RT) took the soap box, saying:
: perl -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print; print"a=@a"'

: Argument "b" isn't numeric in range (or flop) at -e line 1.
: Use of uninitialized value in range (or flop) at -e line 1.
: Use of uninitialized value in range (or flop) at -e line 1.
: <B>
: a=0

If I add a /e modifier on the end of your s///, and fix up the
replacement section, I get the following:

perl -wle '@F="A".."BZ"; $_="a"; s/./join $F[@a="b"..$&], "<", ">"/e; print; print"a=@a"'

<Z>
a=b c d e f g h i j k l m n o p q r s t u v w x y z

: I expected $& to be "a", so @a should become (a b c ....z), so fetch

: element 26 from @F, which should be "Z"

I think that I got what you expected. When you want to evaluate code
in a regular expression you must add the /e modifier. I think that
this is notabug. Am I missing something?

Casey West

--
Good Idea: Taking a deep breath before jumping into a swimming pool.
Bad Idea: Taking a deep breath after jumping into a swimming pool.

[Chip Salzenberg]

According to Casey West:

> It was Monday, February 23, 2004 when perl-...@ton.iguana.be (via RT) took the soap box, saying:
> : perl -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print; print"a=@a"'
> : Argument "b" isn't numeric in range (or flop) at -e line 1.
> : Use of uninitialized value in range (or flop) at -e line 1.
> : Use of uninitialized value in range (or flop) at -e line 1.
> : <B>
> : a=0
>

> If I add a /e modifier on the end of your s/// ...

Oh, that's interesting. I think it's a real bug.

The subscript of an array element is always evaluated as code, even in
a double-quoted string:

$ perl -le '@F=qw(a b c d e); print "$F[1+2]"'
d

I see no reason why C<1+2> should work but C<@a = "b" .. $&> should not.
--
Chip Salzenberg - a.k.a. - <ch...@pobox.com>
"I wanted to play hopscotch with the impenetrable mystery of existence,
but he stepped in a wormhole and had to go in early." // MST3K

[Ronald J Kimball]

On Tue, Feb 24, 2004 at 11:56:51AM -0500, Casey West wrote:
> It was Monday, February 23, 2004 when perl-...@ton.iguana.be (via RT) took the soap box, saying:
> : perl -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print; print"a=@a"'
> : Argument "b" isn't numeric in range (or flop) at -e line 1.
> : Use of uninitialized value in range (or flop) at -e line 1.
> : Use of uninitialized value in range (or flop) at -e line 1.
> : <B>
> : a=0
>
> If I add a /e modifier on the end of your s///, and fix up the
> replacement section, I get the following:
>
> perl -wle '@F="A".."BZ"; $_="a"; s/./join $F[@a="b"..$&], "<", ">"/e; print; print"a=@a"'
> <Z>
> a=b c d e f g h i j k l m n o p q r s t u v w x y z
>
> : I expected $& to be "a", so @a should become (a b c ....z), so fetch
> : element 26 from @F, which should be "Z"
>
> I think that I got what you expected. When you want to evaluate code
> in a regular expression you must add the /e modifier. I think that
> this is notabug. Am I missing something?

Perl evaluates code inside array and hash indexes inside double-quotish
strings. MJD takes advantage of this with his Interpolation module
(e.g. print "3 + 4 = $E{3+4}";). The warnings from Ton's example indicate
that the flip flop operator is being executed; the bug is that $& isn't
being set.

Ronald

[Father Chrysostomos via RT]

On Sun Apr 29 09:45:23 2012, Hugmeir wrote:

> This is still present in blead, albeit behaving a tad differently:
> $ ./perl -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print;
> print"a=@a"'
> Use of uninitialized value $& in range (or flop) at -e line 1.
> <A>
> a=
>
> Strangely enough, you can get $& to work in the flipflop if it's
> evaluated beforehand somehow:
> $ ./perl -Ilib -wle '@F="A".."BZ"; $_="a"; s/./<$F[!!$& &&
> (@a="b"..$&)]>/; print; print"a=@a"'

> <Z>
> a=b c d e f g h i j k l m n o p q r s t u v w x y z
>

> Putting an sv_dump() after the SvGETMAGIC(right) in pp_flop gives us this:
> $ ./perl -Ilib -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print;
> print"a=@a"'
> [...]
> SV = PVMG(0x854ab74) at 0x855c02c
> REFCNT = 1
> FLAGS = (GMG,SMG)
> IV = 0
> NV = 0
> PV = 0
> MAGIC = 0x855d3f4
> MG_VIRTUAL = &PL_vtbl_sv
> MG_TYPE = PERL_MAGIC_sv(\0)
> MG_OBJ = 0x855c01c
> MG_LEN = 1
> MG_PTR = 0x855d314 "&"
> Use of uninitialized value $& in range (or flop) at -e line 1.
> <A>
> a=
>
>
> I don't know the first thing about magic or the regex engine, so I'm
> stopping here. Could someone take a peek?

I seem to remember looking into this before. IIRC, it has to do with
s/// guessing whether it needs to set up match information for the rhs
based on its op tree. Or something like that.

--

Father Chrysostomos


---
via perlbug: queue: perl5 status: open
https://rt.perl.org:443/rt3/Ticket/Display.html?id=26986

[Brian Fraser via RT]

On Tue Feb 24 14:12:49 2004, rjk-pe...@tamias.net wrote:

This is still present in blead, albeit behaving a tad differently:

$ ./perl -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print;
print"a=@a"'

Use of uninitialized value $& in range (or flop) at -e line 1.
<A>
a=

Strangely enough, you can get $& to work in the flipflop if it's
evaluated beforehand somehow:
$ ./perl -Ilib -wle '@F="A".."BZ"; $_="a"; s/./<$F[!!$& &&

(@a="b"..$&)]>/; print; print"a=@a"'

<Z>
a=b c d e f g h i j k l m n o p q r s t u v w x y z

Putting an sv_dump() after the SvGETMAGIC(right) in pp_flop gives us this:

$ ./perl -Ilib -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print;
print"a=@a"'

[...]
SV = PVMG(0x854ab74) at 0x855c02c
REFCNT = 1
FLAGS = (GMG,SMG)
IV = 0
NV = 0
PV = 0
MAGIC = 0x855d3f4
MG_VIRTUAL = &PL_vtbl_sv
MG_TYPE = PERL_MAGIC_sv(\0)
MG_OBJ = 0x855c01c
MG_LEN = 1
MG_PTR = 0x855d314 "&"
Use of uninitialized value $& in range (or flop) at -e line 1.
<A>
a=


I don't know the first thing about magic or the regex engine, so I'm
stopping here. Could someone take a peek?

[Father Chrysostomos via RT]

This is basically the same as #49190. op.c:pm_runtime, in its if (repl)
block, guesses whether the rhs of a substitution can change from one
iteration to the next. If it can change, it gets a substcont op.
Otherwise, pp_subst hangs on to the same string. The logic in
pm_runtime is deserving of a whole list of expletives.

[Father Chrysostomos via RT]

On Mon Oct 08 22:09:22 2012, sprout wrote:
> On Sun Apr 29 11:00:17 2012, sprout wrote:
> > On Sun Apr 29 09:45:23 2012, Hugmeir wrote:
> > > On Tue Feb 24 14:12:49 2004, rjk-pe...@tamias.net wrote:
> > > > On Tue, Feb 24, 2004 at 11:56:51AM -0500, Casey West wrote:
> > > > > It was Monday, February 23, 2004 when perl-...@ton.iguana.be
(via
> > > > RT) took the soap box, saying:
> > > > > : perl -wle '@F="A".."BZ"; $_="a"; s/./<$F[@a="b"..$&]>/; print;
> > > > print"a=@a"'
> > > > > : Argument "b" isn't numeric in range (or flop) at -e line 1.
> > > > > : Use of uninitialized value in range (or flop) at -e line 1.
> > > > > : Use of uninitialized value in range (or flop) at -e line 1.
> > > > > : <B>
> > > > > : a=0
> > > > >
> > > > > If I add a /e modifier on the end of your s///, and fix up the
> > > > > replacement section, I get the following:
> > > > >
> > > > > perl -wle '@F="A".."BZ"; $_="a"; s/./join $F[@a="b"..$&], "<",
> > > > ">"/e; print; print"a=@a"'
> > > > > <Z>
> > > > > a=b c d e f g h i j k l m n o p q r s t u v w x y z

> This is basically the same as #49190. op.c:pm_runtime, in its if (repl)
> block, guesses whether the rhs of a substitution can change from one
> iteration to the next. If it can change, it gets a substcont op.
> Otherwise, pp_subst hangs on to the same string. The logic in
> pm_runtime is deserving of a whole list of expletives.
>

In particular, it is following op_next pointers to look for ‘dangerous’
ops (i.e., ops with side effects). So any logop can fool it, as its ops
are not in a simple op_next loop. Range ops are logops.

[Father Chrysostomos via RT]

I have fixed this particular case in commit 86e69b18632c, by foregoing
the optimisation when the replacement contains logops. I am marking
this as resolved, as bug #49190 lists the other problems.

[Reini Urban via RT]

This was not a resolving fix.

You simply disable s/// optimizations when a LOGOP is present.

You correctly analyzed it, but you should follow the LOGOP
alternative path instead if disabling the optimization at all.

On Tue Oct 09 17:52:00 2012, sprout wrote:
> On Mon Oct 08 22:37:14 2012, sprout wrote:
> > On Mon Oct 08 22:09:22 2012, sprout wrote:
> > > On Sun Apr 29 11:00:17 2012, sprout wrote:
> > > > On Sun Apr 29 09:45:23 2012, Hugmeir wrote:
> > > > > On Tue Feb 24 14:12:49 2004, rjk-pe...@tamias.net wrote:
> > > > > > On Tue, Feb 24, 2004 at 11:56:51AM -0500, Casey West wrote:
> > > > > > > It was Monday, February 23, 2004 when perl-

5....@ton.iguana.be

Reini Urban

---
via perlbug: queue: perl5 status: resolved
https://rt.perl.org:443/rt3/Ticket/Display.html?id=26986