[perl #60508] Assertion rx->sublen >= (s - rx->subbeg) + i failed: file "regcomp.c"
Clinton Gormley
# Please include the string: [perl #60508]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=60508 >
This is a bug report for perl from cl...@traveljury.com,
generated with the help of perlbug 1.36 running under perl 5.10.0.
-----------------------------------------------------------------
I've just had this error message return:
Assertion rx->sublen >= (s - rx->subbeg) + i failed: file "regcomp.c",
line 5109 at (eval 147)
I can't tell you what regex or value was being checked at the time,
but it might have been this call:
return HTML::Entities::encode_entities( $_[0], q(<>&"') );
All of my text is decoded utf8.
I don't know what other information I can give you to help
debug this?
thanks
Clint
[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=medium
---
This perlbug was built using Perl 5.10.0 - Tue Jul 15 14:37:49 UTC 2008
It is being executed now by Perl 5.10.0 - Tue Jul 15 14:31:57 UTC 2008.
Site configuration information for perl 5.10.0:
Configured by abuild at Tue Jul 15 14:31:57 UTC 2008.
Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
Platform:
osname=linux, osvers=2.6.25, archname=x86_64-linux-thread-multi
uname='linux stravinsky 2.6.25 #1 smp 20080210 20:01:04 utc x86_64
x86_64 x86_64 gnulinux '
config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr
-Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm
-Duseshrplib=true -Doptimize=-O2 -fmessage-length=0 -Wall
-D_FORTIFY_SOURCE=2 -fstack-protector -g -Wall -pipe
-Accflags=-DPERL_USE_SAFE_PUTENV'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV
-DDEBUGGING -fno-strict-aliasing -pipe -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
optimize='-O2 -fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2
-fstack-protector -g -Wall -pipe',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV
-DDEBUGGING -fno-strict-aliasing -pipe'
ccversion='', gccversion='4.3.1 20080507 (prerelease)
[gcc-4_3-branch revision 135036]', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib64'
libpth=/lib64 /usr/lib64 /usr/local/lib64
libs=-lm -ldl -lcrypt -lpthread
perllibs=-lm -ldl -lcrypt -lpthread
libc=/lib64/libc-2.8.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.8'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/usr/lib/perl5/5.10.0/x86_64-linux-thread-multi/CORE'
cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64'
Locally applied patches:
---
@INC for perl 5.10.0:
/usr/lib/perl5/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/5.10.0
/usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.10.0
/usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.10.0
/usr/lib/perl5/vendor_perl
.
---
Environment for perl 5.10.0:
HOME=/root
LANG=en_GB.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/opt/apache/sites/iAnnounce/bin:/usr/local/bin:/usr/sbin:/bin:/usr/bin:/sbin
PERL_BADLANG (unset)
SHELL=/bin/bash
Clinton Gormley
> If this bug is reproducible, we'd need a small program to replicate
> the bug, if possible.
Yeah, sorry - I realise it's a rubbish bug report. Unfortunately, I've
only seen this once, and I know neither the data nor the regex involved.
There are only two places I could see this coming from, neither of them
directly involving regexes:
The first is Template Toolkit (v2.20) using the XS stash:
From the compiled template:
$output .= $stash->get(['form', 0, 'field', [ 'keywords' ], 'render', 0]);
This is pretty standard stuff, and shouldn't have anything to do with user data.
And the second (which is the only regexy thing called by the above), is:
#===================================
sub _encode {
#===================================
return HTML::Entities::encode_entities( $_[0], q(<>&"') );
}
where $_[0] would be the UTF8-decoded value passed in by the user (the
value of which I don't know).
The error was thrown on a polish site, so it is likely that non Latin-1
characters were present.
hope this helps (a little)
clint
George Greer via RT
- - - 8< - - - 8< - - -
sub f { $_[0] =~ s/([&<>])/xxxxxx/g; }
$k1 = "<a> xxxxx x x <>";
f($k1);
$k2 = "pu\x{f1}al \x{2022}";
$k2 =~ s/([\200-\377])/"&#".ord($1).";"/ge;
f($k2);
- - - 8< - - - 8< - - -
Notes:
1. The function call is important, but it doesn't matter if it is built
from a string eval'd sub, an anonymous sub, or a regular sub.
2. The \200-\377 replacement is also important, although I did not test
with the \x{f1} changed to outside that range.
3. The seemingly innocuous $k1 work is required, as well as the contents
of $k1 itself are important.
4. The length of the replacement in f() matters, as well as the /g.
For Perl 5.10.0 from Fedora (perl-5.10.0-56.fc10.i386):
Assertion rx->sublen >= (s - rx->subbeg) + i failed: file "regcomp.c",
line 5098 at crash.pl line 5.
For bleadperl in git as of this writing
(6fea91d48c4e490e2fa422b26e5ff3883c86f34c):
perl5.11.0: regcomp.c:5176: Perl_reg_numbered_buff_fetch: Assertion
`rx->sublen >= (s - rx->subbeg) + i' failed.
ava...@gmail.com
According to git bisect (bisect scripts attached):
c74340f9cdee6010339b6bfd0e8b0dc8bc875344 is first bad commit
commit c74340f9cdee6010339b6bfd0e8b0dc8bc875344
Author: Yves Orton <deme...@gmail.com>
Date: Wed Nov 15 13:29:39 2006 +0100
Re: [PATCH] Fix RT#19049 and add relative backreferences
Message-ID: <9b18b3110611150329l206...@mail.gmail.com>
p4raw-id: //depot/perl@29279
:040000 040000 83188149c434a6cc022cceedd0d4f022869d37e9
c927650cf235b482e33028a4b0d60eed91e8c11b M ext
:040000 040000 c7f57d9a189a95bc7a6b7fa8b1b68e24f704d8af
702f39622af0427c6426ba58e6762e36f00288bc M pod
:100644 100644 9099194b070773063fc815d9c9b9de68422c1b87
6d916f1ff1fa552ccfa7cba5536fbcbada51ef0c M regcomp.c
:100644 100644 a0637a822f4ff0d41a81dbf743f9eab61f7c2efc
d547ff71a29fb1441d81faa7daab29d38805cf06 M regexec.c
:100644 100644 8d0868229068588c11c9d131d1a4c5462aedce62
d59fa832b26b48ab7b13cc309fd65a2d63ea3d27 M regexp.h
:040000 040000 ad6b74de72e11914a7f96973fba04a4f11781326
38db3cc9445020366949d615834b07dac267787d M t
bisect run success
ava...@gmail.com
Hrm, actually that's a different failure, from git bisect output:
Assertion rx->sublen >= (s - rx->subbeg) + i failed: file "regcomp.c",
line 4775 at /home/avar/src/bisect.pl line 1.
Bisecting: 455 revisions left to test after this
[5fe1b2e43d7b85573608a94dd98d2a8b7f06d58c] Swap SVpbm_TAIL and SVpbm_VALID.
[...]
Assertion rx->sublen >= s1 failed: file "mg.c", line 866 at
/home/avar/src/bisect.pl line 1.
Bisecting: 227 revisions left to test after this
[cdfeb707a2638190212953e4a52d8460de223429] was RE: Perl_die() /
Perl_croak() From: "Robin Barker" <Robin....@npl.co.uk> Message-ID:
<2C2E01334A940D4792B3\
E115F95B7...@exchsvr1.npl.ad.local>
I.e. the revision I found has an assertion error too, but somewhere
between cdfeb707a2638190212953e4a52d8460de223429 and
[5fe1b2e43d7b85573608a94dd98d2a8b7f06d58c we get a different assertion
error, which is the subject of this thread.
But I'm done bisecting for today :)
George Greer via RT
- - - 8< - - - 8< - - -
perl5.11.0: regcomp.c:5176: Perl_reg_numbered_buff_fetch: Assertion
`rx->sublen >= (s - rx->subbeg) + i' failed.
Program received signal SIGABRT, Aborted.
0x009ab416 in __kernel_vsyscall ()
(gdb) bt
#0 0x009ab416 in __kernel_vsyscall ()
#1 0x004c0460 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x004c1e28 in abort () at abort.c:88
#3 0x004b940e in __assert_fail (
assertion=0x83252c8 "rx->sublen >= (s - rx->subbeg) + i",
file=0x8321f9e "regcomp.c", line=5176,
function=0x83252a9 "Perl_reg_numbered_buff_fetch") at assert.c:78
#4 0x08105b1f in Perl_reg_numbered_buff_fetch (r=0x83b9da0, paren=1,
sv=0x83b9eb0) at regcomp.c:5176
#5 0x08141747 in Perl_magic_get (sv=0x83b9eb0, mg=0x83bf638) at mg.c:934
#6 0x0813e9c9 in Perl_mg_get (sv=0x83b9eb0) at mg.c:221
#7 0x0820fc97 in Perl_save_scalar (gv=0x83b9ea0) at scope.c:206
#8 0x0811dc98 in Perl_save_re_context () at regcomp.c:9886
#9 0x082b780e in Perl_swash_init (pkg=0x83917bd "utf8", name=0x838d7bc "",
listsv=0x83b9db0, minbits=1, none=0) at utf8.c:1687
#10 0x082b2467 in Perl_regclass_swash (prog=0x83bdf48, node=0x83ad53c,
doinit=1 '\001', listsvp=0x0, altsvp=0xbfffecf4) at regexec.c:5733
#11 0x082b271b in S_reginclass (prog=0x83bdf48, n=0x83ad53c,
p=0x83cd429 "▒\200▒", lenp=0x0, do_utf8=1 '\001') at regexec.c:5792
#12 0x0829a9dc in S_find_byclass (prog=0x83bdf48, c=0x83ad53c,
s=0x83cd429 "▒\200▒", strend=0x83cd42c "", reginfo=0xbffff0f0)
at regexec.c:1204
#13 0x082a22a8 in Perl_regexec_flags (rx=0x83b9da0,
stringarg=0x83cd423 "241al ▒\200▒", strend=0x83cd42c "",
strbeg=0x83cd420 "pu&241al ▒\200▒", minend=0, sv=0x83b9e40, data=0x0,
flags=25) at regexec.c:2089
#14 0x0817afe6 in Perl_pp_subst () at pp_hot.c:2330
#15 0x0812da53 in Perl_runops_debug () at dump.c:1981
#16 0x08086738 in S_run_body (oldscope=1) at perl.c:2313
#17 0x08085c7b in perl_run (my_perl=0x83a6008) at perl.c:2233
#18 0x0805e175 in main (argc=2, argv=0xbffff634, env=0xbffff640)
at perlmain.c:117
- - - 8< - - - 8< - - -
Some details at the time of the assertion:
- - - 8< - - - 8< - - -
#4 0x08105b1f in Perl_reg_numbered_buff_fetch (r=0x83b9da0, paren=1,
sv=0x83b9eb0) at regcomp.c:5176
5176 assert(rx->sublen >= (s - rx->subbeg) + i );
(gdb) info locals
rx = (struct regexp * const) 0x83bdf48
s = 0x83cd42e ":\b"
i = 1
s1 = 14
t1 = 15
__PRETTY_FUNCTION__ = "Perl_reg_numbered_buff_fetch"
(gdb) p *rx
$1 = {xnv_u = {xnv_nv = 8.6089369239461098e-311, xgv_stash = 0x0,
xpad_cop_seq = {xlow = 0, xhigh = 4057}, xbm_s = {xbm_previous = 0,
xbm_flags = 217 '▒', xbm_rare = 15 '\017'}}, xpv_cur = 16, xpv_len
= 20,
xiv_u = {xivu_iv = 0, xivu_uv = 0, xivu_p1 = 0x0, xivu_i32 = 0,
xivu_namehek = 0x0, xivu_hv = 0x0}, xmg_u = {xmg_magic = 0x0,
xmg_ourstash = 0x0}, xmg_stash = 0x0, engine = 0x8321f00, mother_re
= 0x0,
extflags = 34603008, minlen = 1, minlenret = 1, gofs = 0,
substrs = 0x83ad600, nparens = 1, intflags = 0, pprivate = 0x83ad520,
lastparen = 1, lastcloseparen = 1, swap = 0x83b4c40, offs = 0x83ad670,
subbeg = 0x83cd420 "pu&241al ▒\200▒", sublen = 12, pre_prefix = 8,
seen_evals = 0}
(gdb) p *rx->offs
$4 = {start = 14, end = 15}
- - - 8< - - - 8< - - -
Assert: rx->sublen >= (s - rx->subbeg) + i
rx->sublen = 12
s - rx->subbeg = 14
i = 1
12 >= 14 + 1
Interestingly:
- - - 8< - - - 8< - - -
(gdb) p *PL_reg_state.re_state_regoffs
$25 = {start = 2, end = 3}
- - - 8< - - - 8< - - -
which is what I would expect since "([&<>])" matches the & in
"pu&241..." there. Last bit of RE debug lines are:
- - - 8< - - - 8< - - -
Matching REx "([&<>])" against "pu&241al %x{2022}"
UTF-8 string...
Matching stclass ANYOF[&<>][] against "pu&241al %x{2022}" (12 chars)
2 <pu> <&241al > | 1:OPEN1(3)
2 <pu> <&241al > | 3:ANYOF[&<>][](14)
3 <pu&> <241al > | 14:CLOSE1(16)
3 <pu&> <241al > | 16:END(0)
Match successful!
Matching REx "([&<>])" against "241al %x{2022}"
UTF-8 string...
Matching stclass ANYOF[&<>][] against "241al %x{2022}" (9 chars)
perl5.11.0: regcomp.c:5176: Perl_reg_numbered_buff_fetch: Assertion
`rx->sublen >= (s - rx->subbeg) + i' failed.
- - - 8< - - - 8< - - -
Something not restoring ->offs (or clobbering the saved one) or I'm
confused or both, which is likely since I'm new to digging into Perl
internals. "s" at least points off into la-la land.
Michael G Schwern via RT
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-07/msg00161.html
I've written up a test for it. I couldn't fathom what file to put it in
or even enough about what its about to come up with a good name for it,
so its just named reg_60508.t. Feel free to change.
There's also a small patch to fresh_perl_is(). It already strips the
trailing newlines off of the results to normalize them (VMS is
unpredictable about that) but it doesn't off the expected results. This
is surprising for the test author who's unaware of the stripping, so
fresh_perl_is() now strips the expected results to match.
George Greer via RT
> Looks like this is hung waiting for review of George's last patch.
>
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-07/msg00161.html
Which reminded me to go make the 3rd revision of the patch which updates
the documentation and the two places that free/NULL that I missed in the
2nd revision because I hadn't (temporarily) removed the "swap" struct
definition to find lingering uses.
> I've written up a test for it. I couldn't fathom what file to put it in
> or even enough about what its about to come up with a good name for it,
> so its just named reg_60508.t. Feel free to change.
It is probably better to use fresh_perl_is() like yours, but take a look
at the test I have in my patch. Figuring out how to get it to fail
without -DEBUGGING would be nice, since it relies on the assert() in the
code.
-George
George Greer
> George Greer via RT wrote:
>> It is probably better to use fresh_perl_is() like yours, but take a look
>> at the test I have in my patch. Figuring out how to get it to fail
>> without -DEBUGGING would be nice, since it relies on the assert() in the
>> code.
>
> What, in English or Perl, does that assert failing indicate has gone wrong?
It means the paren-capture offsets point outside the match string. For
example, one of the assertion failures in the ticket:
(gdb) info locals
s = 0x83cd42e ":\b"
i = 1
s1 = 14
t1 = 15
gdb) p *rx
...
subbeg = 0x83cd420 "pu&241al ▒\200▒", sublen = 12
...
which results in:
Assert: rx->sublen >= (s - rx->subbeg) + i
rx->sublen = 12
s - rx->subbeg = 14
i = 1
12 >= 14 + 1
In the case above, the utf8 SWASHNEW Perl code is where the lingering
match came from and has nothing to do with the user match that the code
was trying to do.
Unfortunately the assert only notices if the capture is later in the
string than the current match target, so there are cases that
(theoretically) garbage could result.
--
George Greer
Demerphq
> On Sun Jul 12 02:59:44 2009, schwern wrote:
>> Looks like this is hung waiting for review of George's last patch.
>>
> http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-07/msg00161.html
>
> Which reminded me to go make the 3rd revision of the patch which updates
> the documentation and the two places that free/NULL that I missed in the
> 2nd revision because I hadn't (temporarily) removed the "swap" struct
> definition to find lingering uses.
>
>> I've written up a test for it. I couldn't fathom what file to put it in
>> or even enough about what its about to come up with a good name for it,
>> so its just named reg_60508.t. Feel free to change.
>
> at the test I have in my patch. Figuring out how to get it to fail
> without -DEBUGGING would be nice, since it relies on the assert() in the
> code.
Hi, sorry it took so long to get to this, but I have applied it as
7a68ade9729c0afc78c5f9cbadc5c77928cfedb8
with some minor commit message munging.
Thanks a lot! I think your solution makes more sense than mine did.
Cheers
yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Dave Mitchell
> Hi, sorry it took so long to get to this, but I have applied it as
>
> 7a68ade9729c0afc78c5f9cbadc5c77928cfedb8
With 03fbf60b09ed44cc7d2e021c979a259636ae4488, I've backed out the change
to ext/Devel-PPPort/parts/embed.fnc, since this is a dual-life distro
whose changes should be made upstream first (CCing marcus).
--
A power surge on the Bridge is rapidly and correctly diagnosed as a faulty
capacitor by the highly-trained and competent engineering staff.
-- Things That Never Happen in "Star Trek" #9
Dave Mitchell
> > On Sun Jul 12 02:59:44 2009, schwern wrote:
> >> Looks like this is hung waiting for review of George's last patch.
> >>
> > http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-07/msg00161.html
> >
> > Which reminded me to go make the 3rd revision of the patch which updates
> > the documentation and the two places that free/NULL that I missed in the
> > 2nd revision because I hadn't (temporarily) removed the "swap" struct
> > definition to find lingering uses.
> >
> >> I've written up a test for it. �I couldn't fathom what file to put it in
> >> or even enough about what its about to come up with a good name for it,
> >> so its just named reg_60508.t. �Feel free to change.
> >
> > It is probably better to use fresh_perl_is() like yours, but take a look
> > at the test I have in my patch. �Figuring out how to get it to fail
> > without -DEBUGGING would be nice, since it relies on the assert() in the
> > code.
>
> Hi, sorry it took so long to get to this, but I have applied it as
>
> 7a68ade9729c0afc78c5f9cbadc5c77928cfedb8
I'm having great difficulty pulling this into maint; specifically the
regexec.c part, due the changes in blead (but not maint) that made REGEXPs
into SVs; for example, the maint:
Perl_regexec_flags(pTHX_ REGEXP * const prog, ...
becomes the blead:
Perl_regexec_flags(pTHX_ REGEXP * const rx, ...
...
struct regexp *const prog = (struct regexp *)SvANY(rx);
then theres lots of places in the function which sometimes refer to rx and
sometimes to prog, and I think it would require someone with a clearer
head than what mine is at the moment to work it all out :-(
--
"Do not dabble in paradox, Edward, it puts you in danger of fortuitous wit."
-- Lady Croom, "Arcadia"
Demerphq
Ill give it a try tomorrow if George doesnt beat me to it.
cheers,
Yves
Dave Mitchell
> On Mon, 27 Jul 2009, Dave Mitchell wrote:
>
>> I'm having great difficulty pulling this into maint; specifically the
>> regexec.c part, due the changes in blead (but not maint) that made REGEXPs
>> into SVs; for example, the maint:
>>
>> Perl_regexec_flags(pTHX_ REGEXP * const prog, ...
>>
>> becomes the blead:
>>
>> Perl_regexec_flags(pTHX_ REGEXP * const rx, ...
>> ...
>> struct regexp *const prog = (struct regexp *)SvANY(rx);
>>
>> then theres lots of places in the function which sometimes refer to rx and
>> sometimes to prog, and I think it would require someone with a clearer
>> head than what mine is at the moment to work it all out :-(
>
> (perhaps by luck?). I've attached the patch, which has the commit
> message edited with a proper 1-line summary like the blead version did.
Thanks, but given the closeness to 5.10.1 release, and the risk of this
patch, I think I'll defer it till after 5.10.1.
--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.
Demerphq
Dave Mitchell
I haven't really caught up with my p5p mailbox since 5.10.1 was released,
and I haven't yet initiated a discussion about the nature of future 5.10.x
releases (a nd who will release it), so for the moment anything 5.10.x
related is kind of on hold.
--
Lady Nancy Astor: If you were my husband, I would flavour your coffee
with poison.
Churchill: Madam - if I were your husband, I would drink it.