[perl #116735] lexical subs don't seem to honor prototypes

[Peter Martini]

# New Ticket Created by Peter Martini
# Please include the string: [perl #116735]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=116735 >


This is a bug report for perl from peterc...@gmail.com,
generated with the help of perlbug 1.39 running under perl 5.17.9.


-----------------------------------------------------------------
[Please describe your issue here]

lexical subs don't seem to honor prototypes:

perl5.17.9 -Mfeature=lexical_subs -e 'my sub a($$){} a 1;'

Get rid of the 'my', and that dies.

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=low
---
Site configuration information for perl 5.17.9:

Configured by pmartini at Wed Feb 6 02:51:49 EST 2013.

Summary of my perl5 (revision 5 version 17 subversion 9) configuration:
Derived from: 52a2a812ca95071d6e5d921cf74061d912278729
Platform:
osname=linux, osvers=3.2.0-32-generic, archname=i686-linux-thread-multi
uname='linux pmlinlaptop 3.2.0-32-generic #51-ubuntu smp wed sep
26 21:32:50 utc 2012 i686 i686 i386 gnulinux '
config_args='-Dusedevel -DDEBUGGING -Dusethreads -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.6.3', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/i386-linux-gnu /lib/../lib
/usr/lib/i386-linux-gnu /usr/lib/../lib /lib /usr/lib
libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.15'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib
-fstack-protector'

Locally applied patches:


---
@INC for perl 5.17.9:
/usr/local/lib/perl5/site_perl/5.17.9/i686-linux-thread-multi
/usr/local/lib/perl5/site_perl/5.17.9
/usr/local/lib/perl5/5.17.9/i686-linux-thread-multi
/usr/local/lib/perl5/5.17.9
/usr/local/lib/perl5/site_perl
.

---
Environment for perl 5.17.9:
HOME=/home/pmartini
LANG=en_US.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
PERL_BADLANG (unset)
SHELL=/bin/bash

[Peter Martini via RT]

So, this is because prototype handling operates on an rv pointing to a
cv, and padcv is left out (since there's no rv pointing to it).

Fixing that gets to this little bit of ugliness, as explained in op.c:

if (!namegv) { /* expletive! */
/* XXX The call checker API is public. And it guarantees that
a GV will be provided with the right name. So we have
to create a GV. But it is still not correct, as its
stringification will include the package. What we
really need is a new call checker API that accepts a
GV or string (or GV or CV). */

I'm not sure what the best way forward is here.

[Peter Martini via RT]

The tests for lexical subs (t/cmd/lexsub.t) includes this:

package main;
{
sub me ($);
is prototype eval{\&me}, '$', 'my sub with proto';
is prototype "me", undef, 'prototype "..." ignores my subs';
}

That last test seems like a bug rather than a feature, and is part of
the reason prototypes don't work with lexical variables - prototypes are
only found if they're looked up via rv2cv, rather than via the new padcv
op.

If we can agree that that test should also return '$', then I can submit
a patch (for review) that gets prototypes working for lexical subs.

It's a rather involved patch...

[Rafael Garcia-Suarez]

On 17 February 2013 06:36, Peter Martini via RT

In any event, I think that either 5.18 should have lexical-sub
prototypes working, or warning (or dying?) just to make sure no-one
starts writing code that assumes the current behaviour is correct...

[Peter Martini]

We're in good shape on that point:

perl5.17.9 -E 'my sub foo {say @_} foo 1;'
Experimental "my" subs not enabled at -e line 1.

perl5.17.9 -Mfeature=lexical_subs -E 'my sub foo {say @_} foo 1;'
The lexical_subs feature is experimental at -e line 1.
1

It warns no matter what, but doesn't even execute unless you
specifically enable it.

[Peter Martini via RT]

Further testing:

perl5.17.9 -Mfeature=lexical_subs -e 'my sub a($$){} a 1;'

Does not check prototypes.

perl5.17.9 -Mfeature=lexical_subs -e 'my sub a($$){} a(1);'

*Does* check prototypes.

The difference is the second one generates an rv2cv pointing to the
padcv, while the first one uses the padcv directly.

I'd been playing with ways to fix this, and checked in my work-in-
progress (as a single large-ish commit) this morning:
https://github.com/PeterMartini/perl/commit/lexical-proto

There are two components, which are separable:

1. Change ck_subr to grab the CV directly from padcv if that's the last
op, which is the simple fix, and I'll defer to anyone more knowledgeable
on whether it's the correct fix.

2. Father C had noted that the Perl_call_checker API passes a GV*, which
no longer works, since a lexical sub won't have a GV. The solution as
of right now is a faked up GV, which as noted is not ideal, as it
includes the current package.

I added an alternate API, Perl_call_checker_sv and appropriate get/set
functions, so that the name could be passed be specified as an SV*. For
backwards compatibility reasons, if a custom Perl_call_checker was set,
it will use that; if a custom Perl_call_checker_sv was set, it will use
that; otherwise, it will use the default Perl_call_checker_sv.

These both get stored in checkcall magic, so only one can be active at a
time. If the current, GV form, is used in a set, and the SV form is
used in a get, the get returns a NULL pointer. In the reverse case,
where the new API is used to set an override, and the old API is used to
get it back, the code will croak. The reason for the difference is the
old API implies the call will always succeed, so returning a NULL would
be a bad idea, while the new version is documented to return NULL to
indicate that the alternate API should be used.

************

Now that I see that prototypes are actually partially honored, applying
just the first part would get us to consistency with minimal risk of
side effects, and the second part can be polished up as a nice to have.

[Peter Martini via RT]

Father C -

This change:

diff --git a/op.c b/op.c
index c9a1b53..9c2d06a 100644
--- a/op.c
+++ b/op.c
@@ -8135,7 +8135,6 @@ Perl_newCVREF(pTHX_ I32 flags, OP *o)
dVAR;
o->op_type = OP_PADCV;
o->op_ppaddr = PL_ppaddr[OP_PADCV];
- return o;
}
return newUNOP(OP_RV2CV, flags, scalar(o));
}

Seems like it should be sufficient for the fix, but it's causing
assertion failures on assert(hek) in the case of a const sub:

'my sub if(){44} if;' # boom!

I haven't been able to chase down a fix for that part yet, though.

[Reverend Chip]

On 2/16/2013 9:36 PM, Peter Martini via RT wrote:
> package main;
> {
> sub me ($);
> is prototype eval{\&me}, '$', 'my sub with proto';
> is prototype "me", undef, 'prototype "..." ignores my subs';
> }
>
> That last test seems like a bug rather than a feature, and is part of
> the reason prototypes don't work with lexical variables - prototypes are
> only found if they're looked up via rv2cv, rather than via the new padcv
> op.

This test seems fine. After all, compare with scalars. While $me
might be a lexical, ${"me"} never is. Why should & break this pattern?

Which isn't to say that lexical subs should ignore prototypes.

[Peter Martini]

Ah, that makes sense, and proves the point that my original approach
was too hackish for its own good.

Removing the early return in newCVREF certainly looks like a much
saner (and smaller) fix, but I haven't had a chance to look into why
the conversion to a const sub causes so much trouble.

[Father Chrysostomos via RT]

On Mon Feb 18 10:55:41 2013, pcm wrote:
> On Tue Feb 12 20:53:03 2013, pcm wrote:
> Further testing:
>
> perl5.17.9 -Mfeature=lexical_subs -e 'my sub a($$){} a 1;'
>
> Does not check prototypes.
>
> perl5.17.9 -Mfeature=lexical_subs -e 'my sub a($$){} a(1);'
>
> *Does* check prototypes.
>
> The difference is the second one generates an rv2cv pointing to the
> padcv, while the first one uses the padcv directly.
>
> I'd been playing with ways to fix this, and checked in my work-in-
> progress (as a single large-ish commit) this morning:
> https://github.com/PeterMartini/perl/commit/lexical-proto
>
> There are two components, which are separable:
>
> 1. Change ck_subr to grab the CV directly from padcv if that's the last
> op, which is the simple fix, and I'll defer to anyone more knowledgeable
> on whether it's the correct fix.

I’ve done what I think is the more correct fix, which is to avoid
generating two disparate op trees to begin with, in commit 9a5e6f3cd84.

> 2. Father C had noted that the Perl_call_checker API passes a GV*, which
> no longer works, since a lexical sub won't have a GV. The solution as
> of right now is a faked up GV, which as noted is not ideal, as it
> includes the current package.
>
> I added an alternate API, Perl_call_checker_sv and appropriate get/set
> functions, so that the name could be passed be specified as an SV*. For
> backwards compatibility reasons, if a custom Perl_call_checker was set,
> it will use that; if a custom Perl_call_checker_sv was set, it will use
> that; otherwise, it will use the default Perl_call_checker_sv.
>
> These both get stored in checkcall magic, so only one can be active at a
> time. If the current, GV form, is used in a set, and the SV form is
> used in a get, the get returns a NULL pointer. In the reverse case,
> where the new API is used to set an override, and the old API is used to
> get it back, the code will croak. The reason for the difference is the
> old API implies the call will always succeed, so returning a NULL would
> be a bad idea, while the new version is documented to return NULL to
> indicate that the alternate API should be used.
>
> ************
>
> Now that I see that prototypes are actually partially honored, applying
> just the first part would get us to consistency with minimal risk of
> side effects, and the second part can be polished up as a nice to have.

Would you be willing to do that? :-)

One thing I thought about was to create a new API function, maybe called
cv_name, which can be passed, a CV, a GV, or any stringifiable SV. It
would return an SV containing the name of the sub (a new mortal for a CV
or GV; the SV itself otherwise).

Whatever value is passed through the new call checker API could be
passed through to cv_name when an error is reported.

--

Father Chrysostomos


---
via perlbug: queue: perl5 status: open
https://rt.perl.org:443/rt3/Ticket/Display.html?id=116735

[Father Chrysostomos via RT]

That part I’ve fixed in commit 83a72a15a3.

[Father Chrysostomos via RT]

On Sun Jun 02 13:36:14 2013, sprout wrote:
> On Mon Feb 18 10:55:41 2013, pcm wrote:
> > I'd been playing with ways to fix this, and checked in my work-in-
> > progress (as a single large-ish commit) this morning:
> > https://github.com/PeterMartini/perl/commit/lexical-proto

That URL does not work any more.

I was about to do that (separate out part 2 from your patch and polish it up), but, as noted above, cannot access that URL.

Do you still have the patch floating around somewhere? If not, I will just have to write it from scratch. I need it right now for the stuff I’m doing on the sprout/cvgv branch. (ck_subr reifies GVs and I need the alternate call checker API to remove the need for that.)

--

Father Chrysostomos


---
via perlbug: queue: perl5 status: open

https://rt.perl.org/Ticket/Display.html?id=116735

[Father Chrysostomos via RT]

On Wed Sep 10 20:39:40 2014, sprout wrote:
> I was about to do that (separate out part 2 from your patch and polish
> it up), but, as noted above, cannot access that URL.
>
> Do you still have the patch floating around somewhere? If not, I will
> just have to write it from scratch. I need it right now for the stuff
> I’m doing on the sprout/cvgv branch. (ck_subr reifies GVs and I need
> the alternate call checker API to remove the need for that.)

Actually, having a separate API that takes an SV* would require us to duplicate all the built-in call checkers.

Instead, how about a cv_set_call_checker_flags, and the only flag is CALL_CHECKER_REQUIRE_GV? cv_set_call_checker calls _flags with that flag.

The name thingy that gets passed to the call checker can be cast to GV *.

--

Father Chrysostomos


---
via perlbug: queue: perl5 status: open

https://rt.perl.org/Ticket/Display.html?id=116735

[Ricardo Signes]

* Father Chrysostomos via RT <perlbug-...@perl.org> [2014-09-10T23:39:41]

> On Sun Jun 02 13:36:14 2013, sprout wrote:
> > On Mon Feb 18 10:55:41 2013, pcm wrote:
> > > I'd been playing with ways to fix this, and checked in my work-in-
> > > progress (as a single large-ish commit) this morning:
> > > https://github.com/PeterMartini/perl/commit/lexical-proto
>
> That URL does not work any more.

(This is why, even though it can be a pain in the butt, we ask for patches to
be sent in.)

--
rjbs

[Peter Martini]

On Wed, Sep 10, 2014 at 11:39 PM, Father Chrysostomos via RT
<perlbug-...@perl.org> wrote:
> On Sun Jun 02 13:36:14 2013, sprout wrote:
>> On Mon Feb 18 10:55:41 2013, pcm wrote:
>> > I'd been playing with ways to fix this, and checked in my work-in-
>> > progress (as a single large-ish commit) this morning:
>> > https://github.com/PeterMartini/perl/commit/lexical-proto
>
> That URL does not work any more.
>

I'm not quite sure at what point that got borked, since my local git
repo still listed it as only a remote branch on github (even though
github didn't find it). So for my own sanity, I did a local checkout
and pushed it back up to github unmodified. All that said, it's
probably not useful at all :-)

[Father Chrysostomos via RT]

On Thu Sep 11 22:50:24 2014, pcm wrote:
> I'm not quite sure at what point that got borked, since my local git
> repo still listed it as only a remote branch on github (even though
> github didn't find it). So for my own sanity, I did a local checkout
> and pushed it back up to github unmodified. All that said, it's
> probably not useful at all :-)

Thank you anyway. Or, rather, thank you for not restoring it till now, because I might not have been prompted by laziness to come up with a simpler solution otherwise. :-)

I do think it is over-engineered and that cv_set_call_checker_flags is a better solution.

BTW, have you seen what I am doing on the sprout/cvgv branch? It is almost ready for merging, but not quite. Maybe a few more days. In any case, ‘sub foo{} foo() \&foo’ no longer has to create a *foo glob, saving memory.

[Father Chrysostomos via RT]

On Fri Sep 12 00:26:55 2014, sprout wrote:
> I do think it is over-engineered and that cv_set_call_checker_flags is
> a better solution.
>
> BTW, have you seen what I am doing on the sprout/cvgv branch? It is
> almost ready for merging, but not quite. Maybe a few more days. In
> any case, ‘sub foo{} foo() \&foo’ no longer has to create a *foo glob,
> saving memory.

It is now in blead as merge commit f9d9e965c. The new call checker API is in commit aa38f4b16, and cv_name was added in c5569a55, fb094047 and b5e03f43ef.

[Father Chrysostomos via RT]

On Thu Sep 11 22:50:24 2014, pcm wrote:

> I'm not quite sure at what point that got borked, since my local git
> repo still listed it as only a remote branch on github (even though
> github didn't find it). So for my own sanity, I did a local checkout
> and pushed it back up to github unmodified. All that said, it's
> probably not useful at all :-)
>

I’m attaching it here for future readers.

[bulk88 via RT]

On Mon Sep 15 08:31:16 2014, sprout wrote:
> It is now in blead as merge commit f9d9e965c. The new call checker
> API is in commit aa38f4b16, and cv_name was added in c5569a55,
> fb094047 and b5e03f43ef.

Bug needs to be reopened.

Visual C 2003 is complaining of new warnings because of these commits.

cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -G7 -GL -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\op.obj ..\op.c
op.c
op.c(7897) : warning C4146: unary minus operator applied to unsigned type, result still unsigned
op.c(7958) : warning C4146: unary minus operator applied to unsigned type, result still unsigned
op.c(10751) : warning C4098: 'Perl_cv_get_call_checker' : 'void' function returning a value
op.c(10825) : warning C4244: '=' : conversion from 'U32' to 'U8', possible loss of data

line 7897 7958 comes from Sept 15 2014 commit http://perl5.git.perl.org/perl.git/commit/2eaf799e74b14dc77b90d5484a3fd4ceac12b46a

line 10751 comes from Sept 15 2014 commit
http://perl5.git.perl.org/perl.git/commit/9c98a81fd30898ed03895d1368f4f9f2761b69da

line 10825 comes from Sept 15 2014 commit
http://perl5.git.perl.org/perl.git/commit/aa38f4b16ec84f790a5473b0ff1ffe264bd93f5a


--
bulk88 ~ bulk88 at hotmail.com

---
via perlbug: queue: perl5 status: resolved
https://rt.perl.org/Ticket/Display.html?id=116735

[Father Chrysostomos via RT]

On Mon Sep 15 17:43:35 2014, bulk88 wrote:
> On Mon Sep 15 08:31:16 2014, sprout wrote:
> > It is now in blead as merge commit f9d9e965c. The new call checker
> > API is in commit aa38f4b16, and cv_name was added in c5569a55,
> > fb094047 and b5e03f43ef.
>
> Bug needs to be reopened.
>
> Visual C 2003 is complaining of new warnings because of these commits.
>
> cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32
> -D_CONSOLE -DNO_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG
> -G7 -GL -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\op.obj
> ..\op.c
> op.c
> op.c(7897) : warning C4146: unary minus operator applied to unsigned
> type, result still unsigned
> op.c(7958) : warning C4146: unary minus operator applied to unsigned
> type, result still unsigned
> op.c(10751) : warning C4098: 'Perl_cv_get_call_checker' : 'void'
> function returning a value
> op.c(10825) : warning C4244: '=' : conversion from 'U32' to 'U8',
> possible loss of data

Thank you.

>
> line 7897 7958 comes from Sept 15 2014 commit
> http://perl5.git.perl.org/perl.git/commit/2eaf799e74b14dc77b90d5484a3fd4ceac12b46a

Oops.

> line 10751 comes from Sept 15 2014 commit
> http://perl5.git.perl.org/perl.git/commit/9c98a81fd30898ed03895d1368f4f9f2761b69da

Oops.

> line 10825 comes from Sept 15 2014 commit
> http://perl5.git.perl.org/perl.git/commit/aa38f4b16ec84f790a5473b0ff1ffe264bd93f5a

Dumb compiler, but whatever.

I have fixed, or at least hope I have fixed, these in 53d063424.

--

Father Chrysostomos


---
via perlbug: queue: perl5 status: resolved
https://rt.perl.org/Ticket/Display.html?id=116735