Trying my hand at lexical subroutines

[Rob Hoelz]

Hello p5p,

I think lexical subroutines are a really cool feature, so I figured I'd
try to implement them. I realize this is probably a huge undertaking,
but I'd like to at least try so I can have some experience playing with
the Perl interpreter internals. Even if I don't complete this task,
I may leave some code behind that another can work off of.

I've read a lot of the basic documentation (perlhack, perlhacktips,
perlsource, perlinterp, perlapi, perlguts), but I figured I'd come here
and ask for some guidance about how to go about this task. I know I
probably have to do something with the pad API, and I see that the
parser already recognizes "my sub". I was thinking of doing something
like the following:

* When "my sub" is encountered, add a new entry to the current pad
with the given name, and the complied subroutine reference as the
value. A special flag will probably need to be set on the entry.

* When a subroutine call is parsed, look for an entry with the given name
in the current pad (or enclosing lexical pads). If an entry is
found and the new special flag is set, insert the opcodes to invoke the
subroutine reference contained within that pad entry. Otherwise, continue
with the old behavior.

The reason for the "special flag" is so that the following does not
work:

my $foo = sub { ... };

foo(1,2 , 3);

I should probably spend some time thinking about lexical subs with the
same name, but in nested scopes/the same scope/etc.

With any luck, lexical subroutines will be in perl 5.18. =)

-Rob

[Zefram]

Rob Hoelz wrote:
>I think lexical subroutines are a really cool feature, so I figured I'd
>try to implement them. I realize this is probably a huge undertaking,

Not too difficult to implement on CPAN. There's enough in the
core to support it. Basic plan: new keyword my_sub is defined via
Devel::CallParser. It performs custom parsing that turns "my_sub
foo {...}" into ops looking rather like "my $sub_foo = sub {...};",
but using a lexical variable name that is malformed from the point of
view of ordinary Perl syntax. For an example of the name trick see
Scope::Escape::Sugar. Checker on rv2cv turns &foo into &{$sub_foo}
(again using the magic variable name) where such a declaration is
in scope. Example of this sort of check magic in Lexical::Var.

my_sub is on my personal to-do list, but my plan involves writing
another module first. I'd like to separate out the check-time
name-to-arbitrary-op translation part of Lexical::Var from the
implementation of static variables. The intention is that other
modules such as my_sub can share that lower layer with Lexical::Var,
so that they'll play nicely sharing a single namespace. Currently, if
you implement the name lookup independently then the result of a name
clash between my_sub and Lexical::Var will depend on which module was
loaded first, rather than the desired lexical shadowing.

-zefram

[Aaron Sherman]

On Mon, Apr 16, 2012 at 10:20 AM, Rob Hoelz <r...@hoelz.ro> wrote:

 my $foo = sub { ... };

 foo(1,2 , 3);

Admittedly I'm in a rush and can't read your whole message right now, but isn't that spelled:

$foo->(1,2, 3);

?

 

--
Aaron Sherman <a...@ajs.com>

P: 617-440-4332 Google Talk: a...@ajs.com / aaronj...@gmail.com

"Toolsmith" and developer. Player of games. Buyer of gadgets.

[Rob Hoelz]

I'm describing a potential pitfall with my idea for implementing
lexical subs. If I just store lexical subs as subroutine refs in the
current pad, and I look for lexical subroutines when generating opcode
for 'foo(1, 2, 3)', I will pick the subroutine stored in $foo by
mistake.

[David Mertens]

No, Rob's original spelling was precisely what he meant.

[David Nicol]

Without reviewing the existing positions of the existing people with positions on this question, it seems to me that current named subroutines, which all go into the symbol table, are identified by (1) no sigil (2) context differentiating them from file handles, which haven't got a sigil either.

I don't know what happens if a file handle and a subroutine both have the same name, especially if the subroutine takes no args and returns a file handle. Could be ugly. But that's a digression.

Following the example of the sigiled data types, it seems that

  sub foo { say "outer" }
  {
    sub my foo { say "inner" }
    foo
  }
  foo
__END__

or

  sub foo { say "outer" }
  {
    my foo;
    sub foo { say "inner" }
    foo
  }
  foo
__END__

should output "inner" then output "outer"

while

  sub foo { say "outer" }
  {
    my foo;
    foo
  }
  foo
__END__

will abend at runtime.

Also, requiring the ampersand (oh yeah, subs DO have a sigil, which is optional because it is generally perfectly clear from context) somewhere would make sense, so the declarations might turn into

          my &foo;  # with sigil

or

          my sub foo;  # without sigil, or with "verbose" sigil

[Rob Hoelz]

I agree; if you want to "forward declare" a lexical subroutine, my
opinion is that it should look like this:

my &foo;

And nesting rules should work exactly as David described. You could
maybe even do:

my &foo;

sub foo { ... }

# code

sub foo { ... }

although I'm not sure how I feel about this.

I like these rules; they seems to consistent with Perl's treatment of
lexicals. Thoughts?

-Rob

[Brad Baxter]

On 4/16/12, David Nicol <david...@gmail.com> wrote:
> I don't know what happens if a file handle and a subroutine both have the
> same name, especially if the subroutine takes no args and returns a file
> handle. Could be ugly. But that's a digression.

$ vi qt

1 #!/usr/local/bin/perl -l
2
3 use strict;
4 use warnings;
5 use IO::File;
6
7 open FH, "qt" or die;
8 print scalar <FH>;
9 close FH;
10
11 print FH;
12
13 sub FH {
14 return IO::File->new();
15 }
16
17 print FH;
18
19 open FH, "qt" or die;
20 print scalar <FH>;
21 close FH;
22
23 print FH;
24
25 __END__

$ ./qt

#!/usr/local/bin/perl -l

print() on closed filehandle FH at ./qt line 11.
IO::File=GLOB(0x1e5898)
readline() on closed filehandle FH at ./qt line 20.

close() on unopened filehandle GEN2 at ./qt line 21.
IO::File=GLOB(0x161e70)

--
Brad

[Aaron Sherman]

On Tue, Apr 17, 2012 at 2:23 AM, Rob Hoelz <r...@hoelz.ro> wrote:

 my &foo;

 sub foo { ... }

I can't really get my head around the idea of why I would want to spell out my variable name twice. This smacks of Python's "global", and there's no reasons for that either.

How about "sub &" as an alias for "my & ... sub " e.g.:

sub &foo { ... }

which is currently not valid syntax.

If you really feel like spelling it out as you have, above, I have no problem with that, but I don't see a reason not to allow the shorter form.

[Jesse Luehrs]

On Tue, Apr 17, 2012 at 11:18:38AM -0400, Aaron Sherman wrote:
> On Tue, Apr 17, 2012 at 2:23 AM, Rob Hoelz <r...@hoelz.ro> wrote:
>
> >
> >
> > my &foo;
> >
> > sub foo { ... }
> >
> >
> I can't really get my head around the idea of why I would want to spell out
> my variable name twice. This smacks of Python's "global", and there's no
> reasons for that either.
>
> How about "sub &" as an alias for "my & ... sub " e.g.:
>
> sub &foo { ... }
>
> which is currently not valid syntax.
>
> If you really feel like spelling it out as you have, above, I have no
> problem with that, but I don't see a reason not to allow the shorter form.

I'm not really sure why we seem to be getting away from

my sub foo { }

with

my sub foo;

for predeclaration. 'my' is the lexical marker for everything else,
anyway.

-doy

[Rob Hoelz]

I think both 'my sub foo' and 'my &foo' should be accepted syntax.
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[Aaron Sherman]

Seems to make sense.

[Eric Brine]

On Mon, Apr 16, 2012 at 10:32 AM, Zefram <zef...@fysh.org> wrote:

Rob Hoelz wrote:
>I think lexical subroutines are a really cool feature, so I figured I'd
>try to implement them.  I realize this is probably a huge undertaking,

Not too difficult to implement on CPAN.  There's enough in the
core to support it.  Basic plan: new keyword my_sub is defined via
Devel::CallParser.  It performs custom parsing that turns "my_sub
foo {...}" into ops looking rather like "my $sub_foo = sub {...};"

Nit: One would expect the following to work so it would have to be a bit different.

sub recursive {
    my $global_state = ...;

    my_sub worker {
        ...
        worker(...);
        ...
    }

    worker(...);
}

[Eric Brine]

On Tue, Apr 17, 2012 at 5:00 PM, Ricardo Signes <perl...@rjbs.manxome.org> wrote:

* Rob Hoelz <r...@hoelz.ro> [2012-04-17T12:11:03]

> I think both 'my sub foo' and 'my &foo' should be accepted syntax.

"&foo" is generally used to invoke a subroutine in the olde timey fashion,
or to refer to an existing subroutine in those very limited places where the
language syntax has been special cased to notice it: defined and goto.

You forgot one special case (to bypass prototypes), but more importantly, you forgot two banal cases: sub references (\&foo) and sub calls by reference (&{$sub}(...)). "&" is far more common than mentioned, and that's not counting the *many* who always use "&" in their sub calls.

Given the choice between C<<my sub f;>> and C<<my &sub;>>, I would use the latter. It's shorter and (more importantly) stylistically consistent with existing code.

- Eric

[Jesse Luehrs]

On Tue, Apr 17, 2012 at 05:17:49PM -0400, Eric Brine wrote:
> On Tue, Apr 17, 2012 at 5:00 PM, Ricardo Signes
> <perl...@rjbs.manxome.org>wrote:
>
> > * Rob Hoelz <r...@hoelz.ro> [2012-04-17T12:11:03]
> > > I think both 'my sub foo' and 'my &foo' should be accepted syntax.
> >
> > "&foo" is generally used to invoke a subroutine in the olde timey fashion,
> > or to refer to an existing subroutine in those very limited places where
> > the
> > language syntax has been special cased to notice it: defined and goto.
> >
>
> You forgot one special case (to bypass prototypes), but more importantly,
> you forgot two banal cases: sub references (\&foo) and sub calls by
> reference (&{$sub}(...)). "&" is far more common than mentioned, and that's
> not counting the *many* who always use "&" in their sub calls.

Sure, but those two both deal with explicit subroutine references, which
lexical subs don't.

> Given the choice between C<<my sub f;>> and C<<my &sub;>>, I would use the
> latter. It's shorter and (more importantly) stylistically consistent with
> existing code.

How would you define a lexical sub then? Having C<<my &foo = sub { }>>
do something completely different from C<<my $foo = sub { }>> seems
potentially confusing, and having different syntax for declaring and
defining lexical subs also seems like a bad idea.

-doy

[Aristotle Pagaltzis]

Hi Rob,

* Rob Hoelz <r...@hoelz.ro> [2012-04-16 16:25]:

> I think lexical subroutines are a really cool feature, so I figured
> I'd try to implement them. I realize this is probably a huge
> undertaking, but I'd like to at least try so I can have some
> experience playing with the Perl interpreter internals. Even if
> I don't complete this task, I may leave some code behind that another
> can work off of.

you probably want to be aware of the previous implementation effort from
Florian Ragwitz:
http://www.nntp.perl.org/group/perl.perl5.porters/;msgid=1241375686-21452-1-...@debian.org

Also, you may want to speak with Mark Jason Dominus:
http://www.nntp.perl.org/group/perl.perl5.porters/;msgid=2007030919134...@plover.com

As well, the rest of both threads.

To my knowledge, MJD’s bookful of test cases is not codified anywhere;
if they are and someone knows of it, please pipe up. Or if you speak
with him and transcribe the things he says to test cases – that alone
would be valuable.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

[Aristotle Pagaltzis]

* Eric Brine <ike...@adaelis.com> [2012-04-17 23:20]:

> You forgot one special case (to bypass prototypes), but more
> importantly, you forgot two banal cases: sub references (\&foo) and
> sub calls by reference (&{$sub}(...)). "&" is far more common than
> mentioned, and that's not counting the *many* who always use "&" in
> their sub calls.

What do any of these have to do with sub predeclaration? That’s
a nonsequitur as far as I can tell.

> Given the choice between C<<my sub f;>> and C<<my &sub;>>, I would use
> the latter. It's shorter and (more importantly) stylistically
> consistent with existing code.

Maybe it would be prettier if we didn’t already have `sub f;` to declare
subs for the last 20 years. But gratuitous exploitation of opportunities
to diverge for the mere sake of beauty leads not to beauty but ugliness.

If you really want `my &f` then please spend some time thinking about
how to retrofit this mechanism to an equivalent for global subs
(`our &f`?) and their differences in semantics, and whether to get rid
of the old (`sub f;`) or provide *both* avenues in both cases. Getting
rid of the old is not feasible in this case so you are stuck with the
latter option. And then consider whether it is worth doing that.

Let’s keep the æsthetics of the language as an entity in mind, rather
than treating as a loose conglomeration – agglomeration – of special-
purpose features designed in a vacuum. If you need a demonstration of
that school of design just look at PHP.

[Darin McBride]

On Tuesday April 17 2012 6:24:44 PM Ricardo Signes wrote:
> * Jesse Luehrs <d...@tozt.net> [2012-04-17T17:25:32]

>
> > How would you define a lexical sub then? Having C<<my &foo = sub { }>>
> > do something completely different from C<<my $foo = sub { }>> seems
> > potentially confusing, and having different syntax for declaring and
> > defining lexical subs also seems like a bad idea.
>

> It would also make these two things completely different:
>
> my &foo = sub { ... }
> &foo = sub { ... }

I'm curious how one would go about assigning to such a lexical other than
during declaration?

my &foo;
# now make C<foo(...)> call bar(@_)?
&foo = sub { &bar }; # um?

I like the consistency of "my &foo" but this falls down, IMO, when you try to
assign it other than when you declare it, making them basically read-only.

(Disclaimer: I'm also missing out on the use-case scenario here, cf code refs
in a scalar combined with __SUB__, and my google-fu isn't showing old
conversations on *why* only *how*. But I'm not entering the debate on that,
merely asking about syntax.)

[Jesse Luehrs]

The other issue here is that the actual assignment in every other case
of "my $foo = ..." happens at runtime, but subroutine calls are resolved
at compile time. Using the "my &foo = sub { ... }" syntax would mean
either deferring all lexical sub lookups to runtime (with potentially
surprising side effects) or moving that assignment to compile time
(meaning that "my $foo = sub { ... }; my &foo = $foo" becomes
nonsensical, among other things).

What you really have to decide here is, what do you want to be
consistent with? "my &foo = ..." is consistent with other variable
assignments, and "my sub foo { ... }" is consistent with other
subroutine declarations. I think that lexical subs should really be more
like non-lexical subs than like lexical variables.

-doy

[Eric Brine]

On Tue, Apr 17, 2012 at 5:25 PM, Jesse Luehrs <d...@tozt.net> wrote:

> Given the choice between C<<my sub f;>> and C<<my &sub;>>, I would use the
> latter. It's shorter and (more importantly) stylistically consistent with
> existing code.

How would you define a lexical sub then?

First, I thought it was between

my sub foo = sub { };
and
my &foo = sub { };

but a more careful reading shows that it was between

my sub foo; sub foo { }
and
my &foo; sub foo { }

I retract my comment.

~~~~~~~~~~~~~~~~~~~~~

The following have been the proposed syntax.

1. my_sub foo {}
2. sub my foo {}
3. my foo; sub foo {}
4. my sub foo; sub foo {}
5. my &foo; sub foo {}      --- XXX Looks like you can do <<my &foo = ...>>.
6. sub &foo {}              --- XXX Doesn't say lexical to me. Maybe even the opposite
7. my sub foo {}
8. my &foo = sub {};        --- XXX Doesn't happen at compile time.          

Possible required features:

(A) Compile-time effect (e.g. in case the sub has a prototype).
(B) Support for pre-declaration (like C<<sub foo($); foo($x); sub foo($) {}>>).
(C) Support for re-declaration (like C<<my $x; my $x;>>).
(D) Support for using package sub over lexical sub (like C<<my $x = 123; our $x; print $x;>>).

Both (B) and (C) prevent the same syntax from being used by a separate declaration and definition. As such, I think the best candidates are

C<<my sub foo>> for declaring (with or without a definition)
and
C<<sub foo {}>> for defining a pre-declared sub.

# (A)
my sub foo { say "a" }
foo();  # a

# (B)
my sub foo;
foo();  # b
sub foo { say "b" }  # pre-declared lexical

# (C)
my sub foo { say "c1" }
foo();  # c1
my sub foo { say "c2" }
foo();  # c2

We don't actually need a new syntax for (D) (which doesn't mean it wouldn't be useful).

# (D)
sub foo { say "d1" }
my sub foo { say "d2" }
main::foo();  # d1

The only soft spot is the following looks weird:

my sub foo;
sub foo { say "1" }  # pre-declared lexical
sub foo { say "2" }
foo();  # 1

- Eric

[Jesse Luehrs]

This could be avoided by requiring 'my sub' in both the predeclaration
and the definition:

my sub foo;
my sub foo { say "1" }

sub foo { say "2" }
foo(); # 1

This makes it a bit clearer what's going on.

-doy

[Eric Brine]

On Tue, Apr 17, 2012 at 11:00 PM, Jesse Luehrs <d...@tozt.net> wrote:

On Tue, Apr 17, 2012 at 10:52:17PM -0400, Eric Brine wrote:
> Both (B) and (C) prevent the same syntax from being used by a separate
> declaration and definition. As such, I think the best candidates are

 

This could be avoided by requiring 'my sub' in both the predeclaration

and the definition:

I said that would break (B) and (C), but I guess it doesn't actually. (You already have to define the sub before any other lexical declarations.) I'm all for using C<<my sub foo>> to define pre-declared sub.

- Eric

[Zefram]

Jesse Luehrs wrote:
> Using the "my &foo = sub { ... }" syntax would mean
>either deferring all lexical sub lookups to runtime (with potentially
>surprising side effects) or moving that assignment to compile time

People are getting more confused here than is warranted. The sub lookup
is perfectly simple: the exact sub (CV) to use cannot be determined
until runtime, because in the general case the sub isn't *generated*
until runtime, because it's a closure over variables that don't exist
until a particular run of the code. "foo(123)" where foo() is a lexcial
subroutine necessarily compiles to code equivalent to "$foo_sub->(123)".
This has *nothing at all* to do with the syntax used to declare the sub.

As for the syntax, "my &foo =" is silly, because "&foo =" already means
something with very different semantics. (It involves calling foo().)
Aside from avoiding such inconsistency with existing syntax, the details
of the syntax don't really matter. It's the semantics that matter.

-zefram

[Rob Hoelz]

I'm going to take back my earlier wish for 'my &foo' as preclaring a
lexical sub; I think 'my sub foo' would suffice. On to my next
question...

Let's say I predeclare foo:

my sub foo;

What would be the syntax for defining it? I can think of two:

# later in the code
sub foo { ... } # this
my sub foo { ... } # or this

The first has the advantage of being more concise; the second has the
disadvantage of being a bit redundant, but the advantage of being much
more clear. Personally, I vote for the second.

[John Imrie]

On 19/04/2012 07:51, Rob Hoelz wrote:
> I'm going to take back my earlier wish for 'my&foo' as preclaring a

> lexical sub; I think 'my sub foo' would suffice. On to my next
> question...
>
> Let's say I predeclare foo:
>
> my sub foo;
>
> What would be the syntax for defining it? I can think of two:
>
> # later in the code
> sub foo { ... } # this
> my sub foo { ... } # or this
>

For lexical subs to work correctly would you have to declare and define
them in the same scope, or can you declare the sub in one scope and
define it in another.

If the latter what happens when you declare a lexical sub in two
different scopes and define it in a third?

John

[demerphq]

On 16 April 2012 16:20, Rob Hoelz <r...@hoelz.ro> wrote:
> With any luck, lexical subroutines will be in perl 5.18. =)

While I have no objection I find it weird that we are contemplating
lexical subroutines when we never managed to get something as useful
as $^THIS_SUB into core.

For me $^THIS_SUB eliminates any real need for named lexical
subroutines, and doesn't require any syntax changes or anything like
that, nor addressing any hairy issues like "what happens if you define
a lexically named subroutine in a package that has the subroutine
already" (does it act like local *sub=sub {} ?) and a bunch of related
hairy edge cases.

cheers,
Yves





--
perl -Mre=debug -e "/just|another|perl|hacker/"

[Jesse Luehrs]

On Thu, Apr 19, 2012 at 09:17:28AM +0200, demerphq wrote:
> On 16 April 2012 16:20, Rob Hoelz <r...@hoelz.ro> wrote:
> > With any luck, lexical subroutines will be in perl 5.18. =)
>
> While I have no objection I find it weird that we are contemplating
> lexical subroutines when we never managed to get something as useful
> as $^THIS_SUB into core.

__SUB__ is already in 5.15, if that's what you're referring to.

-doy

[demerphq]

Does it return a reference or a string?

[Jesse Luehrs]

$ perl -E'sub foo { say __SUB__ } foo'
CODE(0x1f31de0)

-doy

[demerphq]

Ah cool. So from my point of view we dont need lexical subroutines at
all then :-)

I am so glad I was wrong on this point.

Regarding lexical subroutines, consider the following code. If I
changed the C<local *foo> to C<my sub foo> what is the expected
result? If the expected result is the same then are we not actually
discussing "dynamically scoped subroutines"? If it is not the expected
result can someone explain the difference between a lexically scoped
subroutine and a dynamically scoped subroutine, and how this will
impact things like optimizations, method caching, etc?

$ cat t.pl
use strict;
use warnings;


sub foo {
print "package foo: $_[0]\n";
no warnings 'redefine';
local *foo= sub {
return if !$_[0] or $_[0]<0;
print "local foo:",$_[0],"\n";
bar($_[0]-1);
};
foo($_[0]);
}

sub bar {
return if !$_[0] or $_[0]<0;
print "package bar:",$_[0],"\n";
foo($_[0]-1);
}

foo(10);

$ perl t.pl
package foo: 10
local foo:10
package bar:9
local foo:8
package bar:7
local foo:6
package bar:5
local foo:4
package bar:3
local foo:2
package bar:1

[Rob Hoelz]

I'm obviously no expert on the Perl core, but I would expect that
lexical subroutines would be faster (at least by a little). With a
non-lexical subroutine, I need to do a symbol table lookup for the
given subroutine, apply any dynamic effects
(like local *foo = sub { ... }), etc. With a lexical subroutine,
I should be able to resolve the location of the subroutine at compile
time, cutting out a hash lookup.

There's also the benefit of keeping your module namespace clean, which I
feel is the true win. Let's say I have the following code:

package Greeter;

use strict;
use warnings;
use feature 'say';

sub _get_username {
local $| = 1;
print "Your name: ";
my $name = <STDIN>;
chomp $name;
return $name;
}

sub greet {
my $name = _get_username();
say "Hello, $name!";
}

1;

As it stands, _get_username is a helper function that should only be
used by the Greeter module. I may change its interface in the future,
get rid of it altogether, or even do nothing to it as time passes; the
point is that it's part of an internal API that I don't want others to
touch. Right now, barring things like namespace::clean, someone can
do just that. However, if I make _get_username a lexical sub, they
cannot.

-Rob

[demerphq]

I understand generally the motivation here. But what are the expected
semantics?

Lexical scoping implies something quite different from dynamic
scoping. Are we talking about dynamic scoping or true lexical scoping?

My point here is we have a lexical sub with a given name, it calls a
sub which is declared in a different lexical context, which calls a
subroutine with the same name we chose for our lexical. What is
supposed to happen? Do you think it would be confusing? How would it
interact with dyanmic subroutines? What happens if I declare a
recursive lexical sub inside of a sub of the same name, what happens?

A bunch of edge cases like that come up with this problem and before
we do any work on this id like to see them spelled out in detail. We
have had way too many half-baked ideas added to core that required
years of cleanup before we solidified the semantics properly. Id like
to avoid that in this case.

In other words we need a clear understanding of how lexically scoped
subs are supposed to interact with goto, recursion, tail-recursion,
package level subs, and dynamically scoped subs. Thats a lot of cases.

Yves

[Zefram]

demerphq wrote:
>Regarding lexical subroutines, consider the following code. If I
>changed the C<local *foo> to C<my sub foo> what is the expected
>result?

Simulating it now, I get:

$ perl t.pl
package foo: 10
local foo:10
package bar:9

package foo: 8

local foo:8
package bar:7

package foo: 6

local foo:6
package bar:5

package foo: 4

local foo:4
package bar:3

package foo: 2

local foo:2
package bar:1

package foo: 0

It differs in that the foo() call in sub bar always refers to the
top-level sub foo; bar doesn't see the lexically-defined foo.

> If it is not the expected
>result can someone explain the difference between a lexically scoped
>subroutine and a dynamically scoped subroutine,

Visibility. A dynamically-scoped anything (sub, scalar variable,
whatever) is really a global variable, whose value is temporarily changed
during the execution of some block. The temporary value is visible to
all parts of the program, because anyone can look at a global variable,
but the temporary value is only there for a limited time.

Note that in the Perl implementation the "global variable" I refer to is
a hidden layer, not what you usually think of as a variable; "local $foo"
doesn't change the value of $foo, it changes the value of *foo{SCALAR};
the thing stored in that variable slot is itself usually a variable.
For subs, *foo{CODE} is a global variable allowing for localisation of
&foo, but &foo itself is not variable in the usual sense.

A lexically-scoped anything does not have this intermediation. It makes a
specified name refer to a specified object for the purposes of code within
the lexical scope of the declaration. That is, it's for code that's
textually located within the block where it's declared. That code can
include code that will execute later, after the main execution of that
lexical block has finished, and the later-executed code will still see
the lexical declaration (this is closure). Other, textually-distant,
code called during the main execution of the lexical block does not see
the lexical declaration.

> and how this will
>impact things like optimizations, method caching, etc?

Methods are not affected at all. Method lookup happens in package
namespaces, i.e., in global variables, to which lexical declarations
are irrelevant. Lexical subs inherently cannot participate in method
resolution. (Slightly more strictly: it's the *names*, not the subroutine
objects themselves, that are lexical or packaged. A single sub object
may have many names.)

Optimisation is nearly unaffected. Ignoring prototypes for a moment, a
call to a lexical sub (i.e., a sub call via a lexical name) will involve a
pad lookup (a padsv op or similar) where a call to a package sub involves
a gv op. On threaded builds the gv op is actually a type of pad op, so
there's nearly no difference there. Logic within the entersub op takes
different cases depending on whether the sub is supplied as a GV, CV, or
RV: in all cases it's just a matter of following a couple of pointers,
using fixed offsets into structs. There's no dramatic change like
avoiding a hash lookup, because package-based sub lookups do the hash
lookup (in the stash, to get the GV) once at compile time, not at runtime.

Presuming that we have some support for prototypes on lexical subs
(good idea, not entirely trivial to implement), custom call checkers can
perform optimisations like inlining. This would be essentially the same
for lexical subs as it is for package subs.

-zefram

[Zefram]

demerphq wrote:
>Lexical scoping implies something quite different from dynamic
>scoping. Are we talking about dynamic scoping or true lexical scoping?

Since people have said "lexical subs" and thrown around the keyword
"my", I presume that they mean lexical. I note that we already have
dynamic scoping, including of subs, and dynamic scoping generally sucks,
so a new way to dynamically scope subs wouldn't be interesting.

>My point here is we have a lexical sub with a given name, it calls a
>sub which is declared in a different lexical context, which calls a
>subroutine with the same name we chose for our lexical. What is
>supposed to happen?

The sub in the remote lexical scope cannot see the lexical sub. It would
look for a sub of that name in its own lexical namespace, or failing
that it would look for the global (possibly dynamic) sub of that name
in its package. There is no other possibility.

> Do you think it would be confusing?

About as confusing as lexical scalar variables. Do you think this
is confusing:

sub foo {
my $num = 123;
print bar() + $num;
}
$num = 1;
sub bar {
return $num;
}

On the whole I think we've got over that one. *Dynamic* scope, *that's*
the confusing one.

> How would it
>interact with dyanmic subroutines?

Same way lexical and dynamic scalar variables interact. Lexical
declaration (or lack thereof) controls what a name refers to; when
referring to the {sub,scalar} in a glob then the second layer of that
lookup can be temporarily modified by "local". There is no other
possibility.

> What happens if I declare a
>recursive lexical sub inside of a sub of the same name, what happens?

Ooh, we've actually got an option here. The question arises (has already
been touched upon) of whether the scope of a lexical sub definition begins
before or after the body of that sub. Different languages have different
conventions on whether lexical scopes start before or after the lexical
declaration. In C it's before, so you can do "void *p = (void *) &p;" to
get a variable pointing at itself. In Lisp by default (with "let") it's
after, so that a lexical variable can be initialised using an outer-scope
variable that might have the same name, but some other operators make it
before, so that you can lexically declare mutually-recursive functions.
Perl's "my" currently makes it after, like Lisp's "let".

I reckon that plain "my sub foo {...}" ought to give the same kind of
scoping as "my $foo = ...;". That is, the scope of the declaration
starts after the declaration, so in your hypothetical case with "sub
foo { my sub foo { foo() } }" the inner "foo()" refers to the package
namespace and hence to the outer sub foo. But there's a non-trivial
convenience in some situations in having the scoping work the other way,
and it's greater for subs than for ordinary variables, so there probably
ought to be some variant formulation that works the other way.

>A bunch of edge cases like that come up with this problem and before
>we do any work on this id like to see them spelled out in detail.

Don't really need that if it's prototyped on CPAN. People will notice
edge cases more easily once a prototype implementation is available,
and the prototype can be extensively modified before we start committing
the core to any version of it. There can even be multiple independent
prototypes. This is why I initially spelled it "my_sub" rather than "my
sub". The intention is that the module provides my_sub, and eventually
the core implements whichever version of that we're happiest with under
the "my sub" keywords.

>In other words we need a clear understanding of how lexically scoped
>subs are supposed to interact with goto,

Same as "goto &$foo".

> recursion, tail-recursion,

Name scoping discussed above; apart from that, same as "$foo->()".
We have no specific tail recursion mechanism. Sub::Call::Tail amounts
to a "goto &" underneath.

>package level subs, and dynamically scoped subs.

Same way lexically scoped variables interact with package variables.

-zefram

[demerphq]

On 19 April 2012 15:18, Zefram <zef...@fysh.org> wrote:

> demerphq wrote:
>>                                   What happens if I declare a
>>recursive lexical sub inside of a sub of the same name, what happens?
>
> Ooh, we've actually got an option here.  The question arises (has already
> been touched upon) of whether the scope of a lexical sub definition begins
> before or after the body of that sub.  Different languages have different
> conventions on whether lexical scopes start before or after the lexical
> declaration.  In C it's before, so you can do "void *p = (void *) &p;" to
> get a variable pointing at itself.  In Lisp by default (with "let") it's
> after, so that a lexical variable can be initialised using an outer-scope
> variable that might have the same name, but some other operators make it
> before, so that you can lexically declare mutually-recursive functions.
> Perl's "my" currently makes it after, like Lisp's "let".

Thanks for the detailed replies! Commenting on this point alone, more
detailed response will come in a follow up.

The problem I have with the "follow the current rules for lexicals" is
that it seems like you could not write a recursive lexically scoped
subroutine without using __SUB__, which IMO seems suboptimal (but also
make me all the happier we finally got __SUB__ :-)

cheers,

[Zefram]

Father Chrysostomos wrote:
>If we can write
>
> my $x
> { $x = 3 }
>
>then we need a way to do the equivalent for lexical subs.

Not necessarily. We have no equivalent of assignment for global subs;
there's no inherent reason why we should have one for lexical subs.
(There's "*foo = sub{...}", but that's not assigning to a sub, that's
creating a new sub and putting a reference to it somewhere.)

Given that "my sub foo {...}" declares the lexical name and defines the
sub that it refers to, we have a problem about forward declarations.
"my sub foo;" looks like a forward declaration, but then if you try "my
sub foo {...}" to define the body (following the pattern for global subs),
that looks like a second lexical declaration that should shadow the first.

I'm dubious about allowing forward declarations at all for lexical subs; I
think we'd generally be better (avoiding a couple of conceptual problems)
without them. People who really want to mess around with lexical subs
in variable-like ways can use a, er, variable. Like they already can.
"my $foo; if($this) { $foo = sub {...} }".

>Is "my sub foo { ... }" going to be a run-time statement?

It would be silly for that not to create a closure. So there's inevitably
a runtime aspect.

>I do think "our sub foo" should make a package sub visible to other
>packages in the same lexical scope, just like "our $foo":

Yes, that's sensible.

-zefram

[Abigail]

On Thu, Apr 19, 2012 at 02:18:46PM +0100, Zefram wrote:
> demerphq wrote:
>
> > What happens if I declare a
> >recursive lexical sub inside of a sub of the same name, what happens?
>
> Ooh, we've actually got an option here. The question arises (has already
> been touched upon) of whether the scope of a lexical sub definition begins
> before or after the body of that sub. Different languages have different
> conventions on whether lexical scopes start before or after the lexical
> declaration. In C it's before, so you can do "void *p = (void *) &p;" to
> get a variable pointing at itself. In Lisp by default (with "let") it's
> after, so that a lexical variable can be initialised using an outer-scope
> variable that might have the same name, but some other operators make it
> before, so that you can lexically declare mutually-recursive functions.
> Perl's "my" currently makes it after, like Lisp's "let".

And it's not just the equivalence of my. Consider:

use strict;
sub foo {
...
foo;
...
}

This is currently a compile-time error, as 'foo' doesn't get resolved
until after the entire 'sub foo' has been parsed. It would be strange
if

use strict;
my sub foo {
...
foo;
...
}

would call itself -- and then fail to compile if someone removed the 'my'.


Besides, if we do the binding after, it makes it easier to write
temporary wrapper functions:

sub whatever { ... }

{
my sub whatever {
say "whatever called with: @_";
my $r = whatever @_;
say "whatever returned: $r";
$r;
}

... debug the calls to whatever ...

}



Abigail

[Eric Brine]

On Thu, Apr 19, 2012 at 8:39 AM, demerphq <deme...@gmail.com> wrote:

Lexical scoping implies something quite different from dynamic
scoping. Are we talking about dynamic scoping or true lexical scoping?

I haven't seen any indication that lexical was misused to mean dynamic scoping. This is about creating symbols only seen in the lexical scope where they are declared.

... bar is not visible here ...

sub foo {
    ... bar is not visible here ...

    my sub bar {
        ... I hope bar is visible here ...
    }

     ... bar is visible here ...
}

 ... bar is not visible here ...

What happens if I declare a recursive lexical sub inside of a sub of the same name, what happens?

Why would it be different than lexical vars? Yes, you can obscure an existing var/sub. But it's wholely preventable.

sub foo {
    ... foo refers to package sub ...
    my sub foo {
        ... foo refers to outer lexical sub ...
        my sub foo {
             ... foo refers to inner lexical sub ...
        };
        ... foo refers to inner lexical sub ...
    };
    ... foo refers to outer lexical sub ...
}
... foo refers to package sub ...

how this will impact things like [...] method caching

Methods are looked up by name (to support inheritance), so they must be package subs.

 -Eric

[Nicholas Clark]

On Thu, Apr 19, 2012 at 12:09:08PM -0400, Eric Brine wrote:
> On Thu, Apr 19, 2012 at 8:39 AM, demerphq <deme...@gmail.com> wrote:
>
> > Lexical scoping implies something quite different from dynamic
> > scoping. Are we talking about dynamic scoping or true lexical scoping?
> >
>
> I haven't seen any indication that lexical was misused to mean dynamic
> scoping. This is about creating symbols only seen in the lexical scope
> where they are declared.
>
> ... bar is not visible here ...
>
> sub foo {
> ... bar is not visible here ...
>
> my sub bar {
> ... I hope bar is visible here ...

To behave consistently with how lexical scalars (etc) work, bar should not
be visible here.

$ perl
use strict;
my $s = do { \$s };
__END__
Global symbol "$s" requires explicit package name at - line 2.
Execution of - aborted due to compilation errors.

> }
>
> ... bar is visible here ...
> }
>
> ... bar is not visible here ...

I do not mean to be arguing which behaviour is right. Merely to note that
lexical variables are not in scope until the end of the statement which
declares them.

Nicholas Clark

[demerphq]

That was my point, if we obey the full rules of lexical declarations
you cant write lexical subs that are recursive without using __SUB__
or otherwise jumping through hoops.

[Zefram]

Eric Brine wrote:
>sub foo {
> my sub helper;
> my sub helper {
> helper();
> }
>}

That's problematic, because "my sub helper { ..." looks like a lexical sub
declaration unto itself, which ought to shadow any previous declaration
of the same name. As I said elsewhere in the thread, if you want to
separate declaration from definition then you need a different syntax
for the non-declaring definition. But I don't think such a facility
would pull its weight.

-zefram

[Eric Brine]

On Thu, Apr 19, 2012 at 12:14 PM, Nicholas Clark <ni...@ccl4.org> wrote:

I do not mean to be arguing which behaviour is right. Merely to note that
lexical variables are not in scope until the end of the statement which
declares them.

That's why I said "I hope" to the inconsisten behaviour. Mind you, it's not necessary as long as this works:

[Eric Brine]

On Thu, Apr 19, 2012 at 12:37 PM, Zefram <zef...@fysh.org> wrote:

Eric Brine wrote:
>sub foo {
>   my sub helper;
>   my sub helper {
>      helper();
>   }
>}

That's problematic, because "my sub helper { ..." looks like a lexical sub
declaration unto itself

The syntax was irrelevant to my point

As I said elsewhere in the thread, if you want to
separate declaration from definition then you need a different syntax
for the non-declaring definition

I said as much too, thought I believe it's not necessary. I don't care either way.

But I don't think such a facility would pull its weight.

Meaning pre-declaration might not be supported? I don't think I've ever pre-declared a sub, so as long as I can do recursive lexical or anon subs, I'm happy :) (And apparently, I can with __SUB__ if I understand correctly. Off to look up what that is.)

[Rob Hoelz]

Thanks for the tips! I think I'll wait for the dust to settle on the
particulars of this feature before I dive in. =)

On Tue, 17 Apr 2012 23:28:14 +0200
Aristotle Pagaltzis <paga...@gmx.de> wrote:

> Hi Rob,
>
> * Rob Hoelz <r...@hoelz.ro> [2012-04-16 16:25]:
> > I think lexical subroutines are a really cool feature, so I figured
> > I'd try to implement them. I realize this is probably a huge
> > undertaking, but I'd like to at least try so I can have some
> > experience playing with the Perl interpreter internals. Even if
> > I don't complete this task, I may leave some code behind that
> > another can work off of.
>
> you probably want to be aware of the previous implementation effort
> from Florian Ragwitz:
> http://www.nntp.perl.org/group/perl.perl5.porters/;msgid=1241375686-21452-1-...@debian.org
>
> Also, you may want to speak with Mark Jason Dominus:
> http://www.nntp.perl.org/group/perl.perl5.porters/;msgid=2007030919134...@plover.com
>
> As well, the rest of both threads.
>
> To my knowledge, MJD’s bookful of test cases is not codified anywhere;
> if they are and someone knows of it, please pipe up. Or if you speak
> with him and transcribe the things he says to test cases – that alone
> would be valuable.
>
> Regards,

[Johan Vromans]

Father Chrysostomos <spr...@cpan.org> writes:

> I do think "our sub foo" should make a package sub visible to other
> packages in the same lexical scope, just like "our $foo":
>

> package main;
> our sub foo;
> package might;
> foo() # main::foo

That's just one tiny step away from a pragma that makes all subroutines
private (static, lexical, whatever) *unless* the sub is declared with
our.

package Foo;
use feature qw( JustAnotherVerbAbused );
sub foo { ... } # not visible outside the package
our sub bar { ... } # visible as Foo::bar

And that's just one tiny step away from a useful :method attribute.

package Foo;
use feature qw( JustAnotherVerbAbused );
sub foo { ... } # not visible outside the package
sub bar : method { ... } # not visible but reachable via obj ref

Just some ideas...

-- Johan

[Dave Mitchell]

On Thu, Apr 19, 2012 at 02:45:44PM +0100, Zefram wrote:
> >Is "my sub foo { ... }" going to be a run-time statement?
>
> It would be silly for that not to create a closure. So there's inevitably
> a runtime aspect.

So presumably, you're of the school who prefers

my sub foo { ... }
foo(...)

to be syntactical sugar for

my $foo = sub { ... }
$foo->(...)

as opposed to the other "obvious" interpretation of lexical subs, which is

that

my sub foo { ... }

foo(...)

is just like

sub foo { ... }
foo(...)

except that the compile-time look-up of the function name 'foo'
resolves to different things depending on which scope you are in?

--
Any [programming] language that doesn't occasionally surprise the
novice will pay for it by continually surprising the expert.
-- Larry Wall

[Zefram]

Dave Mitchell wrote:
>to be syntactical sugar for
>
> my $foo = sub { ... }
> $foo->(...)

Yes. I described it in such terms in my first message on this thread.

>as opposed to the other "obvious" interpretation of lexical subs, which is

Static sub definitions with lexical visibility. Has rather limited
utility. Already available on CPAN in Lexical::Sub.

-zefram

[Dave Mitchell]

On Thu, Apr 19, 2012 at 08:12:27AM -0400, Rob Hoelz wrote:
> I'm obviously no expert on the Perl core, but I would expect that
> lexical subroutines would be faster (at least by a little). With a
> non-lexical subroutine, I need to do a symbol table lookup for the
> given subroutine, apply any dynamic effects
> (like local *foo = sub { ... }), etc. With a lexical subroutine,
> I should be able to resolve the location of the subroutine at compile
> time, cutting out a hash lookup.

With existing package subs, a symbol table lookup is not performed;
instead the typeglob corresponding to the function name is located at
compile time and pointed to from the OP_GV op; at run-time, its just a
case of accessing the gp_cv slot within the gp struct pointed to by the
GV.
So any time savings will be negligible.

--
My get-up-and-go just got up and went.

[Aristotle Pagaltzis]

* Dave Mitchell <da...@iabyn.com> [2012-04-20 16:45]:

> On Thu, Apr 19, 2012 at 02:45:44PM +0100, Zefram wrote:
> > >Is "my sub foo { ... }" going to be a run-time statement?
> >
> > It would be silly for that not to create a closure. So there's inevitably
> > a runtime aspect.
>
> So presumably, you're of the school who prefers
>
> my sub foo { ... }
> foo(...)
>
> to be syntactical sugar for
>
> my $foo = sub { ... }
> $foo->(...)
>
> as opposed to the other "obvious" interpretation of lexical subs,
> which is that
>
> my sub foo { ... }
> foo(...)
>
> is just like
>
> sub foo { ... }
> foo(...)
>
> except that the compile-time look-up of the function name 'foo'
> resolves to different things depending on which scope you are in?

That second interpretation never even occurred to me, it seems of so
obviously little use. The first interpretation has many uses, all of
which are already possible obviously, but which are accessible only
through such clumsy and self-conscious expression that the option of
doing these things is unlikely to be taken by anyone in practice.

E.g. it is already possible to write lexical subs at the top scope of
a module that defines a class, to keep them from being invokable as
methods – and I have done that, but only on rare occasion. Saddling the
sub with an underscore or several in its name makes it ugly but the code
at the call sites then reads steady and sure of itself, which is more
important.

Another benefit of opting the design which can be described as “this has
the exact same semantics as this other more complicated syntax” is just
that – that it can be described that way. It means the language will be
more linguistically coherent – easier to learn, to remember, to teach.
Easier to speak.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

[Paul Johnson]

On Sat, Apr 21, 2012 at 02:13:23PM +0200, Aristotle Pagaltzis wrote:
> * Dave Mitchell <da...@iabyn.com> [2012-04-20 16:45]:
> > On Thu, Apr 19, 2012 at 02:45:44PM +0100, Zefram wrote:
> > > >Is "my sub foo { ... }" going to be a run-time statement?
> > >
> > > It would be silly for that not to create a closure. So there's inevitably
> > > a runtime aspect.
> >
> > So presumably, you're of the school who prefers
> >
> > my sub foo { ... }
> > foo(...)
> >
> > to be syntactical sugar for
> >
> > my $foo = sub { ... }
> > $foo->(...)
> >
> > as opposed to the other "obvious" interpretation of lexical subs,
> > which is that
> >
> > my sub foo { ... }
> > foo(...)
> >
> > is just like
> >
> > sub foo { ... }
> > foo(...)
> >
> > except that the compile-time look-up of the function name 'foo'
> > resolves to different things depending on which scope you are in?
>
> That second interpretation never even occurred to me, it seems of so
> obviously little use.

The utility is in the benefit to those reading and writing the code. By
limiting the scope of the subroutine the code becomes easier to reason
about to both programmers and, potentially, to tools.

The situation is directly analogous to the benefit provided by using
lexical variables in preference to global variables.

This doesn't argue against the first interpretation but simply suggests
that there is still a lot to be said for the second.

> Another benefit of opting the design which can be described as “this has
> the exact same semantics as this other more complicated syntax” is just
> that – that it can be described that way. It means the language will be
> more linguistically coherent – easier to learn, to remember, to teach.
> Easier to speak.

You're arguing for the first option here. The same basic argument could
probably be used for the second.

--
Paul Johnson - pa...@pjcj.net
http://www.pjcj.net

[Aristotle Pagaltzis]

* Paul Johnson <pa...@pjcj.net> [2012-04-21 15:05]:

You’re right. The second option can be implemented several more ways
than the first option, and some of those scenarios do not yield this
benefit – but no, this is not a option-1-vs-2 distinguisher. Mea culpa.

> >That second interpretation never even occurred to me, it seems of so
> >obviously little use.
>
> The utility is in the benefit to those reading and writing the code.
> By limiting the scope of the subroutine the code becomes easier to
> reason about to both programmers and, potentially, to tools.
>
> The situation is directly analogous to the benefit provided by using
> lexical variables in preference to global variables.

I do not think that analogy is useful, though I cannot argue it clearly
yet – but to give you an idea for where my gut feel comes from, consider
how many functions an arbitrary piece of code might define and how often
it mentions them, vs how many variables.

[Aristotle Pagaltzis]

* Dave Mitchell <da...@iabyn.com> [2012-04-20 16:45]:

> On Thu, Apr 19, 2012 at 02:45:44PM +0100, Zefram wrote:
> > >Is "my sub foo { ... }" going to be a run-time statement?
> >
> > It would be silly for that not to create a closure. So there's inevitably
> > a runtime aspect.
>
> So presumably, you're of the school who prefers
>
> my sub foo { ... }
> foo(...)
>
> to be syntactical sugar for
>
> my $foo = sub { ... }
> $foo->(...)
>
> as opposed to the other "obvious" interpretation of lexical subs, which is
> that
>
> my sub foo { ... }
> foo(...)
>
> is just like
>
> sub foo { ... }
> foo(...)
>
> except that the compile-time look-up of the function name 'foo'
> resolves to different things depending on which scope you are in?

I just realised something else: the second interpretation will cause no
end of consternation because of this:

Variable "$x" will not stay shared

This is the only place where Perl scoping is weird. Normal lexical scope
is easy to understand – but if you nest subs, things get bizarre. (In an
utterly useless way, to boot.)

Consider how to explain to a novice what that message means and how to
achieve what they meant to do when they got this warning from perl.

At least in the case of nested global subs that behaviour is essentially
inevitable: the fact that they get compiled only once and installed in
the package leaves no other choice in spite of its uselessness.

I would like less of that already, I certainly have no hankering for it
to become more prevalent instead. (I do wonder now if there is any good
reason that this warning is not simply an error. Is there *ever* any use
for the behaviour that Perl exhibits in that case?)