Pre-RFC: native lexical importing

[Ovid via perl5-porters]

Some people expressed interest in the :export functionality of the module proposal. Thus, another pre-RFC.

Using Exporter.pm or writing your own import() sub is very clumsy, but very common. This should be native to Perl.

The idea is to provide exporting via an `:export` attribute:

    package Foo {
        sub bar :export {...} # @EXPORT_OK
        sub baz :export(:strings) %EXPORT_TAGS
        sub quux :export(:ALWAYS) {...} # @EXPORT
        sub whee {...} # not exported
    }

These exports are set via attributes, they do *not* require the import() sub to be called, and they are lexical.

Open questions.

1. How would one would handle delayed importing? `Foo->import('bar');`

2. Allow renaming of imported functions?

3. "Lexical importing" means "callable until end of enclosing block unless shadowed", but does that also mean that it's not in the symbol table? (I would assume so)

Related prior art on the CPAN:

https://metacpan.org/pod/Perl6::Export::Attrs
https://metacpan.org/pod/Lexical::Import
https://metacpan.org/pod/Export::Lexical
https://metacpan.org/pod/Exporter::Tiny
https://metacpan.org/pod/namespace::autoclean

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

[Paul "LeoNerd" Evans]

On Sat, 12 Feb 2022 09:37:03 +0000 (UTC)
Ovid via perl5-porters <perl5-...@perl.org> wrote:

> Some people expressed interest in the :export functionality of the
> module proposal. Thus, another pre-RFC.

:)

> Using Exporter.pm or writing your own import() sub is very clumsy,
> but very common. This should be native to Perl.
>
> The idea is to provide exporting via an `:export` attribute:
>
>     package Foo {
>         sub bar :export {...} # @EXPORT_OK
>         sub baz :export(:strings) %EXPORT_TAGS
>         sub quux :export(:ALWAYS) {...} # @EXPORT
>         sub whee {...} # not exported
>     }
>
> These exports are set via attributes, they do *not* require the
> import() sub to be called, and they are lexical.

This looks a reasonable first-pass to write up as a longer doc.

> Open questions.
>
> 1. How would one would handle delayed importing? `Foo->import('bar');`

Lexical imports would have to be done at compiletime in order to
actually work. Thus you could BEGIN-wrap it:

BEGIN {
require Foo;
Foo->import('bar');
} # i.e. exactly what use Foo 'bar' does

bar();

but nothing equivalent when done at runtime.

> 2. Allow renaming of imported functions?

I can imagine this might occasionally be required, yes. Perhaps some
sort of symbol on the import line?

use Process qw( run=process_run );
use Athlete qw( run=athlete_run );

# Now we can call process_run() separate from athlete_run()

Or maybe the order of those is confusing, being opposite to the usual
VAR=VALUE assignment syntax. Perhaps

use Module qw( name-to-import-as=name-in-module );

> 3. "Lexical importing" means "callable until end of enclosing block
> unless shadowed", but does that also mean that it's not in the symbol
> table? (I would assume so)

Correct. This is one of their primary goals. By not being in the
symbol-table they don't pollute the namespace of callable methods on an
object, or are visible as package functions/methods to outsiders.


Some other questions that come to mind:

* Can you :export a variable?

our $LOGGER :export;

* For that matter, can you :export lexical things that aren't in the
originating package's symbol table:

my sub do_thing :export {}

my $variable :export;

* What happens if an :export-using package defines a `sub import` as
well? Three main ways it could do that:

sub import { ... }

use Exporter 'import';

use base 'Exporter';

> Related prior art on the CPAN:
>
> https://metacpan.org/pod/Perl6::Export::Attrs
> https://metacpan.org/pod/Lexical::Import
> https://metacpan.org/pod/Export::Lexical
> https://metacpan.org/pod/Exporter::Tiny

> https://metacpan.org/pod/namespace::autoclean

While not directly a prior art for this kind of feature, it is worth
mentioning namespace::autoclean in this area because of the way it's
often used to tidy up imported functions from a package's namespace
when implementing an object class. Lexical imports would remove this
need.

--
Paul "LeoNerd" Evans

leo...@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

[Martijn Lievaart]

On 2/12/22 10:37, Ovid via perl5-porters wrote:

> Some people expressed interest in the :export functionality of the module proposal. Thus, another pre-RFC.
>
> Using Exporter.pm or writing your own import() sub is very clumsy, but very common. This should be native to Perl.
>
> The idea is to provide exporting via an `:export` attribute:
>
>     package Foo {
>         sub bar :export {...} # @EXPORT_OK
>         sub baz :export(:strings) %EXPORT_TAGS
>         sub quux :export(:ALWAYS) {...} # @EXPORT
>         sub whee {...} # not exported
>     }
>
> These exports are set via attributes, they do *not* require the import() sub to be called, and they are lexical.
>
> Open questions.
>

This looks awesome, but how would this interact with Import::Into (and
others)? Or alternatively, how could that functionality be achieved with
this new scheme?


HTH,

M4

[Ovid via perl5-porters]

OK, what you're saying makes sense, though I still believe that `:export` should be core, not CPAN, even if it's a wrapper around your interface. I believe `:export` solves *most* of what people want, so core Perl should too.

> Is it permissible to unexport a lexically exported subroutine?  (I suggest it should not be.)

Can you elaborate? I thought I understood this, but the more I think about it, the more I realize I don't understand. 

> Should the lexical exporter be able to export other variable types lexically 
> — hashes, arrays, and scalars.  I think I'd just want to leave room for doing 
> it in the future, but it seems likely to be pretty confusing, to me.

I'm OK with that. In general, the ability to export mutable variables seems like a popular anti-pattern. I have some client code which does this all the time and it's common to find that *something* has changed the underlying variable, but it's very hard to track down what that something is.

Side note: I also think Perl needs a core way of declaring even complex data structures as immutable, though I haven't given deep thought about this. Something like `immutable %hash = (...);`, but that would be a pre-RFC for another day. Suggestions welcome.

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

On Saturday, 12 February 2022, 14:34:38 CET, Ricardo Signes <perl...@rjbs.manxome.org> wrote:

On Sat, Feb 12, 2022, at 4:37 AM, Ovid via perl5-porters wrote:
> Some people expressed interest in the :export functionality of the module proposal. Thus, another pre-RFC.
>
> Using Exporter.pm or writing your own import() sub is very clumsy, but very common. This should be native to Perl.
>
> The idea is to provide exporting via an `:export` attribute:

I think you're skipping a vital step.  I don't care whether we provide an :export attribute.  It seems okay.  But what we should provide is a simple way to export a lexical subroutine.  I think we need to provide a means for lexical subroutine exports.

In Perl, the process for import/export is generally:
    * calling code in package C says "package X, please import specified things 1,2,3 into me"
    * package X's import method then installs the symbols into package C
This process allows for package X to decide how to interpret 1,2,3 and how to install them in package C.  This is why we can have different exporters, some of which differ in the sugar provided over the basic features of Exporter.pm, and some of which provide substantially more features.  I think this "more than one way" situation is very good!

If the process is mostly abstracted so that the attribute is the only way to negotiate selection, and named subroutines are the only exportable, we lose an enormous amount of flexibility in exchange for what I think is an essential feature.

I believe what should happen instead is that we should provide a core library with an interface like:
lexport->export(
  name1 => coderef1,
  name2 => coderef2,
  ...
);
This would install the given code into currently-compiling scope with the given names.

The attribute-based mechanism would still be easy to write, and at its core, it would do the above.  Other exporters (like Sub::Exporter) could have a means to replace their usual *{"C::1"} = \&{"P::1"} code with that method call.


Open questions:

    * Is it permissible to unexport a lexically exported subroutine?  (I suggest it should not be.)
    * Should the lexical exporter be able to export other variable types lexically — hashes, arrays, and scalars.  I think I'd just want to leave room for doing it in the future, but it seems likely to be pretty confusing, to me.
-- 
rjbs

[G.W. Haywood via perl5-porters]

Hi there,

On Sat, 12 Feb 2022, Ovid via perl5-porters wrote:

> ... Perl needs a core way of declaring even complex data structures as immutable ...

You mean, like, efficient constants?

That would be good.

From what little I can remember of the IBM 370/65 at Harwell in 1974 I
think FORTRAN had them.

--

73,
Ged.

[Andrew Hewus Fresh]

On Sat, Feb 12, 2022 at 02:50:49PM +0000, Ovid via perl5-porters wrote:
> Side note: I also think Perl needs a core way of declaring even
> complex data structures as immutable, though I haven't given deep
> thought about this. Something like `immutable %hash = (...);`, but
> that would be a pre-RFC for another day. Suggestions welcome.

I swear I recall someone suggesting "builtin::const" from Const::Fast,
but I can't find it in the archives now. However, folks I trust at work
recommend it highly.

https://metacpan.org/pod/Const::Fast

[G.W. Haywood via perl5-porters]

Hi there,

On Sat, 12 Feb 2022, Ricardo Signes wrote:

> ...
> Open questions:
> * Is it permissible to unexport a lexically exported subroutine? ...

Depends on how you feel about things crashing.

> * Should the lexical exporter be able to export other variable types lexically — hashes, arrays, and scalars. ...

Yes please.

--

73,
Ged.

[Darren Duncan]

I feel that doing immutable structures well is a very deep and fundamental thing
such that you basically want them to be completely separate core data types and
not just variants on existing ones.

For one thing, to be efficient, it would likely be a good idea to implement them
using the flyweight design pattern and/or at least partially symbolically.

So rather than immutable Hash/Array/etc, have a new set of parallel data types,
maybe ball them Map/List/etc.

For my part with my Muldis Data Language I have chosen to make all data types
immutable as a matter of course, so its baked in at the lowest levels, and how
one gets mutability is via a special type or class of types where each value is
a simple reference to an anonymous variable that can be assigned a different
value, and incorporating values of these reference types into data structures is
how one gets opt-in mutability.

-- Darren Duncan

[Salvador Fandiño]

On 12/2/22 14:33, Ricardo Signes wrote:
> On Sat, Feb 12, 2022, at 4:37 AM, Ovid via perl5-porters wrote:

>> Some people expressed interest in the :export functionality of the
>> module proposal. Thus, another pre-RFC.
>>
>> Using Exporter.pm or writing your own import() sub is very clumsy, but
>> very common. This should be native to Perl.
>>
>> The idea is to provide exporting via an `:export` attribute:
>

> I think you're skipping a vital step.  I don't care whether we provide

> an :export attribute.  It seems okay.  But what we /should/ provide is a
> simple /way to export/ a lexical subroutine.  I think we need to provide

> a means for lexical subroutine exports.

Ricardo, when you talk about "lexical subroutine exports" are you
referring to subs that have been defined as lexical in the package where
they are defined or otherwise, you mean subs that you want exported as
lexically subroutines in the calling scope (the one containing the 'use'
statement)? or to both things?

[Salvador Fandiño]

On 12/2/22 14:33, Ricardo Signes wrote:
> On Sat, Feb 12, 2022, at 4:37 AM, Ovid via perl5-porters wrote:

>> Some people expressed interest in the :export functionality of the
>> module proposal. Thus, another pre-RFC.
>>
>> Using Exporter.pm or writing your own import() sub is very clumsy, but
>> very common. This should be native to Perl.
>>
>> The idea is to provide exporting via an `:export` attribute:
>

> I think you're skipping a vital step.  I don't care whether we provide
> an :export attribute.  It seems okay.  But what we /should/ provide is a
> simple /way to export/ a lexical subroutine.  I think we need to provide
> a means for lexical subroutine exports.

Ricardo, when you talk about "lexical subroutine exports" are you
referring to subs that have been defined as lexical in the package where
they are defined or otherwise, you mean subs that you want exported as

lexical subroutines in the calling scope (the one containing the 'use'
statement)? or maybe to both things?

[h...@crypt.org]

Ovid via perl5-porters <perl5-...@perl.org> wrote:

:The idea is to provide exporting via an `:export` attribute:
[...]
:These exports are set via attributes, they do *not* require the import()

:sub to be called, and they are lexical.

I don't understand this last bit - what is it that "does not require"
import() be called?

Most of the examples in the subsequent thread have 'use Foo' to load them,
is the idea that a simple 'require Foo' would put them in your namespace,
and that they would also be imported even with 'use Foo ()'?

Hugo

[Nicholas Clark]

On Sun, Feb 13, 2022 at 02:40:42PM +0000, G.W. Haywood via perl5-porters wrote:
> Hi there,
>

> On Sat, 12 Feb 2022, Dan Book wrote:

> > On Sat, Feb 12, 2022 at 4:15 PM G.W. Haywood via perl5-porters wrote:
> > > On Sat, 12 Feb 2022, Ricardo Signes wrote:
> > > > ...
> > > > Open questions:
> > > > * Is it permissible to unexport a lexically exported subroutine? ...
> > >
> > > Depends on how you feel about things crashing.
> >

> > What do you mean? ...
>
> I mean that when I read posts on this list sometimes I feel a bit like
> a twelve year old mountain boy who's just jumped on his bobsleigh to
> head to the toyshop in town. It seemed like a good idea to him at the
> time but he's starting to wonder what will happen when he gets there.
>
> A whistle-stop history:
>
> 8<-----------------------------------------------
> 2000-03-22: $ grep crash perl5060delta | wc => 0
> 2002-07-18: $ grep crash perl5080delta | wc => 4
> 2007-18-12: $ grep crash perl5100delta | wc => 0
> 2010-04-12: $ grep crash perl5120delta | wc => 3
> 2011-05-14: $ grep crash perl5140delta | wc => 34
> 2012-05-20: $ grep crash perl5160delta | wc => 27
> 2013-05-18: $ grep crash perl5180delta | wc => 30
> 2014-05-27: $ grep crash perl5200delta | wc => 33
> 2015-06-01: $ grep crash perl5220delta | wc => 29
> 2016-05-08: $ grep crash perl5240delta | wc => 6
> 2017-05-30: $ grep crash perl5260delta | wc => 23
> 2018-06-22: $ grep crash perl5280delta | wc => 10
> 2019-05-22: $ grep crash perl5300delta | wc => 1
> 2020-06-20: $ grep crash perl5320delta | wc => 1
> 2021-05-20: $ grep crash perl5340delta | wc => 1
> 8<-----------------------------------------------
>
> Of course this is more than a little superficial but it seems to tell
> me that things went astray for the better part of a decade from 2010,
> before getting back onto an even keel. Hopefully the stability will
> continue to improve, and may even in future be taken for granted, but,
> if development is more a pot pourri of feature-creep than a journey to
> a clearly defined objective, then year on year I fear that we may see
> these numbers creeping back up again.

Your analysis is probably backwards.

We document when we find and fix crashes, not when we create them. (We abhor
regressions, so if a crash is detected as the side effect of a new feature,
that new feature would be reverted rather than ship with a new crash)

You *seem* also to be assuming some correlation between when crashes are
discovered (and documented) and when they were (inadvertently introduced).
For example, that crashes are found the year after they shipped.

If that were true, then yes, one could reasonably argue that things
stabilised in the past 4 years. The reality is that crashes have lain
undiscovered for many years, and some even since perl 5.000 shipped. As
better tools became available (valgrind, ASAN, Coverity, fuzzers) we've been
finding (and fixing) ever more of these subtle bugs. (And hence documenting
them)

So if anything, the counts above *might* indicate that the decade you
describe as "astray" was actually the golden decade, and since then we've
got worse at improving things.


I don't think that this is the reality either. If anything, the incentives
for security researchers to find bugs has increased, and the tools have
continued to improve. So we *ought* to be seeing higher crashes found and
fixed, if the quality of the code remained approximately constant (and so
chance of finding a bug remained constant).


I suspect more what we're seeing here is that it's getting harder to find
crashing bugs. We used to get several reports to perl-s...@perl.org
per year with "fuzzing found this crash". We're now down to one or two a
year. If so, this would suggest that our code quality is improving. Not
getting worse.


Nicholas Clark

[Ovid via perl5-porters]

On Sunday, 13 February 2022, 13:43:27 CET, h...@crypt.org <h...@crypt.org> wrote:

> I don't understand this last bit - what is it that "does not require"
> import() be called?

This proposal isn't to take away import. `use Some::Module` would still call
import and `require Some::Module` would not. This frees use of the `import`
subroutine for anything the author, but doesn't require them to wire anything
together to handle importing.

Most importantly, the old importing should still work unchanged.

> Most of the examples in the subsequent thread have 'use Foo' to load them,
> is the idea that a simple 'require Foo' would put them in your namespace,
> and that they would also be imported even with 'use Foo ()'?

I would imagine that `use Foo ()` would not call `import` but would *not* affect the new importing mechanism. If you want something imported, you have to ask for that explicitly.

But now I see a problem. What if I write a package that has both exporting
*and* tries to use `import` for something else, expecting different arguments?
Do we strip the names of exported symbols from the `import()` argument list,
similar to what `Getopt::Long` and friends do?

Or we we find a way of decoupling a new export functionality and `import` altogether? Or disable `import` if we use this new functionality? 

I think more thought is required here and while I was confident in writing an
RFC before, I'm less sure now.

[Ovid via perl5-porters]

On Saturday, 12 February 2022, 19:00:28 CET, Andrew Hewus Fresh <and...@afresh1.com> wrote:

> > Side note: I also think Perl needs a core way of declaring even
> > complex data structures as immutable, though I haven't given deep
> > thought about this. Something like `immutable %hash = (...);`, but
> > that would be a pre-RFC for another day. Suggestions welcome.
> 
> I swear I recall someone suggesting "builtin::const" from Const::Fast,
> but I can't find it in the archives now. However, folks I trust at work
> recommend it highly.
> 
> https://metacpan.org/pod/Const::Fast

Const::Fast is a great module and I use it frequently.

But given more thought, I'd prefer a new `:immutable` attribute to (avoiding the keyword explosion).

    my %hash :immutable = (...);

[h...@crypt.org]

Ovid <curtis_...@yahoo.com> wrote:

:On Sunday, 13 February 2022, 13:43:27 CET, h...@crypt.org <h...@crypt.org> wrote:
:
:> I don't understand this last bit - what is it that "does not require"
:> import() be called?
:
:This proposal isn't to take away import. `use Some::Module` would still call
:import and `require Some::Module` would not. This frees use of the `import`
:subroutine for anything the author, but doesn't require them to wire anything
:together to handle importing.
:
:Most importantly, the old importing should still work unchanged.
:
:> Most of the examples in the subsequent thread have 'use Foo' to load them,
:> is the idea that a simple 'require Foo' would put them in your namespace,
:> and that they would also be imported even with 'use Foo ()'?
:
:I would imagine that `use Foo ()` would not call `import` but would *not* affect the new importing mechanism. If you want something imported, you have to ask for that explicitly.
:
:But now I see a problem. What if I write a package that has both exporting
:*and* tries to use `import` for something else, expecting different arguments?
:Do we strip the names of exported symbols from the `import()` argument list,
:similar to what `Getopt::Long` and friends do?
:
:Or we we find a way of decoupling a new export functionality and `import` altogether? Or disable `import` if we use this new functionality? 
:
:I think more thought is required here and while I was confident in writing an
:RFC before, I'm less sure now.

Sorry if I'm being stupid, but I don't understand what you said here.

In particular:

I would imagine that `use Foo ()` would not call `import` but would
*not* affect the new importing mechanism.

I infer from "would not affect" that symbols would be imported.

But the next sentence:

If you want something imported, you have to ask for that explicitly.

implies the opposite.

So let me go back to:

package Foo {
sub bar :export {...} # @EXPORT_OK
sub baz :export(:strings) %EXPORT_TAGS
sub quux :export(:ALWAYS) {...} # @EXPORT
sub whee {...} # not exported
}

Is quux() imported by C< use Foo (); >?

I have an additional thought that may resolve my confusion:

I was reading "they do *not* require the import() sub to be called" in
terms of what a _user_ of the module does/doesn't need to do to invoke
it, but I wonder now whether you're talking about what the _author_ of
the module does/doesn't need to _provide_ - ie that the author does not
need to provide an import() sub because the relevant mechanism is provided
automatically by perl (and from the user's point of view will act the same:
exporting behaviour will be triggered either implicitly by 'use' or
explicitly by '->import(...)', but either way providing the user the same
semantics they currently enjoy).

Is that close?

If it is, then I think everything else makes sense to me.

Hugo

[Paul "LeoNerd" Evans]

On Sat, 12 Feb 2022 17:49:12 -0500
"Ricardo Signes" <perl...@rjbs.manxome.org> wrote:

> On Sat, Feb 12, 2022, at 1:33 PM, Dan Book wrote:
> > [unimport] never been done with normal subroutines because deleting
> > symbols from a package often does more harm than good. I would
> > posit that with lexical imports, it's clear and useful to allow.
>
> It may be semantically clear, but I'm not sure it's simple to
> implement. I would like to hear someone wise in the way of the
> internals chime in.
>
> I don't think we ever delete entries from the lexical pad. Maybe we
> can shadow the entry with a tombstone, but again: I don't think it's
> a built-in behavior.

Nothing is ever deleted from a pad. A pad is a data structure owned by
an entire function (or file-level scope). But lexicals definitely do
get hidden, on end-of-scope.

The way that lexicals are hidden from view when their own scope is over
is that every statement inside a function is given a sequence number,
from a monotonically-increasing sequence. Because of the way scopes
work, visibility is determined by storing the min/max sequence number of
statements for which that pad slot should be visible. Upon reaching a
close-of-scope brace, all the variables that now need to become
invisible have the current sequence number written into their "max"
field.

It would be quite simple to implement `no lexically ...` by hiding
imported lexicals in the same way - just write in the current sequence
number into the pad.

[G.W. Haywood via perl5-porters]

Hi there,

On Sun, 13 Feb 2022, Paul "LeoNerd" Evans wrote:
> On Sun, 13 Feb 2022 12:55:45 -0500 Dan Book <gri...@gmail.com> wrote:
>
>>> Shouldn't much of the new stuff be slated for Perl versions 7, 8,
>>> ... while Perl 5 goes into long term maintenance?
>>>
>> This is no longer the plan.
>
> If anything, (and I say this with my PSC hat on), the plan is
> rathermuch the converse. That we keep adding and extending features and
> generally ...

Would it be too unfair to represent that as

"We don't know where our product is going, but we'll know when it gets there."

and, if you were a salesman, would you want to be in the business of
selling this product?

--

73,
Ged.

[Darren Duncan]

On 2022-02-13 4:22 p.m., G.W. Haywood via perl5-porters wrote:
> Would it be too unfair to represent that as
>
> "We don't know where our product is going, but we'll know when it gets there."
>
> and, if you were a salesman, would you want to be in the business of
> selling this product?

Considering this is a volunteer-run product with limited resources, I feel it is
perfectly reasonable to take it very few steps at a time and not declare
timelines beyond the next few months that one can be fairly certain about. We
don't really know what will actually be implemented and ready further out than a
few months, practically speaking, from what I've seen. So it really is best to
decide at the time sufficient features are already merged when to call it
version 7 or whatever. Holding it off for certain anticipated but yet
unimplemented things is not the best idea, as those could be delayed, or
sufficient others could come meanwhile. -- Darren Duncan

[Ovid via perl5-porters]

On Sunday, 13 February 2022, 21:47:09 CET, h...@crypt.org <h...@crypt.org> wrote:

> Sorry if I'm being stupid, but I don't understand what you said here.

You're not being stupid at all. It's a more confusing topic than I realized and your questions are helping to bring clarity. Thank you.

> So let me go back to:
>   package Foo {
>     sub bar :export {...} # @EXPORT_OK
>     sub baz :export(:strings) %EXPORT_TAGS
>     sub quux :export(:ALWAYS) {...} # @EXPORT
>     sub whee {...} # not exported
>   }
> Is quux() imported by C< use Foo (); >?

I _think_ that the correct thing to do is to completely decouple the behavior of `:export` modifiers and `import()`. Thus, in the above, `quux` is always imported, even if you do `use Foo ()` because the latter prevents the `import` from being invoked, but that has no bearing on `:export`.

If modules abuse `:export(ALWAYS)`, the lexical nature of the new mechanism might help.

    {
        use Foo (); # we still have `quux`
        ...
    }
    # we no longer have `quux`

How well this would work in practice remains to be seen.

> I have an additional thought that may resolve my confusion:
> 
> I was reading "they do *not* require the import() sub to be called" in
> terms of what a _user_ of the module does/doesn't need to do to invoke
> it, but I wonder now whether you're talking about what the _author_ of
> the module does/doesn't need to _provide_ - ie that the author does not
> need to provide an import() sub because the relevant mechanism is provided
> automatically by perl

That is correct. I think `import` should be decoupled entirely from the new mechanism (though it means that `import()` is now poorly named).

> (and from the user's point of view will act the same:

From the user's POV, it will be *almost* entirely the same.

> exporting behaviour will be triggered either implicitly by 'use' or
> explicitly by '->import(...)', but either way providing the user the same
> semantics they currently enjoy).
> 
> Is that close?
> 
> If it is, then I think everything else makes sense to me.

Calling `->import` would not affect any behavior for `:export`. You cannot block a naughty `:export(:ALWAYS)`, but you can trap it in a block. However, both the old style and the new style, while decoupling `:export` from `import()`, they're still fighting for the same syntax:

        use Module LIST

So that's the core issue here. What gets passed to `import()`?

Going back to the example:

   package Foo {
     sub bar  :export {...}          # @EXPORT_OK

     sub baz  :export(:strings)      # %EXPORT_TAGS

     sub quux :export(:ALWAYS) {...} # @EXPORT
     sub whee {...}                  # not exported
   }

There, we have no `import()`, so it doesn't matter. But what happens if we add
that?

    use Foo 'bar', 'baz', 1, 2;

Does `import()` get `qw(Foo bar baz 1 2)` or `qw(Foo 1 2)`? I _think_ the best solution is that we intercept and strip relevant entries from the argument list so the author doesn't have to try to remember to do that every time they want to write an `import()` method.

[Darren Duncan]

On 2022-02-14 12:57 a.m., Paul "LeoNerd" Evans wrote:
> On Mon, 14 Feb 2022 04:17:25 +0100
> demerphq <deme...@gmail.com> wrote:
>
>> But i wonder:
>>
>> my $hash= $cond ? { foo => 1 }:ro : { bar => 2}:rw;
>>
>> seems problematic.
>
> As a brief aside: This is one of the reasons why Raku renamed the
> ternary condition operator's spelling to COND ?? TRUE :: FALSE
> because otherwise that sort of colon syntax gets in the way far too
> often.
>
> With cond renamed thusly, you're now free to put attributes in a lot
> more places than before;
>
> my $hash = $cond ?? { foo => 1 } :ro :: { bar => 2 } :rw;

Interesting, I had thought it was mainly about huffman coding to make the less
frequently used ternary doubled up. But what you said makes a lot of sense. --
Darren Duncan

[Ovid via perl5-porters]

> my $hash= $cond ? { foo => 1 }:ro : { bar => 2}:rw;

I think Yves' comment about having mutable and immutable data structures mixed is spot-on. We can do that now, but I doubt that people use immutable structures enough that the problem has surfaced. Discovering that `$hash->{foo} = 2` is fatal and `$hash->{bar} = 3 ` not fatal will be a minefield. It also implies (to me) that we need an easy way to test for mutability.

We can't prevent this confusion other than to document explain "best practices":

    my %hash1 :immutable = (...);
    my %hash2 = (...);

    # Welcome to the bugfest
    my %hash3 = (
        mine  => \%hash1,
        field => \%hash2,
    );

So recommendations to not mix mutable and immutable data structures would be important. To my mind, that means `:immutable` would imply deep immutability.

One our use cases is that we have "exchanges" in our Tau Station codebase that build up messages of various types. Once those messages escape the exchange system, they very much want to be immutable because they've been assembled into their canonical form.

We pass these to our front end via JSON, so immutability doesn't matter since we're effectively cloning them, but if we pass them to other parts of our system, we don't want invisible action at a distance breaking our code.

So we want immutability to be declarable at runtime, being able to apply once it crosses a trust boundary (e.g., our exchanges know how to build messages, but we can't risk the rest of our code making a mistake). Thinking about all of this and further examples from Yves, we either skip the idea of immutability in core (which I think would be bad), or ...

* `:immutable` is deep immutability
* Need an easy way to ask if a structure was declared with `:immutable`
* Best practices in docs (don't mix mutable and immutable)
* `exists` would still recommended for testing data structures, but ...
* ... an immutable `$hash->{no}{such}[$index]{path}` should short-circuit and return false on the first missing element
    * Without being fatal
    * Without auto-vivifying anything (because that would be mutable)

If we apply immutability at runtime, it means this might be slow.

    # this implies a deep clone
    my %immutable :immutable = %hash;

    # this does not clone (but we're slurping in a list? Hmm,
    # maybe we need a keyword after all...
    %hash :immutable = %hash;

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

[G.W. Haywood via perl5-porters]

Hi there,

On Mon, 14 Feb 2022, Paul "LeoNerd" Evans wrote:
> On Mon, 14 Feb 2022 04:17:25 +0100
> demerphq <deme...@gmail.com> wrote:
>

>> ... my $hash= $cond ? { foo => 1 }:ro : { bar => 2}:rw;

>> seems problematic.
>
> As a brief aside: This is one of the reasons why Raku renamed the
> ternary condition operator's spelling to COND ?? TRUE :: FALSE

> ...

As I'm sure will have been observed already a change like that

(a) breaks just about everything that's ever been written and

(b) makes for yet another irritation where the exact same idea
is expressed in what appears to be a gratuitously different way.

It's tempting to say that more characters are needed in the set.

APL, anyone?

https://en.wikipedia.org/wiki/Digital_encoding_of_APL_symbols

/ducks for cover

--

73,
Ged.

[G.W. Haywood via perl5-porters]

Hi there,

On Mon, 14 Feb 2022, Ovid via perl5-porters wrote:

> # ... maybe we need a keyword after all...
> %hash :immutable = %hash;

How about

%hash := immutable;

?

/where's that coat?

--

73,
Ged.

[h...@crypt.org]

Ovid <curtis_...@yahoo.com> wrote:

:On Sunday, 13 February 2022, 21:47:09 CET, h...@crypt.org <h...@crypt.org> wrote:
:
:> Sorry if I'm being stupid, but I don't understand what you said here.
:
:You're not being stupid at all. It's a more confusing topic than I realized and your questions are helping to bring clarity. Thank you.
:
:> So let me go back to:
:> package Foo {
:> sub bar :export {...} # @EXPORT_OK
:> sub baz :export(:strings) %EXPORT_TAGS
:> sub quux :export(:ALWAYS) {...} # @EXPORT
:> sub whee {...} # not exported
:> }
:> Is quux() imported by C< use Foo (); >?
:
:I _think_ that the correct thing to do is to completely decouple the behavior of `:export` modifiers and `import()`. Thus, in the above, `quux` is always imported, even if you do `use Foo ()` because the latter prevents the `import` from being invoked, but that has no bearing on `:export`.

Thanks for the clarification. I would regret that behaviour: I think the
user of a module _should_ have the option to control what is imported.
In particular I care a lot about discoverability, so I tend either to
fully qualify stuff:
use Foo ();
my $quuxed = Foo::quux($source);
or explicitly to list symbols to import:
use Foo qw{ quux };
my $quuxed = quux($source);
so that it is always easy to find what is providing a function by searching
within the file that invokes it.

I developed and promoted that habit particularly in work contexts, because
so often it was unobvious which of 30 modules listed at the top of a file
was providing a given function.

This aspect of the proposal would lose the benefits of that approach.
Moving the 'use' statement to the point of use to take advantage of
lexicality could partially bring it back, but at the cost of no longer
being able to see easily at the top of a file the externals it relies on.

Hugo

[G.W. Haywood via perl5-porters]

Hi there,

On Mon, 14 Feb 2022, h...@crypt.org wrote:

> ... so that it is always easy to find what is providing a function by searching

> within the file that invokes it.

+1

--

73,
Ged.

[Darren Duncan]

On 2022-02-14 2:43 a.m., Ovid via perl5-porters wrote:
> So recommendations to not mix mutable and immutable data structures would be important. To my mind, that means `:immutable` would imply deep immutability.

I agree with huffmanizing for deep immutability. It should still be easy enough
to do mixed, but give the most "friendly" syntax the deep immutability meaning.

It occurs to me there is a complicating factor that we need to explicitly define
the behavior of, and that is what happens when we have BLESSED hashes etc.

So far existing discussion has implied non-blessed ones but we need to define
how things work for both. For example, if someone defines a nested structure
including blessed hashes and the outer structure is a non-blessed hash with
":immutable", what does that mean for the blessed hashes?

Also, would the attempt to either bless or unbless a hash that is immutable fail
because that is logically a mutation of the hash? I would say yes probably.

> If we apply immutability at runtime, it means this might be slow.
>
>     # this implies a deep clone
>     my %immutable :immutable = %hash;
>
>     # this does not clone (but we're slurping in a list? Hmm,
>     # maybe we need a keyword after all...
>     %hash :immutable = %hash;

I strongly recommend that we don't break existing expectations on that "%foo =
%bar" or "%foo = (...)" do, which is ALWAYS make a NEW hash with SHALLOW copies
of the elements, and any shallow immutable attribute of the right side doesn't
copy over, though any immutable attributes of elements still do.

If you want something that doesn't clone, we already have that with "hashrefs",
such as "$rh_foo = $rh_bar" or "$rh_foo = {...}" or "$rh_foo = \%bar" etc. When
you do THAT, then it is not a clone and the immutable attribute carries over,
otherwise no.

Your examples I would expect to make a NEW hash which is a SHALLOW clone and
THEN the new hash is SHALLOWLY immutable.

The only way to make the attribute deeply recursive is when it is on the right
hand side when the hashes are instantiated in the first place.

Now if you want to do any kind of deep cloning, that should be a NEW OPERATOR,
and NOT "=". Better yet, it should be a function which returns the deep clone,
and then that result can either be nested in an expression or assigned to a
variable, so much more versatile than an assignment replacement, because what if
someone wants to clone without an assignment.

-- Darren Duncan

[G.W. Haywood via perl5-porters]

Hi there,

On Mon, 14 Feb 2022, Darren Duncan wrote:

> ... if you want to do any kind of deep cloning, that should be a NEW
> OPERATOR ...

:=

--

73,
Ged.