Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

disabling smartmatch and when()?

119 views
Skip to first unread message

Dave Mitchell

unread,
Jun 20, 2022, 10:00:04 AM6/20/22
to perl5-...@perl.org
So...

A while back, we collectively agreed that the smartmatch operator
and the when() keyword (which sometimes uses smartmatch under the hood),
had fundamental design flaws, and we retrospectively made
~~/given/when/break experimental in 5.18.0 (nine years ago!).

Since then we have failed to reach a consensus as to how these should be
fixed, or indeed whether they should removed altogether. This is an
unhappy state of affairs.

I propose that right now (so appearing in 5.38.0), we disable C<~~> and
when(), and possibly C<given> and C<break> too. Any attempt to use them
will give a meaningful compile-time error (as opposed to a plain syntax
error).

This will provide two positive outcomes. First, it will force users to
stop using it. Second, it provides an air-gap for if and when we introduce
the new and improved smart-match operator - i.e. rather the behaviour just
changing in one release, there is a clear demarcation between the old and
new behaviour, with people who test their new code on old releases getting
a clear signal, in that their code won't even compile on 5.38 ..5.44 say.

Important note: please, PLEASE, don't turn this thread into yet another
discussion on how to revamp smartmatch: if you want to discuss that,
start a new thread (and read all the old threads about it first). This
thread is purely to discuss whether and how to disable smartmatch until
something better comes along (which could in theory be included in 5.38 if
the other hypothetical thread bears fruit).

So I think that at a minimum, the use of C<~~> and <when() should give
meaningful compile-time errors. The main issue is whether we should also
disable C<given> and C<break> too. Also, whether 'C<use feature 'switch'>
should be an error or warning too.

Any opinions?

Lastly, here's a handy summary history of what happened to smartmatch and
the switch feature in various perl releases.

--------------------------------------------------------

5.10.0

C<~~> and the given/when/break keywords were added.
The new keywords need C<use feature 'switch'>, while C<~~> doesn't.

<given($foo) { ...}> is like C<for($foo) {...}>, except that it assigned
C<$foo> to a localised C<$_> rather than aliasing it: so modifying C<$_>
within the block didn't change C<$foo>. (If a lexical C<$_> is in scope,
then that is the variable which was assigned to.)

C<when ($foo)> is interpreted as C<when($ ~~ $foo)>, with some exceptions
for common cases where it acts directly as a boolean, e.g.
C<when(defined($foo))>, C<when(/foo/)> etc.

The arguments to C<~~> were treated as commutative, i.e. C<$a ~~ $b>
and C<$b ~~ $a> gave the same result.

-----

5.10.1 / 5.12.0

Major changes were made to smartmatch; in particular, it is no longer
commutative; instead, the action is based largely on the type of the RHS.
For example this matches in 5.10.0 but not in 5.10.1:

/foo/ ~~ "foo"

More cases were added where when()'s argument is treated as a boolean,
rather than as the argument to a smartmatch:

when (/^=begin/ .. /^=end/) { ... }
when ((expr1 // expr2) { ... }

-----

5.12.0

-DM smartmatch resolution tracing added.

-----

5.16.0

Bug fix: C<~~> now correctly handles the precedence of Any~~Object, and is
not tricked by an overloaded object on the left-hand side.

Pod for smartmatch fixed and moved to perlop

-----

5.18.0

The following all became retrospectively experimental:
~~
when/given/break;
lexical $_

when() changed to be like for() in that it now aliases C<$_> to the
value rather than copying.

-----

5.24.0

lexical $_ is removed.

-----

5.28.0

During development, zefram merged a major 'simplify smartmatch' branch,
but this was quickly reverted as it was decided that a consensus hadn't
been reached.

--
"Emacs isn't a bad OS once you get used to it.
It just lacks a decent editor."

Paul "LeoNerd" Evans

unread,
Jun 20, 2022, 10:00:04 AM6/20/22
to perl5-...@perl.org
On Mon, 20 Jun 2022 14:45:48 +0100
Dave Mitchell <da...@iabyn.com> wrote:

> So...
>
> A while back, we collectively agreed that the smartmatch operator
> and the when() keyword (which sometimes uses smartmatch under the
> hood), had fundamental design flaws, and we retrospectively made
> ~~/given/when/break experimental in 5.18.0 (nine years ago!).
>
> Since then we have failed to reach a consensus as to how these should
> be fixed, or indeed whether they should removed altogether. This is an
> unhappy state of affairs.
>
> I propose that right now (so appearing in 5.38.0), we disable C<~~>
> and when(), and possibly C<given> and C<break> too. Any attempt to
> use them will give a meaningful compile-time error (as opposed to a
> plain syntax error).

Perl 5.36 already took a good step in this direction. From
perl5360delta.pod:

>> Finally, with this release, the experimental switch feature,
>> present in every feature bundle since they were introduced in
>> v5.10, has been removed from the v5.36 bundle. If you want to use
>> it (against our advice), you'll have to enable it explicitly.

--
Paul "LeoNerd" Evans

leo...@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

Dave Mitchell

unread,
Jun 20, 2022, 11:00:04 AM6/20/22
to Tomasz Konojacki, perl5-...@perl.org
On Mon, Jun 20, 2022 at 04:10:12PM +0200, Tomasz Konojacki wrote:
> I'm not sure what exactly you are proposing.
>
> If it's turning ~~ into an enabled by default feature flag (although
> disabled in the ":5.38" bundle) à la "indirect", then +1 from me.

I mean that smartmatch and when() are completely removed, cannot be
(re)enabled, and if used will always produce a compile-time error.

> If you actually want to remove it, then I'm *strongly* opposed. We can't
> remove smartmatch until we provide a replacement *and* it goes through a
> proper deprecation cycle (and by "proper" I mean much longer than the
> usual 2 release cycles).

It's gone through a 9 year deprecation cycle, with a mandatory
experimental warning that you have to explicitly disable.

Whether we remove it or change it, its going to break any existing code
which uses it.

> Smartmatch is *very* widely used in production code and we *completely* failed
> to communicate that it's on its way out.

We have been telling people not to use it for nearly a decade.

--
Music lesson: a symbiotic relationship whereby a pupil's embellishments
concerning the amount of practice performed since the last lesson are
rewarded with embellishments from the teacher concerning the pupil's
progress over the corresponding period.

Dave Mitchell

unread,
Jun 20, 2022, 1:15:04 PM6/20/22
to Tomasz Konojacki, perl5-...@perl.org
On Mon, Jun 20, 2022 at 05:54:52PM +0200, Tomasz Konojacki wrote:
> On Mon, 20 Jun 2022 15:43:44 +0100
> Dave Mitchell <da...@iabyn.com> wrote:
>
> > It's gone through a 9 year deprecation cycle, with a mandatory
> > experimental warning that you have to explicitly disable.
>
> So signatures were deprecated too? "Experimental" doesn't mean
> deprecated. If we want smartmatch to be deprecated, then we should
> deprecate it. We should have done that a long time ago, instead of
> pretending that signatures/try/isa/whatever have the same status as
> smartmatch.
>
> Smartmatch was made "experimental" retroactively and for a large portion
> of our users the warnings started appearing as late as 10 years after it
> was introduced!

Smartmatch was introduced in 2007 5.10.0).
It's behaviour was radically changed in 2009 (5.10.1/5.12.0).
It started having a mandatory warning in 2013 (5.18.0).

> RHEL6: released in 2010, supported until 2020: perl 5.10 (no warnings)
> RHEL7: released in 2014, supported until 2024: perl 5.16 (no warnings)
> RHEL8: released in 2019, supported until 2029: perl 5.26 (warns)

RHEL is hardly a relevant comparison. It is an extremely conservative
linux server distribution whose software is usually years out of date.
Virtually no one will have *developed* (as opposed to *deployed*) perl
code on RHEL. People who write CPAN modules won't have developed it on
RHEL, and even if they originally developed it on <= 5.16.0 it's unlikely
that they never tried it on any later perl (apart from abandoned modules),
nor ignored all the cpantesters reports.

> Not to mention that the "experimental" warning doesn't even attempt to
> inform the user what it means:
>
> > perl -E '$z ~~ $b'
> Smartmatch is experimental at -e line 1.

If I were to start coding in python or rust or something, and when I first
used a feature it spat out a warning about it being experimental, I would
exhibit some degree of caution.

> Our documentation isn't much better. The meaning of "experimental" is
> hidden deep in perlpolicy. There is no *loud and clear* warning in e.g.
> the smartmatch section of perlop.

perlop's entry for smartmatch says (and has since 5.20.0), in the first
paragraph:

The smartmatch operator is L<experimental|perlpolicy/experimental> and
its behavior is subject to change.

So, it warns coders that their code may break on future perl releases, and
directs them to the perlpolicy section on experimentalness, which says in
the first sentence:

If something in the Perl core is marked as B<experimental>, we may
change its behaviour, deprecate or remove it without notice

Note the "remove without notice".

Similarly in perlsyn, there is this:

=head2 Switch Statements

....
Exactly what the I<EXPR> argument to C<when> does is hard to describe
precisely
...
The rules are far too difficult to understand to be described here.
See L</"Experimental Details on given and when"> later on
....


=head2 Experimental Details on given and when

As previously mentioned, the "switch" feature is considered highly
experimental; it is subject to change with little notice. In particular,
C<when> has tricky behaviours that are expected to change to become less
tricky in the future.


> I can't count the times I had to explain to people outside of the p5p
> bubble that "experimental" in the case of smartmatch actually means
> "deprecated" and they shouldn't use it.

Perhaps these people have no business being coders?

> The result of the above is that half of CPAN relies on smartmatch:
>
> https://grep.cpanauthors.org/search?q=%7E%7E+OR+%22given+%28%22+OR+%22given%28%22
>
> And that is just CPAN, there's certainly a lot of it in darkpan code too.

Good grief! Many of those are false positives, but people do seem to be
using it with gay abandon. That's a problem, certainly.

> The worst part is that we can't just say "use <X> instead" because there
> is no direct replacement for smartmatch and given/when! For each of the
> affected modules you have to analyse the code and figure out which one
> of the bajillion smartmatch comparison modes was being triggered and
> then you have to replace it with something verbose and ugly (e.g.
> List::Util::any or if/else-chain), which BTW, is not a good look for
> Perl at all.

At least if we remove smartmatch/when completely they'll *know* their code
needs fixing. If we just change its behaviour, we could silently break
their code - often in hard-to-detect edge cases. For example, their
module may continue to fine 99.9% of the time, till someone passes an
overloaded object to one of the module's methods and it suddenly breaks.

Perhaps we should add a new mandatory deprecated warning in addition to
the experimental warning, then remove smartmatch in 3 or so releases time?

PS - I would hazard a guess that the vast bulk of ~~ uses on CPAN can be
replaced with grep or similar, and that most uses of give/when can be
replaced with a chain of if/elsif/else. I suspect that very few places
make use of the recursive smartmatch feature.

In conclusion:

1) we can't do nothing, and the longer we leave it the worse it gets.

2) I think it is better to have a break where smartmatch/when() is a
compile-time error for a few releases before introducing a new smartmatch
with radically different behaviour, rather than just switching to the new
behaviour with no gap.

3) I am not opposed to a further deprecation cycle (possibly with a new
mandatory warning), before we disable it.

--
Standards (n). Battle insignia or tribal totems.

Martijn Lievaart

unread,
Jun 20, 2022, 2:15:03 PM6/20/22
to perl5-...@perl.org
Op 20-06-2022 om 18:58 schreef Dave Mitchell:
> On Mon, Jun 20, 2022 at 05:54:52PM +0200, Tomasz Konojacki wrote:
>> RHEL6: released in 2010, supported until 2020: perl 5.10 (no warnings)
>> RHEL7: released in 2014, supported until 2024: perl 5.16 (no warnings)
>> RHEL8: released in 2019, supported until 2029: perl 5.26 (warns)
> RHEL is hardly a relevant comparison. It is an extremely conservative
> linux server distribution whose software is usually years out of date.
> Virtually no one will have *developed* (as opposed to *deployed*) perl
> code on RHEL.

I work for a very large high tech manufacturing firm. Yup, we develop
with Perl on RHEL7. We are probably not the only ones. People use RHEL
*because* it is a conservative Linux distribution.


M4


Darren Duncan

unread,
Jun 20, 2022, 3:00:03 PM6/20/22
to perl5-...@perl.org
It seems to me that the low hanging fruit is to make Perl 5.38 issue very
explicit "deprecation" warnings for any code using smartmatch/etc, instead of
saying "experimental".

Also the documentation can be updated at the same time which suggests
alternatives for each of the smartmatch/etc use cases that already work now.
Some of these may be more verbose while other common ones may not be. Worst
case scenario, they can always be replaced with if/else chains or such.

A more concise and well designed replacement for smartmatch/etc may be coming
later, but in the meantime people can still be much more strongly and explicitly
encouraged to not use the deprecated smartmatch/etc.

Then the tendency of the existing or new codebases to use smartmatch/etc would
have a tendency to shrink rather than grow, especially in CPAN modules, which
would be a positive development regardless of any other developments such as a
replacement being made.

-- Darren Duncan

Darren Duncan

unread,
Jun 20, 2022, 3:30:06 PM6/20/22
to perl5-...@perl.org
On 2022-06-20 11:44 a.m., Darren Duncan wrote:
> It seems to me that the low hanging fruit is to make Perl 5.38 issue very
> explicit "deprecation" warnings for any code using smartmatch/etc, instead of
> saying "experimental".

Actually, would it be considered a breaking or inappropriate change to do this
in a Perl 5.36.1 release?

One could argue that updating the text of the "experimental" warning for
smartmatch/etc to more explicitly say that the feature is "deprecated" instead,
that this should not be a breaking change, and it would be very beneficial by
getting that messaging in front of users a lot sooner.

Anyone updating to 5.36 anyway would likely keep up with patch versions,
including packagers for Linux distros and such.

The poor user text is effectively a bug that would be reasonable to fix early.

Likewise, patch releases for any other still supported Perl releases could have
that updated user text. But at least for 5.36 since it just came out.

Note that my proposal here isn't meant to be a rationale for a much bigger
change to happen in 5.38, such as the compile time error proposal for
smartmatch/etc, but that the user text update in all supported branches in
addition to 5.38 would help get it seen by more people and sooner.

Would that work?

-- Darren Duncan

SHIRAKATA Kentaro

unread,
Jun 20, 2022, 6:15:03 PM6/20/22
to perl5-...@perl.org
How about declaring smartmatch as "discouraged" instead?

From https://perldoc.perl.org/perlpolicy#discouraged:

---

discouraged

From time to time, we may mark language constructs and features which we consider to have been mistakes as discouraged. Discouraged features aren't currently candidates for removal, but we may later deprecate them if they're found to stand in the way of a significant improvement to the Perl core.

---

I think it meets this criterion.

G.W. Haywood via perl5-porters

unread,
Jun 24, 2022, 10:30:05 AM6/24/22
to Ricardo Signes via Perl 5 Porters
Hi there,

On Fri, 24 Jun 2022, Ricardo Signes wrote:
> On Tue, Jun 21, 2022, at 04:45, Paul "LeoNerd" Evans wrote:
>> I think there's two things we need to do:
>>
>> 1) Make ~~ and given/when much louder about their "you shouldn't use
>> this for new code" status;
>
> Agreed. Let's do that immediately?
>
>> 2) Create a better replacement
>
> I'm all for having a better replacement, but I continue to strongly oppose the idea that we *need* one to remove the old thing.
>
> I write a lot of Perl, and have since before smartmatch was introduced. I never used smartmatch for much. I have banned it from my projects. I have seen it largely unused. Nothing it does can't be done some other way already, although sometime more verbosely. We don't need a replacement, because *we have gotten along fine without using it for 20 years.*
>
> A replacement would be nice, if it's good, but it doesn't need to exist before we take the old thing out.

My sentiments exactly.

I briefly flirted with 'given'. In 2017 I abandoned it, and swore that it would be forever.

Even if there's a new one, I doubt I'd take the risk.

--

73,
Ged.

Paul "LeoNerd" Evans

unread,
Jun 24, 2022, 11:30:05 AM6/24/22
to perl5-...@perl.org
On Fri, 24 Jun 2022 10:07:25 -0400
"Ricardo Signes" <perl...@rjbs.manxome.org> wrote:

> On Tue, Jun 21, 2022, at 04:45, Paul "LeoNerd" Evans wrote:
> > I think there's two things we need to do:
> >
> > 1) Make ~~ and given/when much louder about their "you shouldn't use
> > this for new code" status;
>
> Agreed. Let's do that immediately?

Righty. I'll see what I can hack up.

> > 2) Create a better replacement
>
> I'm all for having a better replacement, but I continue to strongly
> oppose the idea that we *need* one to remove the old thing.
...
> A replacement would be nice, if it's good, but it doesn't need to
> exist before we take the old thing out.

Agree - these are two separate things. We can do them independently.

> I write a lot of Perl, and have since before smartmatch was
> introduced. I never used smartmatch for much. I have banned it from
> my projects. I have seen it largely unused. Nothing it does can't
> be done some other way already, although sometime more verbosely. We
> don't need a replacement, because *we have gotten along fine without
> using it for 20 years.*

Yah I'm inclined to agree. But also we're honestly most of the way
there in terms of designing nicer replacements.

Most of the weird funky cases of the ~~ operator (e.g. the array
distributative nature of $str ~~ @arr) can be handled with things like
List::Util::all - of which I hope to have a nicer core syntax soon.
Plus once I finish working out how to do parametric syntax using
XS::Parse::Infix, I'll have the `in<eq>` and `in<==>` operators. For
now there's `$str elem @arr_of_strings` and `$num ∈ @arr_of_nums`
operators (yes the latter requires UTF-8, because I couldn't find a
free sequence of ASCII symbols that makes sense) provided by
Syntax::Operator::Elem (which I really should get onto CPAN soon).

Most of the simpler uses of given/when are just as tidy with my
match/case syntax, provided by Syntax::Keyword::Match. It's an evolving
design that I hope to drag into core sometime, much like the design
process that lead to try/catch in core.

Paul "LeoNerd" Evans

unread,
Jun 24, 2022, 12:15:04 PM6/24/22
to perl5-...@perl.org
On Fri, 24 Jun 2022 16:11:58 +0100
"Paul \"LeoNerd\" Evans" <leo...@leonerd.org.uk> wrote:

> On Fri, 24 Jun 2022 10:07:25 -0400
> "Ricardo Signes" <perl...@rjbs.manxome.org> wrote:
>
> > On Tue, Jun 21, 2022, at 04:45, Paul "LeoNerd" Evans wrote:
> > > I think there's two things we need to do:
> > >
> > > 1) Make ~~ and given/when much louder about their "you shouldn't
> > > use this for new code" status;
> >
> > Agreed. Let's do that immediately?
>
> Righty. I'll see what I can hack up.

Hmm. Well, first off: we don't have a "discouraged" warning category.

One thing we do have is "deprecated", which I think current policy
requires us to make a formal statement in pod/perldeprecation.pod and
give a timeline for eventual removal. Do we want to commit to that?

Alternatively, I could add a new "discouraged" category. Though I think
I recall trying that sometime before and we eventually decided it
should just be deprecated anyway.
0 new messages