Backporting lexical export to Perl v5.34

[Paul "LeoNerd" Evans]

TL;DR: This is a long sequence of "I tried X and it didn't work", that
ends in "Can anyone help?" request.


I'm attempting to backport the `builtin` stuff to perls before 5.36.
For the most part it's all actually really easy - lots of things like
`true` and `false` are just constants, most of the other funcs are
copied from Scalar::Util, etc.

One hard part is the actual lexical exporting mechanism, which is used
by `builtin` itself and offered to users via the
builtin::export_lexically function.


## ATTEMPT 1

A simple copy of the behaviour from 5.36 works in simple cases, but
fails on any of the more complex closure-wrapping tests, because
clone_cv() segfaults when attempting to clone the pad, because of
attempting to bump the OpREFCNT of an XSUB (which of course doesn't
have an optree). This is the very part of the code that was fixed as
part of adding builtin to v5.36; in this little diff chunk:

https://github.com/Perl/perl5/pull/19232/files#diff-d6972c2c727b9f7dfb3dc6c58950ad9e884aeaa7464c1dfe70ed0c7512719e7fR2212-R2226

So this means we cannot lexically export an XSUB. Well that's no huge
trouble; we can work around this by injecting a little trampoline
pureperl sub, such as `sub { goto &$cv }` to make that call for us.


## ATTEMPT 2

Alright, so lets do this. Rather than injecting SvREFCNT_inc(orig_sv)
into the importing pad, we'll create a wrapper by basically doing

OP *body = newLISTOP(OP_LINESEQ, 0, NULL, NULL);
body = op_append_elem(OP_LINESEQ, body,
newSTATEOP(0, NULL, NULL));
body = op_append_elem(OP_LINESEQ, body,
newUNOP(OP_GOTO, OPf_STACKED,
newSVOP(OP_CONST, 0, newRV_inc(orig_cv))));

CV *wrapper_cv = newATTRSUB(..., body);

and injecting that. ((it already annoys me that the sequence of
operations required to do that is -quite- so long and complex but eh;
that's a rant for another day))

This works fine for simple cases like builtin::true or
builtin::blessed, but fails for builtin::indexed, because the invoked
sub always sees GIMME_V == G_SCALAR, regardless of the original calling
context.

Turns out this might be because the OP_GOTO had OPf_WANT_SCALAR stuck
onto it for some odd reason. No idea why. I tried clearing that by doing

o->op_flags &= ~OPf_WANT;

to un-contextualize it and put it back into UNKNOWN context, but now
the invoked xsub always sees GIMME_V == G_VOID instead, so that's not
much help. I vaguely recall that XSUBs don't get a real context on the
caller stack anyway, so... maybe this is why. Perhaps we can't really
goto into those and we'll have to use a real entersub op. OK


## ATTEMPT 3

Instead of OP_GOTO, I'm now trying to create a real calling trampoline,
the effect of `sub { $cv->(@_) }`, by doing:

OP *callop = newLISTOP(OP_LIST, 0, NULL, NULL);
callop = op_append_elem(OP_LIST, callop,
newUNOP(OP_RV2AV, OPf_WANT_LIST|OPf_MOD, newGVOP(OP_GV, 0, PL_defgv)));
callop = op_append_elem(OP_LIST, callop,
newSVOP(OP_CONST, 0, newRV_inc(orig_sv)));
callop = op_convert_list(OP_ENTERSUB, OPf_STACKED, callop);

OP *body = newLISTOP(OP_LINESEQ, 0, NULL, NULL);
body = op_append_elem(OP_LINESEQ, body,
newSTATEOP(0, NULL, NULL));
body = op_append_elem(OP_LINESEQ, body,
callop);

/* same newATTRSUB as before */

This finally solves the arguments in, results out, and context, in all
of void, scalar and list context. `builtin::indexed` works just fine.
Great.. We're done.. right?

... except, no. Now, warnings don't work properly any more. Because
there really is an extra caller context involved here, when any of the
builtin functions conditionally try to print warnings by calling
Perl_ck_warner_d(), they see the scope of that generated `sub { ... }`
wrapper, rather than the real caller, so no amount of `no warnings ...`
in the real caller actually helps.

## THOUGHT 4

I haven't tried writing code yet, but looking at the way
Perl_ck_warner() calls ckwarn_common(), the latter just looks in
PL_curcop->cop_warnings, so I suppose I could *additionally* wrap all
of my trampolined code in

ENTER;
SAVEVPTR(PL_curcop);
PL_curcop = (fetch the prevcop out of the caller stack);

...

LEAVE;

but at this point this is feeling like an ever-growing large pile of
mess piled on top of more mess.


One thought I've had is that I could implement all of the `builtin::`
funcs as little "pureperl" CVs that have an optree that contains just a
single OP_CUSTOM to contain the code, rather than them being XSUBs.
That would be a weird code structure but it might work, and at least it
would mean these builtin subs are not XSUBs and thus lexically
importable by older perls. It would still leave
`builtin::export_lexically` itself unable to export actual XSUBs, but
that would at least be "the user's problem", and can just be documented
as a known limitation.


Can anyone suggest a different approach at any point which might work
better?

--
Paul "LeoNerd" Evans

leo...@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

[James E Keenan]

On 7/27/23 07:40, Paul "LeoNerd" Evans wrote:
> TL;DR: This is a long sequence of "I tried X and it didn't work", that
> ends in "Can anyone help?" request.
>
>
> I'm attempting to backport the `builtin` stuff to perls before 5.36.
> For the most part it's all actually really easy - lots of things like
> `true` and `false` are just constants, most of the other funcs are
> copied from Scalar::Util, etc.

Are you trying to do this for CPAN release or by modifications to perl
itself (i.e., to core)?

perl-5.34 is out of maintenance, so I wouldn't expect anything to be
backported to it (except possible security fixes).

[Paul "LeoNerd" Evans]

On Thu, 27 Jul 2023 08:05:47 -0400
James E Keenan <jke...@pobox.com> wrote:

> Are you trying to do this for CPAN release or by modifications to
> perl itself (i.e., to core)?
>
> perl-5.34 is out of maintenance, so I wouldn't expect anything to be
> backported to it (except possible security fixes).

Ah, by "backport", I meant "I'm trying to make an XS module to upload
to CPAN", as some sort of vaguely dual-life mechanism to support this
on older perls.

[Paul "LeoNerd" Evans]

On Thu, 27 Jul 2023 12:40:12 +0100
"Paul \"LeoNerd\" Evans" <leo...@leonerd.org.uk> wrote:

> TL;DR: This is a long sequence of "I tried X and it didn't work", that
> ends in "Can anyone help?" request.

Update: I bypassed the entire problem by doing exactly this in the end:

> One thought I've had is that I could implement all of the `builtin::`
> funcs as little "pureperl" CVs that have an optree that contains just
> a single OP_CUSTOM to contain the code, rather than them being XSUBs.

It doesn't solve the problem for general user exports, but it does
sidestep it for the immediate goal of making the module work.

The module is now available on CPAN, and give or take a few caveats,
implements all of the `builtin` functions, as we know them on v5.38,
on every release of Perl back as far as v5.18:

https://metacpan.org/release/PEVANS/builtin-Backport-0.01

(Those caveats being:
* It can't provide is_bool or the created_as_* tests
* It can't lexically export an XSUB, as noted above)

For now I have purposely told it not to index the actual `builtin.pm`
file provided by in distribution, so as not to upset anything on the
CPAN indexes. That's a temporary situation that I'd eventually like to
resolve, once we work out some details about how we might handle
version numbers and general file/API updates in a more general way.

[Diab Jerius]

On 7/27/23 13:07, Paul "LeoNerd" Evans wrote:

The module is now available on CPAN, and give or take a few caveats,
implements all of the `builtin` functions, as we know them on v5.38,
on every release of Perl back as far as v5.18:

  https://metacpan.org/release/PEVANS/builtin-Backport-0.01

(Those caveats being:
  * It can't provide is_bool or the created_as_* tests
  * It can't lexically export an XSUB, as noted above)

For now I have purposely told it not to index the actual `builtin.pm`
file provided by in distribution, so as not to upset anything on the
CPAN indexes. That's a temporary situation that I'd eventually like to
resolve, once we work out some details about how we might handle
version numbers and general file/API updates in a more general way.

And because nature abhors a vacuum, CPAN already sports

    builtin::compat

and

    builtins::compat

Maybe work with one/both of the authors (HARRG,TOBYINK) to claim/deprecate their namespace?

Diab