No . in @INC breaks CPAN

[Todd Rinaldo]

Howdy,

I've done a write up of a recent change to blead perl. In the future it will no longer be possible to count on . being in @INC. This will break many of the existing CPAN installs.

It was suggested I send the RFC here:

http://blogs.perl.org/users/todd_rinaldo/2016/11/how-removing-from-inc-is-about-to-break-cpan.html

In Perl 5.26, it will no longer be a safe assumption to assume . is in @INC. This is a good move towards a more secure Perl, but will break the installation of many CPAN modules. For those of you wondering why this was done, see this post for more information.

Many CPAN modules try to do things like: use inc::Module::Install; This depends on . being in @INC. If you invoke Makefile.PL without it, the script will not even run.

We have come up with several ways to mitigate and ultimately fix the problem:

Short Term

Perl 5.26 will support an environment variable "PERL_USE_UNSAFE_INC=1". If you set this, any perl script invoked will include . at the end of @INC. Tentatively, support for this environment variable will be immediately deprecated since long term, the CPAN modules need to simply take this into account.

My current thinking is that if we inject this environment variable into the CPAN clients (cpanm, cpan, cpanplus, etc.), then they will build and install modules as they normally did. Packagers (RPM, deb, etc.) will also need to amend their build systems by also setting this environment variable. It is also possible that a fix will be require to CPAN smoke testers.

Long Term

We need to fix the CPAN modules themselves. The vast majority of modules can be fixed by putting use lib "."; at the top of Makefile.PL or Build.PL. Others will need to fix additional scripts in their tree to also take this into account. In some cases use blib; might be more appropriate, for instance, in unit tests which invoke child scripts.

My original proposal included a patch to ExtUtils::MakeMaker, Module::Build, and Module::Build::Tiny. At this point I think that this might be the wrong place to fix the problem.

These are all my thoughts based on conversations with others about how this transition needs to be addressed. I do not claim to have all the correct answers. I am using this blog post to anchor the conversations that need to be had with multiple groups (CPAN Testers, P5P, Tool Chain Gang, etc.).

If you want to try a Perl with this functionality now, you will need to build Perl with the Configure option: -Ddefault_inc_excludes_dot. This is available in blead and the next development build of perl.

I'll be opening a ticket with the CPAN clients this week. If you know of other items that need consideration, please let me know in comments or feel free to reach out to me or perl 5 porters.

-- Todd

[James E Keenan]

On 11/14/2016 09:11 AM, Todd Rinaldo wrote:
> Howdy,
>
> I've done a write up of a recent change to blead perl. In the future
it will no longer be possible to count on . being in @INC. This will
break many of the existing CPAN installs.
>
> It was suggested I send the RFC here:
>
>
http://blogs.perl.org/users/todd_rinaldo/2016/11/how-removing-from-inc-is-about-to-break-cpan.html
>
> In Perl 5.26, it will no longer be a safe assumption to assume . is
in @INC. This is a good move towards a more secure Perl, but will break
the installation of many CPAN modules. For those of you wondering why
this was done, see this post for more information.
>
> Many CPAN modules try to do things like: use inc::Module::Install;
This depends on . being in @INC. If you invoke Makefile.PL without it,
the script will not even run.
>

Here is a use case which I found very soon after building a perl at
blead with the new option and then installing App::cpanminus against
that perl.

#####
Summary of my perl5 (revision 5 version 25 subversion 7) configuration:
Commit id: dd4f16e4ff71ea6d0422d277f00fe430e1d93938
[snip]
config_args='-des -Dusedevel -Uversiononly
-Dprefix=/home/jkeenan/testing/nodot -Dman1dir=none -Dman3dir=none
-Ddefault_inc_excludes_dot'
[snip]
@INC:
lib
/home/jkeenan/testing/nodot/lib/perl5/site_perl/5.25.7/x86_64-linux
/home/jkeenan/testing/nodot/lib/perl5/site_perl/5.25.7
/home/jkeenan/testing/nodot/lib/perl5/5.25.7/x86_64-linux
/home/jkeenan/testing/nodot/lib/perl5/5.25.7
#####

I used ./bin/cpanm to install one of my CPAN distros that has a
second-level dependency on another one of those distros,
Perl-StringIdentifier.

#####
Building and testing String-PerlIdentifier-0.05 ... FAIL
! Installing String::PerlIdentifier failed. See
/home/jkeenan/.cpanm/work/1479138905.14485/build.log for details. Retry
with --force to force install it.
#####

Let's inspect that build log.

#####
PERL_DL_NONLAZY=1 "/home/jkeenan/testing/nodot/bin/perl"
"-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef
*Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
Can't locate t/eligible_chars in @INC (@INC contains: t/
/home/jkeenan/.cpanm/work/1479138905.14485/String-PerlIdentifier-0.05/blib/lib
/home/jkeenan/.cpanm/work/1479138905.14485/String-PerlIdentifier-0.05/blib/arch
/home/jkeenan/testing/nodot/lib/perl5/site_perl/5.25.7/x86_64-linux
/home/jkeenan/testing/nodot/lib/perl5/site_perl/5.25.7
/home/jkeenan/testing/nodot/lib/perl5/5.25.7/x86_64-linux
/home/jkeenan/testing/nodot/lib/perl5/5.25.7) at t/Auxiliary.pm line 14.
Compilation failed in require at t/01_basic.t line 8.
BEGIN failed--compilation aborted at t/01_basic.t line 8.
# Looks like your test exited with 2 just after 1.
t/01_basic.t ......
Dubious, test returned 2 (wstat 512, 0x200)
Failed 40/41 subtests
#####

Here's what that distro has in/under its 't/' directory.

#####
$ ls t |cat
01_basic.t
02_specified.t
03_min.t
04_max.t
05_default.t
06_no_scores.t
Auxiliary.pm
eligible_chars
#####

#####
$ head t/01_basic.t
# t/01_basic.t - four basic tests
use Test::More tests => 41;
use strict;
use warnings;

BEGIN { use_ok( 'String::PerlIdentifier' ); }
use lib ("t/");
use Auxiliary qw{ _first_and_subsequent };

four_basic_tests() for (1..10);
#####
$ head -15 t/Auxiliary.pm
package Auxiliary;
use strict;
use warnings;
our ($VERSION, @ISA, @EXPORT_OK);
$VERSION = '0.05';
require Exporter;
@ISA = qw(Exporter);
@EXPORT_OK = qw(
_first_and_subsequent
);
*ok = *Test::More::ok;
use lib ('.');

our (%eligibles, %chars);
require "t/eligible_chars";
#####

The test file attempts to load Auxiliary.pm, which is located in the
same directory as the test file. Auxiliary.pm in turn 'require's a
plain-text file, 'eligible_chars', which is also located in t/. That
'require' fails.

In this case, adding '.' to the distribution's Makefile.PL made no
difference. I had to add "use lib ('.');" to Auxiliary.pm to enable it
to locate 'eligible_chars', after which 'make test' PASSed.

Based on this example and several other failures, my hunch is that many
of the failures which we'll see on CPAN are failures of *tests* rather
than failures of the modules themselves.

Thank you very much.
Jim Keenan

[Aristotle Pagaltzis]

* Todd Rinaldo <to...@cpanel.net> [2016-11-14 15:12]:

> Long Term
>
> We need to fix the CPAN modules themselves.

I’m afraid that’s a pipe dream. You can fix the most popular part of
CPAN but not even close to everything. There are still distributions
containing broken Module::Install versions, years after the last
bugfixes. Any solution that sets out from “we fix everything on CPAN”
as a starting point is no solution at all.

So the reality is this: if we make a change that makes most of CPAN non-
installable then a quite significant fraction of it will be left behind
and will never work again.

My first thought for managing this ran in the direction of introducing
a new flag in META for dists to opt in to installation under dot-free
@INC, and then only dists that explicitly ask for it get it.

The downside is that this will withhold the improvement from dists that
already don’t depend on dot in @INC. It will take time till everything
that can be installed under a dot-free @INC actually is.

The upside is, dists which do depend on dot in @INC and are not actively
maintained will continue being installable, indefinitely.

In the longer term, a new version of the META spec could declare the new
flag to be implicitly turned on, so the META in every new dist does not
have to contain this boilerplate bit forever (much as saying `use 5.022`
replaces the need to `use feature qw/mile-long litany to the present/`).

This is a brain dump, mind you; I am not at all confident that there are
no unintended consequences.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

[James E Keenan]

On 11/14/2016 07:39 PM, Aristotle Pagaltzis wrote:
> * Todd Rinaldo <to...@cpanel.net> [2016-11-14 15:12]:
>> Long Term
>>
>> We need to fix the CPAN modules themselves.
>
> I’m afraid that’s a pipe dream. You can fix the most popular part of
> CPAN but not even close to everything. There are still distributions
> containing broken Module::Install versions, years after the last
> bugfixes. Any solution that sets out from “we fix everything on CPAN”
> as a starting point is no solution at all.
>
> So the reality is this: if we make a change that makes most of CPAN non-
> installable then a quite significant fraction of it will be left behind
> and will never work again.
>

Before we polarize ourselves into the camps of "we have to fix all of
CPAN" and "it's hopeless to try to fix CPAN", it's important to realize
that we now have the conceptual tools with which to assess the scope of
the problem.

We now know that we can visualize CPAN as a river of dependencies. If
we can identify a critical part of the "upstream", we can set up a
CPANtesters-like apparatus to see how much damage (flooding?)
default_inc_excludes_dot will actually cause. Discussion of courses of
action will then be empirically informed.

This change is, after all, just a much larger version of the "blead
breaks CPAN" problem we've been handling for years.

[Aristotle Pagaltzis]

* James E Keenan <jk...@verizon.net> [2016-11-15 14:12]:

> Before we polarize ourselves into the camps of "we have to fix all of
> CPAN" and "it's hopeless to try to fix CPAN", it's important to
> realize that we now have the conceptual tools with which to assess the
> scope of the problem.

Speaking of empirical facts: broken Module::Install versions are still
bundled now. Why is it polarising to point to factual evidence of what
we can and cannot expect to happen?

> We now know that we can visualize CPAN as a river of dependencies.

That covers the CPAN – under the best of cases. I would hope that Perl
has ambitions to serve actual users rather than just people publishing
libraries for it.

> If we can identify a critical part of the "upstream", we can set up
> a CPANtesters-like apparatus to see how much damage (flooding?)
> default_inc_excludes_dot will actually cause.

Sounds good in theory. It *would* be great if we had that. But are you
building it or at least planning to – or at the very least working to
convince someone specific to do so? Because experience says otherwise
it’s not getting built.

And when it comes to results rather than aspirations, it makes no sense
to consider solutions that can be counted on to happen on a par with
ones that cannot.

I’m not arguing against what you said. Those are great things to want.
I am only arguing against it being an argument against what I said. The
things you propose are far less likely than the ones I am interested in,
but ideally of course we would have both.

Personally I wish to prioritise the likely because we cannot afford to
end up with no solution. Considering blue-sky solutions on a par with
ones that are within reach is not very helpful to that end, even if the
blue-sky stuff is necessary to the long term.

> Discussion of courses of action will then be empirically informed.

Broken Module::Install bundles (and the track record of services not
getting built merely by calling for them) are empirically confirmed.
What bar does an empirical fact have to clear before it attains the
power to affect the decision-making process?

> This change is, after all, just a much larger version of the "blead
> breaks CPAN" problem we've been handling for years.

But not well. Something is better than nothing here, but I think we feel
too great about doing a bit and punting on anything more difficult just
because most other communities barely even try addressing this at all.

With this particular change we’re in a different quantitative breakage
league than usually.