Updated CPAN River available

[James E Keenan]

An updated printout of the CPAN River is available here:

https://gist.github.com/jkeenan/a7670b2ffe1ffe3e8d8cbdbc2b3dd721

This is an update of the data presented by David Golden in 2016 here:

https://gist.github.com/xdg/af7a32c5b21d45a6b255

I have chosen to present this in .csv format to make it more easily
consumable. YMMV.

Thanks to David Golden for assistance and encouragement.

Sincerely,
Jim Keenan

[James E Keenan]

On 04/09/2017 08:54 PM, James E Keenan wrote:

> On 04/09/2017 11:33 AM, James E Keenan wrote:
>> An updated printout of the CPAN River is available here:
>>
>> https://gist.github.com/jkeenan/a7670b2ffe1ffe3e8d8cbdbc2b3dd721
>>
>

Today I began a fifth round of testing. As in the previous round of
testing, I am doing so to reflect (a) the generation of a new CPAN
river file, 'cpan.river.20170409.csv'; and (b) an understanding of why
'PERL_USE_UNSAFE_INC' must be set to '0' before using the 'cpanm'
build against Perl 5 blead. But I also hope to detect the effect of
CPAN distributions that have been updated to accommodate
no-dot-by-default-in-@INC.

1. The new input file is, as mentioned above,
'cpan.river.20170409.csv'.

2. The first program continues to be 'csv-get-upriver-distros.pl'. I
will fetch the first 2000, rather than just the first 1000,
distributions from the input file.

#####
$ cd ~/learn/perl/cpan-river
$ perl csv-get-upriver-distros.pl 2000 > fifth-round-top-2000.txt
#####

This program excludes distributions which are shipped with the Perl 5
core as well as certain previously identified as hard-to-test
distributions like mod_perl, Term::ReadLine::Perl, X11-Protocol.

3. I used the previously built perl5 blead for testing:

#####
$ ./bin/perl -v | head -2 | tail -1
This is perl 5, version 26, subversion 0 (v5.26.0
(v5.25.11-50-gb57dd50)) built for x86_64-linux
#####

... and again used Miyagawa's up-to-the-minute cpanm against that
perl.

#####
$ ./bin/perl -MApp::cpanminus -E 'say $App::cpanminus::VERSION;'
1.7043
#####

4. Adapting nudge from KENTNL, I worked from a just-updated local
minicpan repository.

#####
$ minicpan
$ cd ~/testing/blead
$ export PERL_USE_UNSAFE_INC=0
$ date > startcpanm
$ cat ~/learn/perl/cpan-river/fifth-round-top-2000.txt | \
xargs bin/cpanm --mirror ~/minicpan --verbose; date > endcpanm
#####

5. Copied and renamed the 'cpanm' build.log to
20170412-2000-build.log.gz.

#####
$ cd ~/.cpanm/latest-build
$ gzip -v -c build.log > ~/learn/perl/cpan-river/20170412-2000-build.log.gz
#####

6. grepped out relevant lines from the build log:

#####
$ cd ~/learn/perl/cpan-river
$ zgrep FAIL 20170412-2000-build.log.gz | grep -v 'Result: FAIL' | \
sed -e 's/^-> //' > 20170412-2000-fails.txt
#####

7. Used a revised version of 'order-battle.pl' to record the order in
which various modules *first* appeared in the 'fails' file and the
total number of times each module was cited. Results are attached as
'order-of-battle-20170412.txt'.

#####
$ perl order-battle.pl 20170412-2000-fails.txt 1>
order-of-battle-20170412.txt
#####

What is the "order of battle"? It's an approximation of the order in
which, as of today, we need to get new CPAN releases out so that
down-river distros are no longer failing due to failures in their
upstream dependencies.

For me the most interesting aspect of this fifth round is that 29
distributions which appeared in order-of-battle-20170409.txt (the
previous round focusing on no-dot) no longer appear in
order-of-battle-20170412.txt. That is, their no-dot problems -- and,
perhaps more importantly, the no-dot problems of their prerequisites
-- have been resolved and the distros are now installable under
PERL_USE_UNSAFE_INC=0. Thanks to Rafael Garcia-Suarez for uploading a
new version of Sub::Identify which was #7 on the earlier list. This
upload cleared its reverse dependencies to be fully installable as
well.

Thank you very much.
Jim Keenan

[Ricardo Signes]

* James E Keenan <jke...@pobox.com> [2017-04-12T22:15:22]

> For me the most interesting aspect of this fifth round is that 29
> distributions which appeared in order-of-battle-20170409.txt (the
> previous round focusing on no-dot) no longer appear in
> order-of-battle-20170412.txt. That is, their no-dot problems -- and,
> perhaps more importantly, the no-dot problems of their prerequisites
> -- have been resolved and the distros are now installable under
> PERL_USE_UNSAFE_INC=0.

Sub::Exporter::ForMethods appeared on that list, but as far as I know, it
doesn't need a new release, unless it's to require a higher version of prereqs.
That won't actually be a problem, though: what's needed is for a fresh install
of v5.26.0 to find a newer version of the prereq. The downstream module
doesn't actually need to know about it or require it.

I must admit I haven't been following this closely, but it seems to me like the
most useful list is:

* which modules are, on their own, broken
* ...possibly with a count of how many downstream things they break

In this case, Mac::SystemDirectory is a huge culprit, breaking File::HomeDir
installs on macOS, which breaks a holy ton of downstream things. I don't think
it appeared on your list.

--
rjbs

[James E Keenan]

On 04/16/2017 09:46 PM, Ricardo Signes wrote:
> * James E Keenan <jke...@pobox.com> [2017-04-12T22:15:22]
>> For me the most interesting aspect of this fifth round is that 29
>> distributions which appeared in order-of-battle-20170409.txt (the
>> previous round focusing on no-dot) no longer appear in
>> order-of-battle-20170412.txt. That is, their no-dot problems -- and,
>> perhaps more importantly, the no-dot problems of their prerequisites
>> -- have been resolved and the distros are now installable under
>> PERL_USE_UNSAFE_INC=0.
>
> Sub::Exporter::ForMethods appeared on that list, but as far as I know, it
> doesn't need a new release, unless it's to require a higher version of prereqs.
> That won't actually be a problem, though: what's needed is for a fresh install
> of v5.26.0 to find a newer version of the prereq. The downstream module
> doesn't actually need to know about it or require it.
>
> I must admit I haven't been following this closely, but it seems to me like the
> most useful list is:
>
> * which modules are, on their own, broken
> * ...possibly with a count of how many downstream things they break
>

It has become clear that there is more than one way to do an analysis of
CPAN dependencies at the approach of perl-5.26.0.

You could say: Let's rely on the fact that the major CPAN installers
have been patched to export a true value for PERL_USE_UNSAFE_INC into
their environments when they configure, build and test a CPAN
distribution. In this case an error other than "no-dot" in a distro's
prerequisites will prevent one's perfectly sound distro from being
installed, but a "no-dot" error in one's prereqs will not prevent your
distro from being installed.

Or you could say: At some point those CPAN installers will no longer
"cheat"; they will not export a true value for PERL_USE_UNSAFE_INC into
their environments. "no-dot" errors in any distro will be mercilessly
exposed. A "no-dot" error in a distro at the top of a tree of
dependencies will prevent all those dependencies from being
automatically installed.

I've taken each of these approaches -- "lax" and "strict" -- at
different points over the past month. What I took in the "fifth round"
is the second approach. Module-Runtime is probably the ultimate cause
of Sub::Exporter::ForMethods' failure to inst all in the "strict" approach.

Or you could mix and match these two approaches. I believe that Ryan
Voots has, in effect, done this, by first testing a distro under
"strict" and then, if it fails, under "lax".

> In this case, Mac::SystemDirectory is a huge culprit, breaking File::HomeDir
> installs on macOS, which breaks a holy ton of downstream things. I don't think
> it appeared on your list.
>

I'm testing mostly on Linux, a little on FreeBSD. So I won't be able to
pick up Mac- or Win32-specific problems.

Let me know if there's anything else I can provide.