tools and work flows for developing collections of distributions

[David Christensen]

module-authors:

I use h2xs, ExtUtils::MakeMaker, and make(1) on Unix-like platforms for
developing modules and collecting them together into a distribution.


I now have several distributions and would like to collect them together
into a unit (Bundle?), so that I can develop and test them together --
e.g. ensure that changing one distribution does not break another
distribution.


Are there Perl tools and/or work flows to support this?


TIA,

David

p.s. Where can I find documentation that explains Bundles and how to
create them?

[David Cantrell]

On Tue, Nov 20, 2018 at 05:25:47PM -0800, David Christensen wrote:

> p.s. Where can I find documentation that explains Bundles and how to
> create them?

https://metacpan.org/pod/CPAN#Bundles

However, they're a bit magical and require special support in the CPAN
client. These days people tend to just create Task-whatever
distributions, which contain a bit of documentation in Task::whatever
for the benefit of metacpan users, and the usual Makefile.PL to declare
dependencies.

--
David Cantrell | semi-evolved ape-thing

Are you feeling bored? depressed? slowed down? Evil Scientists may
be manipulating the speed of light in your vicinity. Buy our patented
instructional video to find out how, and maybe YOU can stop THEM

[Shlomi Fish]

Hi David,

On Tue, 20 Nov 2018 17:25:47 -0800
David Christensen <dpch...@holgerdanske.com> wrote:

> module-authors:
>
> I use h2xs, ExtUtils::MakeMaker, and make(1) on Unix-like platforms for
> developing modules and collecting them together into a distribution.
>

Please see:

* https://perl-begin.org/topics/cpan/

* https://metacpan.org/pod/Dist::Zilla::Plugin::TaskWeaver

>
> I now have several distributions and would like to collect them together
> into a unit (Bundle?), so that I can develop and test them together --
> e.g. ensure that changing one distribution does not break another
> distribution.
>
>
> Are there Perl tools and/or work flows to support this?
>
>
> TIA,
>
> David
>
> p.s. Where can I find documentation that explains Bundles and how to
> create them?

--
-----------------------------------------------------------------
Shlomi Fish http://www.shlomifish.org/
Escape from GNU Autohell - http://www.shlomifish.org/open-source/anti/autohell/

Scary thought of the day: The Princess Bride - the 3-D remake.

Please reply to list if it's a mailing list post - http://shlom.in/reply .

[David Christensen]

On 11/21/18 2:15 AM, David Cantrell wrote:
> On Tue, Nov 20, 2018 at 05:25:47PM -0800, David Christensen wrote:
>
>> p.s. Where can I find documentation that explains Bundles and how to
>> create them?
>
> https://metacpan.org/pod/CPAN#Bundles
>
> However, they're a bit magical and require special support in the CPAN
> client.

Thank you for replying. :-)


Below, please see an experiment of putting distribution source trees
inside of a bundle source tree, and leveraging the recursive features of
ExtUtils::MakeMaker (EUMM) and make(1):

1. 'perl Makefile.PL' creates a Makefile, etc., for the enclosed
distribution and then the bundle -- good.

EUMM::WriteMakefile() appears to traverse directories in
alphabetical order. Therefore, I can control the traversal order by
prefacing distribution directory names with numbers. This will ensure
that root cause failures appear before dependent failures.

2. 'make' appears to build the bundle and then the distribution -- I
would prefer that the distributions be built before the bundle.

RTFM make(1), I don't see an obvious way to change Make's recursive
traversal order to depth-first (?).

STFW, one work-around is to use Make's --directory/-C option:


https://stackoverflow.com/questions/31159082/is-there-a-way-to-make-a-recursive-make-j-invocation-go-depth-first

Perhaps there is a deeper answer?

https://www.gnu.org/software/make/manual/make.html#Recursion

3. 'make test' appears to run the bundle tests and then the
distribution tests. Again, I would prefer depth-first.

4. 'make dist' creates a tarball for the bundle, but not for the
distributions -- the bundle Makefile must be blocking recursion for the
'dist' target (?). Can I re-enable recursion?

> These days people tend to just create Task-whatever
> distributions, which contain a bit of documentation in Task::whatever
> for the benefit of metacpan users, and the usual Makefile.PL to declare
> dependencies.

Where can I find documentation that explains Tasks and how to create them?


I assume you are referring to the PREREQ_PM argument to WriteMakefile().
As I understand it, this argument assumes that the modules are already
installed So for development, I would need to preface the Perl include
path with the 'lib' directories for each of my distribution source trees
(?).


David


2018-11-21 14:36:15 dpchrist@tinkywinky ~/sandbox/perl
$ cat /etc/debian_version
9.6

2018-11-21 14:36:26 dpchrist@tinkywinky ~/sandbox/perl
$ uname -a
Linux tinkywinky 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27)
x86_64 GNU/Linux

2018-11-21 14:37:34 dpchrist@tinkywinky ~/sandbox/perl
$ perl -v | grep version
This is perl 5, version 24, subversion 1 (v5.24.1) built for
x86_64-linux-gnu-thread-multi

2018-11-21 14:38:32 dpchrist@tinkywinky ~/sandbox/perl
$ h2xs --help 2>&1 | grep version:
version: 1.23

2018-11-21 14:39:12 dpchrist@tinkywinky ~/sandbox/perl
$ perl -MExtUtils::MakeMaker -e 'print $ExtUtils::MakeMaker::VERSION, "\n"'
7.1002

2018-11-21 14:39:04 dpchrist@tinkywinky ~/sandbox/perl
$ make --version | grep 'GNU Make'
GNU Make 4.1

2018-11-21 13:49:06 dpchrist@tinkywinky ~/sandbox/perl
$ cat make-bundle.sh
#!/bin/bash
# $Id: make-bundle.sh,v 1.3 2018/11/21 21:48:36 dpchrist Exp $

set -o xtrace
set -o errexit

tmp=`pwd`/$0.tmp
dists='DistA'
bundle='Bundle::MyBundle'
bdir=$tmp/Bundle-MyBundle
bmod=$tmp/Bundle-MyBundle/lib/Bundle/MyBundle.pm

rm -rf $tmp
mkdir -p $tmp

cd $tmp && h2xs -XAn $bundle 2>/dev/null

for d in $dists; do
cd $tmp && h2xs -XAn $d 2>/dev/null
done

for d in $dists; do
mv $tmp/$d $bdir/.
done

perl -pi -e 's/=cut\n//' $bmod

echo '=head1 CONTENTS' >> $bmod
echo '' >> $bmod
for d in $dists; do
echo " $d" >> $bmod
done
echo '' >> $bmod
echo '=cut' >> $bmod

cd $bdir && perl Makefile.PL

cd $bdir && make

cd $bdir && make test

cd $bdir && make dist

2018-11-21 13:49:07 dpchrist@tinkywinky ~/sandbox/perl
$ bash make-bundle.sh
+ set -o errexit
++ pwd
+ tmp=/home/dpchrist/sandbox/perl/make-bundle.sh.tmp
+ dists=DistA
+ bundle=Bundle::MyBundle
+ bdir=/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle
+
bmod=/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/lib/Bundle/MyBundle.pm
+ rm -rf /home/dpchrist/sandbox/perl/make-bundle.sh.tmp
+ mkdir -p /home/dpchrist/sandbox/perl/make-bundle.sh.tmp
+ cd /home/dpchrist/sandbox/perl/make-bundle.sh.tmp
+ h2xs -XAn Bundle::MyBundle
+ for d in $dists
+ cd /home/dpchrist/sandbox/perl/make-bundle.sh.tmp
+ h2xs -XAn DistA
+ for d in $dists
+ mv /home/dpchrist/sandbox/perl/make-bundle.sh.tmp/DistA
/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/.
+ perl -pi -e 's/=cut\n//'
/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/lib/Bundle/MyBundle.pm
+ echo '=head1 CONTENTS'
+ echo ''
+ for d in $dists
+ echo ' DistA'
+ echo ''
+ echo =cut
+ cd /home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle
+ perl Makefile.PL
Checking if your kit is complete...
Looks good
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for DistA
Writing MYMETA.yml and MYMETA.json
Generating a Unix-style Makefile
Writing Makefile for Bundle::MyBundle
Writing MYMETA.yml and MYMETA.json
+ cd /home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle
+ make
cp lib/Bundle/MyBundle.pm blib/lib/Bundle/MyBundle.pm
make[1]: Entering directory
'/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/DistA'
cp lib/DistA.pm ../blib/lib/DistA.pm
Manifying 1 pod document
make[1]: Leaving directory
'/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/DistA'
Manifying 1 pod document
+ cd /home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle
+ make test
make[1]: Entering directory
'/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/DistA'
Manifying 1 pod document
make[1]: Leaving directory
'/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/DistA'
PERL_DL_NONLAZY=1 PERL_USE_UNSAFE_INC=1 "/usr/bin/perl"
"-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef
*Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/Bundle-MyBundle.t .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.06 cusr
0.00 csys = 0.09 CPU)
Result: PASS
make[1]: Entering directory
'/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/DistA'
PERL_DL_NONLAZY=1 PERL_USE_UNSAFE_INC=1 "/usr/bin/perl"
"-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef
*Test::Harness::Switches; test_harness(0, '../blib/lib',
'../blib/arch')" t/*.t
t/DistA.t .. ok
All tests successful.
Files=1, Tests=1, 1 wallclock secs ( 0.03 usr 0.01 sys + 0.05 cusr
0.02 csys = 0.11 CPU)
Result: PASS
make[1]: Leaving directory
'/home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle/DistA'
+ cd /home/dpchrist/sandbox/perl/make-bundle.sh.tmp/Bundle-MyBundle
+ make dist
rm -rf Bundle-MyBundle-0.01
"/usr/bin/perl" "-MExtUtils::Manifest=manicopy,maniread" \
-e "manicopy(maniread(),'Bundle-MyBundle-0.01', 'best');"
mkdir Bundle-MyBundle-0.01
mkdir Bundle-MyBundle-0.01/lib
mkdir Bundle-MyBundle-0.01/lib/Bundle
mkdir Bundle-MyBundle-0.01/t
Generating META.yml
Generating META.json
tar cvf Bundle-MyBundle-0.01.tar Bundle-MyBundle-0.01
Bundle-MyBundle-0.01/
Bundle-MyBundle-0.01/Changes
Bundle-MyBundle-0.01/lib/
Bundle-MyBundle-0.01/lib/Bundle/
Bundle-MyBundle-0.01/lib/Bundle/MyBundle.pm
Bundle-MyBundle-0.01/t/
Bundle-MyBundle-0.01/t/Bundle-MyBundle.t
Bundle-MyBundle-0.01/README
Bundle-MyBundle-0.01/Makefile.PL
Bundle-MyBundle-0.01/MANIFEST
Bundle-MyBundle-0.01/META.yml
Bundle-MyBundle-0.01/META.json
rm -rf Bundle-MyBundle-0.01
gzip --best Bundle-MyBundle-0.01.tar
Created Bundle-MyBundle-0.01.tar.gz

2018-11-21 14:36:01 dpchrist@tinkywinky ~/sandbox/perl
$ tail -n 6 make-bundle.sh.tmp/Bundle-MyBundle/lib/Bundle/MyBundle.pm

=head1 CONTENTS

DistA

=cut

[David Christensen]

On 11/21/18 9:20 AM, Shlomi Fish wrote:
> Hi David,

Hi, Shlomi! :-)

> On Tue, 20 Nov 2018 17:25:47 -0800
> David Christensen <dpch...@holgerdanske.com> wrote:
>

>> module-authors:
>>
>> I use h2xs, ExtUtils::MakeMaker, and make(1) on Unix-like platforms for
>> developing modules and collecting them together into a distribution.
>>
>

> Please see:
>
> * https://perl-begin.org/topics/cpan/

These seem to be the most relevant links:

* RFC: How to Release Modules on CPAN in 2011

https://www.perlmonks.org/?node_id=879515

* CPAN For Private Code

https://www.slideshare.net/thaljef/cpan-for-private-code


Perhaps my question could be rephrased as "I am looking for development
and build systems for Bundles and Tasks".

> * https://metacpan.org/pod/Dist::Zilla::Plugin::TaskWeaver

Searching for your distributions on MetaCPAN:

https://metacpan.org/search?size=20&q=author%3Ashlomif

Produces 137 results. My needs are nowhere near that scale.


Looking through:

https://github.com/shlomif/Task-BeLike-SHLOMIF

I see:


https://github.com/shlomif/Task-BeLike-SHLOMIF/blob/master/Task-BeLike-SHLOMIF/weaver.ini


I sense a non-trivial learning curve.


I am looking for the most lightweight tools and processes I can find for
developing collections of distributions on a small scale. The closer
they are to h2xs, ExtUtils::MakeMaker, and make(1), the better.


David

[David Cantrell]

On Wed, Nov 21, 2018 at 02:50:26PM -0800, David Christensen wrote:
> On 11/21/18 2:15 AM, David Cantrell wrote:
> >These days people tend to just create Task-whatever
> >distributions, which contain a bit of documentation in Task::whatever
> >for the benefit of metacpan users, and the usual Makefile.PL to declare
> >dependencies.
> Where can I find documentation that explains Tasks and how to create them?

The same place you'd find documentation on creating any other completely
normal distribution.

> I assume you are referring to the PREREQ_PM argument to WriteMakefile().
> As I understand it, this argument assumes that the modules are already

> installed.

A CPAN client will spot the dependencies and install them.

> So for development, I would need to preface the Perl include
> path with the 'lib' directories for each of my distribution source trees
> (?).

If you want to test against stuff that isn't on the CPAN, yes. If you
have all your distributions' sources as sub-dirs of a single directory
you can do something like ...

cd $wherever
export PERL5LIB=`perl -MCwd -e 'print join(":", map { cwd."/$_/lib" } @ARGV)' *`:$PERL5LIB

but note that that won't work for stuff that has XS components, or where
Makefile.PL does funky stuff to edit your code, or where your source
code isn't laid out in the normal hierarchy. Also on at least some
platforms there is a limit on how long PERL5LIB can be.

--
David Cantrell | Enforcer, South London Linguistic Massive

Planckton: n, the smallest possible living thing

[David Christensen]

Okay.


David