spkg size and regenerating configure scripts

[Jean-Pierre Flori]

Dear all,

While working on #9167 and #13325, I was faced with the necessity of modifying upstream build systems based on autotools.
I thought the right way to do so was to modify the autotools files and regenerate the build system with the autotools machinery.
This has the unfortunate consequence of modifying huge parts of files like configure and make the hg history and spkg size explode.

If upstream agrees with the fixes and reacts quickly, and if the Sage spkg is not too much behind so that updating the upstream source does not imply another huge amount of work, then it's not a problem: let's just package the updated upstream sources which include the fix.

If one of the above condition is not met, then, either we proceed as I did, which is the quickest way, and I feel the cleanest, or we directly patch the configure file in as few places as possible, but that's surely a lot more work most of the time, and only include this smaller patch.

Any thought about that?

Best,
JP

[Julien Puydt]

Le 07/08/2012 19:10, Jean-Pierre Flori a �crit :

Why would hg follow upstream? Shouldn't it only follow the changes to
the supporting files?

Snark on #sagemath

[Jean-Pierre Flori]

Why would hg follow upstream? Shouldn't it only follow the changes to
the supporting files?

Snark on #sagemath

Indeed, hg does not follow upstream, but in order to keep a vanilla src directory, I produce patches, put them in SPKG_ROOT/patches and there they got tracked.

[Jeroen Demeyer]

The best solution (but unfortunately, non-trivial) solution is to
regenerate configure and others using the *same version* of
autoconf/automake/libtool. Then the patch files will be small.

[Jean-Pierre Flori]

Yes thats more work as well...

[Volker Braun]

We are shipping > 300mb of compressed sources, I don't think we should be concerned with the size of autotool-generated scripts. In fact, I would argue its better to use newer autotools than some ancient version just because you want to save some disk space.

What would be nice is some convention where to store the autogenerated files and a script that copies them out of / back into the source directory. 

[leif]

On 7 Aug., 20:01, Jeroen Demeyer <jdeme...@cage.ugent.be> wrote:
> The best solution (but unfortunately, non-trivial) solution is to
> regenerate configure and others using the *same version* of
> autoconf/automake/libtool.

Yep, I usually try that as well.

> Then the patch files will be small.

Unfortunately that's not always true; even little changes to e.g.
configure.ac can create huge diffs to the original configure script.


And in some cases, e.g. for GLPK 4.44 [1], newer autotools are needed
to fix the problem (perhaps without actually changing any autoconf or
automake source files).


-leif

[1] http://trac.sagemath.org/sage_trac/ticket/12703

[leif]

On 8 Aug., 02:45, Volker Braun <vbraun.n...@gmail.com> wrote:
> We are shipping > 300mb of compressed sources, I don't think we should be
> concerned with the size of autotool-generated scripts.

Before 'patch' was a requirement (i.e., included into Sage), for at
least some spkgs the copies of a patched 'configure' made up a fair
amount of the spkg size. But even if we only ship and apply patches,
those to autotools-generated files frequently still get huge.

Keeping changes "automatically made" (to generated files) in our
Mercurial history -- forever! -- doesn't make sense to me. The
respective files in patches/ often vanish with the next upstream
upgrade.

And IMHO the ratio of the size of vanilla source to Sage's Mercurial
repo for an spkg matters, not the overall growth to the Sage source
tarball.

> In fact, I would
> argue its better to use newer autotools than some ancient version just
> because you want to save some disk space.

Not just because; autoreconf'ing with newer versions can cause other
trouble, requiring further changes to upstream's sources.


Ceterum censeo we should support and use 'xz' instead of or in
addition to 'bzip2'.


-leif

[Volker Braun]

On Wednesday, August 8, 2012 10:31:52 AM UTC-4, leif wrote:

Before 'patch' was a requirement (i.e., included into Sage), for at 

least some spkgs the copies of a patched 'configure' made up a fair 

amount of the spkg size.

Well back then the by far biggest spkg was a couple of binary builds of fortran compilers for osx weighting in at 30mb. As long as the compressed configure scripts stay less than a MB or so you are barking up the wrong tree. Even if thats a sizeable fraction of one of the smaller spkgs.

But I agree that autogenerated files shouldn't be checked into the repository. Thats why I said that it would be nice to have some script support to copy the autotool-generated files in/out of the source dir.

Not just because; autoreconf'ing with newer versions can cause other 

trouble, requiring further changes to upstream's sources. 

Autotools aren't exactly a fast-moving target. In fact I haven't yet encountered a case where an autoreconf required me to update the sources. You do get useful new features, though; E.g. in cddlib I regenerated everything with an autotools version that knows about libtool.

[Jean-Pierre Flori]

Interesting comment here:
http://trac.sagemath.org/sage_trac/ticket/748#comment:19
Other such examples in Sage?

And another point, not sure if every project does that, but I would personally not track files generated by autotools in my git/hg/svn/cvs tree, only produce them when distributing a release.
So it might be quite strange that we do track them here in Sage.
Although we definitely do not want to regenerate anything at build time as well...

[Jean-Pierre Flori]

On Thursday, August 9, 2012 10:59:00 AM UTC+2, Jean-Pierre Flori wrote:

Interesting comment here:
http://trac.sagemath.org/sage_trac/ticket/748#comment:19
Other such examples in Sage?

See also http://trac.sagemath.org/sage_trac/ticket/12830

[Julien Puydt]

Le 09/08/2012 10:59, Jean-Pierre Flori a �crit :

> Although we definitely do not want to regenerate anything at build time
> as well...

Let me quote from /usr/share/doc/autotools-dev/README.Debian.gz which is
cited in
http://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#bpp-autotools
:
(from the preamble)
<<<
Note: libtool is part of the autotools toolchain, but we don't cover
libtool well in this document. However, there are a few big fat
warnings that I will repeat here:

Your package must use an up-to-date libtool. See the libtool package
for more information. Always retool (rebuild all autotools-generated
files) on package build time when using libtool.
>>>


(from the "Introduction" section)
<<<
Autoconf's more advanced architecture-aware code (invoked when the
AC_CANONICAL family of macros are used, which is always the case if
libtool is being used, for example) also needs two extra helper scripts:
config.sub and config.guess.

These two helpers, config.guess and config.sub, need to be constantly
updated to address new architectures. Unfortunately, Debian's automake
package has failed to provide such timely updates regularly, and even if
it did, most developers are not even aware of the need for such updates.
Their upstream often isn't, either.

The result ends up as serious bugs filled in the BTS by porters. Given
the amount of packages using autoconf and automake in Debian, we are
talking about a rather large number of bugs every time we start supporting
a new architecture.
>>>

I propose to go the debian way instead of the opposite direction.

Snark on #sagemath

[Jean-Pierre Flori]

On Thursday, August 9, 2012 11:42:40 AM UTC+2, Snark wrote:

Le 09/08/2012 10:59, Jean-Pierre Flori a �crit :

 
The slight difference is that Debian surely does not care about requiring autotools, people here might feel differently.
And running autotools also takes a small but non negligible amout of time, for what the same remark apply.

[Jean-Pierre Flori]

And we must also be aware that even if autotools is installed, we cannot really expet anything about the version and its flavour depending on the system, which is also easier to deal for Debian I guess (even though their automake is often outdated).

[Julien Puydt]

Le 09/08/2012 13:04, Jean-Pierre Flori a �crit :

> The slight difference is that Debian surely does not care about
> requiring autotools, people here might feel differently.

How many of the spkg use autotools?

How many of the spkg have an spkg-install which could be reduced to
calling a common spkg-install-autotools?

> And running autotools also takes a small but non negligible amout of
> time, for what the same remark apply.

Well, given the choice of either not running autotools thereby failing
sooner or running autotools and succeeding later, I'll choose the last
option -- and that is what the quoted text is about.

> And we must also be aware that even if autotools is installed, we cannot
> really expet anything about the version and its flavour depending on the
> system, which is also easier to deal for Debian I guess (even though
> their automake is often outdated).

Well, sage has patch and bzip2 which are very widely available in
non-problematic versions ; why wouldn't it have autotools(+libtool) if
it's those are important?

Snark on #sagemath

[Jeroen Demeyer]

autotools is absolutely something to be used at packaging-time, not
build-time.

To solve most of the problems in this thread, I am preparing an
experimental autotools spkg containing the latest m4 and many versions
of autoconf, automake and libtool.

[Volker Braun]

Debian and libtool is a bit of a special case. Since Debian packaging wants no rpaths set for system libraries, they require libtool-1.5.2 or higher. So whenever upstream used an old autotools version, they have to regenerate it to get a new libtool. I don't think we should go the Debian way except for eventually making use of rpaths in a sane way, of course.

[Jean-Pierre Flori]

I'm a little bit confused.

You first say it's a bad idea to use autotools at build-time (which I a'gree with, but that's not relevant).
And then that you are crafting an autotools spkg, which as I understand implies that you suggest to use it at build-time.

Maybe only as temporary solution then, while waiting for upstream packaging, and to avoid killing the hg history?
In this scenario, what about spkg for which upstream basically does not exist anymore ?
Should we run autotools in the spkg as long as the spkg exist?
Or in this specific case, which hopefully will be rare, should we do something else?
Like directly patching the src dir, regenerating the autotools machinery, and providing a separate "already-applied-patches-directory" where the patches to the .am and .ac files only would reside and be tracked (and maybe some indication of what autotools version were used for the regeneration)?

[Jeroen Demeyer]

On 2012-08-09 16:51, Jean-Pierre Flori wrote:
> I'm a little bit confused.
>
> You first say it's a bad idea to use autotools at build-time (which I
> a'gree with, but that's not relevant).
> And then that you are crafting an autotools spkg, which as I understand
> implies that you suggest to use it at build-time.

No, I want this to be used at packaging-time.
Autotools should be used by people *creating* the package.

[Jean-Pierre Flori]

Ok so what's the point of having an autotools package?
To enforce people using it?
I must admit I rarely use anything provided by Sage itself when tweaking packages (except for testing...).
I'd better use my own system-wide mercurial and autotools, but I guess that could change.

And that does not answer the question to know if we should track the changes to the autogenerated files or not :)

[Jeroen Demeyer]

On 2012-08-09 17:44, Jean-Pierre Flori wrote:
> Ok so what's the point of having an autotools package?

It's meant for people needing to patch configure.ac (and hence also
configure) files. It would allow you to re-generate configure from
configure.ac using the *same* version of autoconf, such that the patch
to configure will (hopefully) be small.

[leif]

It would probably make sense to ship newest versions of [m4 and]
libtool, config.sub and config.guess and use these for all spkgs using
autotools.

An optional developer spkg with [probably a couple of different
versions of] autotools doesn't have to (or shouldn't) contain the
former.


W.r.t. the "moving target" (vbraun), my impression is that the
situation today is much better than years ago, where different
versions of automake and autoconf were very likely to be incompatible
to each other.


-leif

[Jeroen Demeyer]

On 2012-08-09 19:22, leif wrote:
> It would probably make sense to ship newest versions of [m4 and]

I agree with m4 but for different reasons: MPIR needs it and PPL needs
it to build its C (not C++) interface.

I don't really see the need of shipping other autotools.

[leif]

On 9 Aug., 19:24, Jeroen Demeyer <jdeme...@cage.ugent.be> wrote:
> On 2012-08-09 19:22, leif wrote:> It would probably make sense to ship newest versions of [m4 and]
>
> I agree with m4 but for different reasons: MPIR needs it and PPL needs
> it to build its C (not C++) interface.

That's not a different reason... ;-) (m4 is a prerequisite for
building Sage.)

config.sub and config.guess are [also] frequently outdated in many
[s]pkgs, so it may make sense to use a single, up-to-date instance of
both.

> I don't really see the need of shipping other autotools.

Me neither, at least not as standard spkgs / for users. Your
(optional) autotools spkg is meant as a convenience for *developers*
if I understand correctly, but neither m4 nor libtool belong to
autotools.


-leif

[Jeroen Demeyer]

On 2012-08-09 19:45, leif wrote:
> if I understand correctly, but neither m4 nor libtool belong to
> autotools.

I think libtool is generally considered to be part of autotools,
i.e. autotools = autoconf+automake+libtool.

Where:
autoconf also contains autom4te, autoreconf, autoheader,...
automake also contains aclocal, config.guess, install-sh,...