How does the pynac spkg work?

[Julien Puydt]

Hi,

upon unpacking sage-5.0.rc1/spkg/standard/pynac-0.2.3.p1.spkg, one
gets :
spkg-install SPKG.txt src/ src.ah/

"diff -urN src/ src.ah/" shows differences only in
autom4te.cache/requests...

What is that src.ah for?

Snark on #sagemath

[Keshav Kini]

src.ah is not even mentioned in spkg-install. Furthermore it shouldn't
exist according to what paltry SPKG specifications we have. Can we
delete it?

-Keshav

----
Join us in #sagemath on irc.freenode.net !

[Jeroen Demeyer]

On 2012-05-13 18:51, Julien Puydt wrote:
> Hi,
>
> upon unpacking sage-5.0.rc1/spkg/standard/pynac-0.2.3.p1.spkg, one
> gets :
> spkg-install SPKG.txt src/ src.ah/

src.ah was mistakenly added by made. The fact that src and src.ah
contain their own .hg archive means that "hg status" doesn't show any
errors. I would recommend to delete the src/.hg directory to prevent
this sort of error in the future.

[Keshav Kini]

Jeroen Demeyer <jdem...@cage.ugent.be> writes:
> src.ah was mistakenly added by made. The fact that src and src.ah
> contain their own .hg archive means that "hg status" doesn't show any
> errors. I would recommend to delete the src/.hg directory to prevent
> this sort of error in the future.

Please don't modify the contents of src/ at all. Eventually we will want
to be able to directly unpack upstream tarballs from a known URL with
known hash into src/ and not even necessarily need to ship it with Sage
at all. Every src directory inside an SPKG which differs from the
upstream source tarball's contents is one more special case to worry
about when designing such a system.

Even more fundamentally, everything that we modify should be tracked by
version control. The fact that we do not put src/ under revision control
then directly implies that we should not touch it.

A better solution, IMO, is to make scripts check the actual contents of
the SPKG for extraneous directories and files. There should be no
directories other than src/ and patches/ ; the line "src/" should be in
.hgignore (which should exist), spkg-install should exist, SPKG.txt
should exist, etc. Anything unknown in the root directory of the SPKG
should generate a warning.

[Jeroen Demeyer]

On 2012-05-13 22:46, Keshav Kini wrote:
> Even more fundamentally, everything that we modify should be tracked by
> version control. The fact that we do not put src/ under revision control
> then directly implies that we should not touch it.

I disagee when it comes to removing parts of a spkg. Several packages
include only partial sources. They contain the upstream tree but with
some files/directories (which Sage doesn't need) removed. I think this
is fine and should be allowed.

In the case of Pynac, the upstream version history in "src/.hg" serves
no purpose for Sage, so I would remove it.


Jeroen.

[William Stein]

On Sun, May 13, 2012 at 5:13 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2012-05-13 22:46, Keshav Kini wrote:
>> Even more fundamentally, everything that we modify should be tracked by
>> version control. The fact that we do not put src/ under revision control
>> then directly implies that we should not touch it.
> I disagee when it comes to removing parts of a spkg.  Several packages
> include only partial sources.  They contain the upstream tree but with
> some files/directories (which Sage doesn't need) removed.  I think this
> is fine and should be allowed.

Indeed. Many spkg's are full of stuff we absolutely don't want to
ship. They have windows binaries in them, java binaries, big pdf's,
and other random stuff that wastes space and makes some people
nervous.

>
> In the case of Pynac, the upstream version history in "src/.hg" serves
> no purpose for Sage, so I would remove it.
>
>
> Jeroen.
>

> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org



--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[William Stein]

On Mon, May 14, 2012 at 12:31 AM, William Stein <wst...@gmail.com> wrote:
> On Sun, May 13, 2012 at 5:13 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> On 2012-05-13 22:46, Keshav Kini wrote:
>>> Even more fundamentally, everything that we modify should be tracked by
>>> version control. The fact that we do not put src/ under revision control
>>> then directly implies that we should not touch it.
>> I disagee when it comes to removing parts of a spkg.  Several packages
>> include only partial sources.  They contain the upstream tree but with
>> some files/directories (which Sage doesn't need) removed.  I think this
>> is fine and should be allowed.
>
> Indeed.  Many spkg's are full of stuff we absolutely don't want to

s/spkg/upstream sources (not spkg's!)

[Julien Puydt]

Le lundi 14 mai, Keshav Kini a écrit:

> A better solution, IMO, is to make scripts check the actual contents
> of the SPKG for extraneous directories and files. There should be no
> directories other than src/ and patches/ ; the line "src/" should be
> in .hgignore (which should exist), spkg-install should exist, SPKG.txt
> should exist, etc. Anything unknown in the root directory of the SPKG
> should generate a warning.

Debian has an utility called 'lintian' which is a 'Static analysis tool
for Debian packages', and which checks that the package satisfies a
precise set of properties. No package goes in if it makes lintian
unhappy.

Either sage would need such, or it should just put things right
(upstream tarballs in a directory -- building scripts in another
directory, under proper revision control).

Snark on #sagemath

[Julien Puydt]

Le dimanche 13 mai, Jeroen Demeyer a écrit:

I thought doing "make dist" gave clean sources, meaning it wouldn't
contain everything in the repository
(neither .hg, .git, .svn, .whatever, nor generated content, nor...).

How was that src/ dir obtained if it contains a .hg?

Snark on #sagemath

[Ivan Andrus]

When you build an spkg it already checks a few things. It shouldn't be hard to add other checks. Currently the checks are not "enforced" in the sense that they just print warnings, but don't change the return status or anything. This is in SAGE_ROOT/local/bin/sage-pkg if anyone wants to look at it.

For example we could exclude repository data in the src/ directory of all spkgs. Deciding whether or not this is desirable is the hard part. There could even be an environment variable (since we all love them) specifying in the spkg-install which other files to exclude.

-Ivan

[Jeroen Demeyer]

On 2012-05-14 07:49, Julien Puydt wrote:
> I thought doing "make dist" gave clean sources, meaning it wouldn't
> contain everything in the repository
> (neither .hg, .git, .svn, .whatever, nor generated content, nor...).

Indeed, this should be the case.

[Keshav Kini]

William Stein <wst...@gmail.com> writes:
> On Mon, May 14, 2012 at 12:31 AM, William Stein <wst...@gmail.com> wrote:
>> On Sun, May 13, 2012 at 5:13 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>>> I disagee when it comes to removing parts of a spkg.  Several packages
>>> include only partial sources.  They contain the upstream tree but with
>>> some files/directories (which Sage doesn't need) removed.  I think this
>>> is fine and should be allowed.
>>
>> Indeed.  Many spkg's are full of stuff we absolutely don't want to
>
> s/spkg/upstream sources (not spkg's!)
>
>> ship.  They have windows binaries in them, java binaries, big pdf's,
>> and other random stuff that wastes space and makes some people
>> nervous.

OK, that's reasonable. I withdraw my objection, though it would still be
nice if we had a better history of what is or was in the src/
directories of SPKGs, if it is no longer considered possible to
determine this simply from the package version minus the patch number.

[John H Palmieri]

On Monday, May 14, 2012 11:43:09 PM UTC-7, Keshav Kini wrote:

William Stein writes:
> On Mon, May 14, 2012 at 12:31 AM, William Stein wrote:

>> On Sun, May 13, 2012 at 5:13 PM, Jeroen Demeyer wrote:
>>> I disagee when it comes to removing parts of a spkg.  Several packages
>>> include only partial sources.  They contain the upstream tree but with
>>> some files/directories (which Sage doesn't need) removed.  I think this
>>> is fine and should be allowed.
>>
>> Indeed.  Many spkg's are full of stuff we absolutely don't want to
>
> s/spkg/upstream sources     (not spkg's!)
>
>> ship.  They have windows binaries in them, java binaries, big pdf's,
>> and other random stuff that wastes space and makes some people
>> nervous.

OK, that's reasonable. I withdraw my objection, though it would still be
nice if we had a better history of what is or was in the src/
directories of SPKGs, if it is no longer considered possible to
determine this simply from the package version minus the patch number.

Any changes like this are supposed to be documented in SPKG.txt, right? If not, the spkg needs work.

--
John

[kcrisman]

You are correct, and preferably there should be a script included
which does all this removal.

[Keshav Kini]

Sure, but a blurb in SPKG.txt is not really the same as having true
versioning of the directory. Of course, we do mostly have copies of all
old versions of all SPKGs (going back at least as far as /home/release
on sage.math contains versions of Sage). So we could reconstruct the
history of these src/ directories with some effort.

Note: I'm not saying we should put src/ under version control. I just
think it's best if we modify it only at install time via patches, rather
than at packaging time, but as Jeroen and William point out, sometimes
it's an issue of file size, or just not wanting to include binary
blobs...

[Jason Grout]

It would be great if we had an spkg-prepare script alongside the
spkg-install script. The spkg-prepare script would do any packaging
steps when we did sage -spkg. That way I could just unzip a src/ file
and do sage -spkg. That would delete the things we don't want from the
src/ file, etc. Then I'd have an spkg to test.

Or maybe it should be spkg-dist, or something.

Thanks,

Jason

[William Stein]

I like spkg-dist, since it *is* spkg-dist, at least in some packages I
made, such as the core sage library.

-- William

>
> Thanks,
>
> Jason

[Jason Grout]

On 5/15/12 7:47 PM, William Stein wrote:
> I like spkg-dist, since it*is* spkg-dist, at least in some packages I

> made, such as the core sage library.

spkg-dist is also what we call the spkg preparing script in the sage
notebook and sage cell server.

Jason

[Robert Bradshaw]

I don't think we should be storing the unpacked src/ directory in the
spkg at all, instead we should store a pointer to a tarball (which
could be external or right in the spkg itself. Upon install this
tarball would get unpacked into src/. The sage-spkg script would
unpack this tarball, compare it to src/ (if it exists) making sure
there are no differences (that are not accounted for by the existing
patches, or perhaps even creating a patch to describe the difference),
and then create the .spkg file excluding the src/ directory (but
possibly including the pristine, unpacked tarball).

A script that takes the pristine upstream tarball and creates a purged
version would also be useful.

- Robert

[William Stein]

I read the above 4 times and I don't understand what you're proposing,
except maybe (?) to make it impossible to build Sage if you're not
connected to the internet...? Or to basically change nothing?
Confused.

>
> - Robert

[Jeroen Demeyer]

On 2012-05-16 02:47, William Stein wrote:
> I like spkg-dist, since it *is* spkg-dist, at least in some packages I
> made, such as the core sage library.

spkg-dist actually does two things:
(1) it prepares the files inside the spkg
(2) it packs and compresses it into a .spkg file

In the PARI spkg, I made a script "spkg-make" (this name might not be
the best) which only does the first part: it creates the src/ directory
from an upstream git repository. I would encourage this for more packages.

To minimize confusion, I would prefer normal Sage packages to have a
script which only does (1) and which is not called sage-spkg. In PARI I
called this "spkg-make" but maybe now I would call it "spkg-src" or
"spkg-prepare" as somebody else proposed.

[William Stein]

I like "sage-src", since it is consistent with "sage-spkg", and so far
everybody like "sage-spkg".
It's consistent because "-src" creates the "src/" directory, just like
"-spkg" creates the ".spkg" file.

William

[Keshav Kini]

William Stein <wst...@gmail.com> writes:
> On Wed, May 16, 2012 at 3:34 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> On 2012-05-16 02:47, William Stein wrote:
>>> I like spkg-dist, since it *is* spkg-dist, at least in some packages I
>>> made, such as the core sage library.
>> spkg-dist actually does two things:
>> (1) it prepares the files inside the spkg
>> (2) it packs and compresses it into a .spkg file
>>
>> In the PARI spkg, I made a script "spkg-make" (this name might not be
>> the best) which only does the first part: it creates the src/ directory
>> from an upstream git repository.  I would encourage this for more packages.
>>
>> To minimize confusion, I would prefer normal Sage packages to have a
>> script which only does (1) and which is not called sage-spkg.  In PARI I
>> called this "spkg-make" but maybe now I would call it "spkg-src" or
>> "spkg-prepare" as somebody else proposed.
>
> I like "sage-src", since it is consistent with "sage-spkg", and so far
> everybody like "sage-spkg".
> It's consistent because "-src" creates the "src/" directory, just like
> "-spkg" creates the ".spkg" file.

FWIW I at first found it slightly difficult to remember whether to use
sage-pkg or sage-spkg.

[William Stein]

Why? Don't you think "sage-spkg" is more logical, since the file it
creates ends in ".spkg"?

>
> ----
> Join us in #sagemath on irc.freenode.net !
>

[Keshav Kini]

William Stein <wst...@gmail.com> writes:
> On Wed, May 16, 2012 at 2:35 AM, Robert Bradshaw
> <robe...@math.washington.edu> wrote:
>> I don't think we should be storing the unpacked src/ directory in the
>> spkg at all, instead we should store a pointer to a tarball (which
>> could be external or right in the spkg itself. Upon install this
>> tarball would get unpacked into src/. The sage-spkg script would
>> unpack this tarball, compare it to src/ (if it exists) making sure
>> there are no differences (that are not accounted for by the existing
>> patches, or perhaps even creating a patch to describe the difference),
>> and then create the .spkg file excluding the src/ directory (but
>> possibly including the pristine, unpacked tarball).
>
> I read the above 4 times and I don't understand what you're proposing,
> except maybe (?) to make it impossible to build Sage if you're not
> connected to the internet...? Or to basically change nothing?
> Confused.

This would make it possible to ship spkg-install scripts and SPKG.txt
files for all optional and experimental SPKGs at a low disk space cost
(because their src/ dirs would not be shipped). Integrating this stuff,
along with everything else, into a single repository, would allow for
easier development. This goes back to what we discussed at Review Days
2.

[William Stein]

This = what? Clearly you claim to understand what Robert is
proposing. I don't even know what "this" is.

>
> -Keshav
>
> ----
> Join us in #sagemath on irc.freenode.net !
>

[Keshav Kini]

William Stein <wst...@gmail.com> writes:
> On Wed, May 16, 2012 at 3:47 AM, Keshav Kini <kesha...@gmail.com> wrote:
>> William Stein <wst...@gmail.com> writes:
>>> I like "sage-src", since it is consistent with "sage-spkg", and so far
>>> everybody like "sage-spkg".
>>> It's consistent because "-src" creates the "src/" directory, just like
>>> "-spkg" creates the ".spkg" file.
>>
>> FWIW I at first found it slightly difficult to remember whether to use
>> sage-pkg or sage-spkg.
>>
>
> Why? Don't you think "sage-spkg" is more logical, since the file it
> creates ends in ".spkg"?

Hang on, now you're confusing me. (Apparently I still find it difficult
to remember the difference between sage-pkg and sage-spkg).

`sage-pkg` creates .spkg files. `sage-spkg` installs .spkg files. Am I
wrong? I seem to be correct, at least looking at the contents of
$SAGE_LOCAL/bin/sage-pkg and $SAGE_ROOT/spkg/bin/sage-spkg .

Now if we're talking about command line arguments, it seems like both
`sage -[-]spkg` and `sage -[-]pkg` are actually calling
`$SAGE_LOCAL/bin/sage-pkg`.

That's kind of confusing.

-Keshav

[Keshav Kini]

Currently:

The package named foo consists of one file called foo-x.y.z.pn.spkg. It
is a tarball containing some files and directories (such as spkg-install
and patches/), among which is a directory called src/. Stuff other than
src/ is under revision control in the root directory inside the tarball,
and consists of stuff written by Sage developers. src/, on the other
hand, is not under revision control, and consists of stuff written by
upstream developers.

If the package is a standard package, the .spkg file is shipped with
Sage. If it is optional or experimental, it is stored on the Sage
mirrors and can be downloaded and installed by users with `sage -i`.

What I am proposing (and I guess it's similar to what Robert was
saying):

The package named foo consists of some files and directories created by
Sage developers, and some files created by upstream developers. The
files and directories created by Sage developers are shipped as part of
Sage - not in a tarball, just in the file tree. The files created by
upstream developers sit in a tarball. That tarball can be shipped with
Sage, or not. The foo-related files created by Sage developers and
shipped in the Sage tree include a datum which gives a location of the
upstream tarball, either as a path in the Sage file tree, or as a URL
for an external downloadable file.

If the package is a standard package, the upstream tarball is shipped
with Sage. If it is optional or experimental, it is stored on the Sage
mirrors. Either way, the spkg-install, patches/, etc. created by Sage
developers are shipped with Sage, and know where to find the
corresponding upstream tarball, whether locally or remotely.


(Hopefully that was easier and not harder to understand than what Robert
said, but I'm not convinced it is...)

[William Stein]

On Wed, May 16, 2012 at 4:00 AM, Keshav Kini <kesha...@gmail.com> wrote:
> William Stein <wst...@gmail.com> writes:
>> On Wed, May 16, 2012 at 3:47 AM, Keshav Kini <kesha...@gmail.com> wrote:
>>> William Stein <wst...@gmail.com> writes:
>>>> I like "sage-src", since it is consistent with "sage-spkg", and so far
>>>> everybody like "sage-spkg".
>>>> It's consistent because "-src" creates the "src/" directory, just like
>>>> "-spkg" creates the ".spkg" file.
>>>
>>> FWIW I at first found it slightly difficult to remember whether to use
>>> sage-pkg or sage-spkg.
>>>
>>
>> Why? Don't you think "sage-spkg" is more logical, since the file it
>> creates ends in ".spkg"?
>
> Hang on, now you're confusing me. (Apparently I still find it difficult
> to remember the difference between sage-pkg and sage-spkg).
>
> `sage-pkg` creates .spkg files. `sage-spkg` installs .spkg files. Am I
> wrong? I seem to be correct, at least looking at the contents of
> $SAGE_LOCAL/bin/sage-pkg and $SAGE_ROOT/spkg/bin/sage-spkg .
>
> Now if we're talking about command line arguments, it seems like both
> `sage -[-]spkg` and `sage -[-]pkg` are actually calling
> `$SAGE_LOCAL/bin/sage-pkg`.
>
> That's kind of confusing.

It is indeed. I was in fact confused. Above, I was talking about
the sage-dist script that is *inside* some spkg's.

>
> -Keshav
>
> ----
> Join us in #sagemath on irc.freenode.net !
>

[Jason Grout]

Your proposal sounds reasonable to me. In fact, it sounds really nice.
So if I do sage -i some-spkg, it would really download the unmodified
(or possibly cleaned-up) upstream source, then use the spkg info that is
distributed with Sage to extract, modify, and build the upstream source.
That's a much more traditional process, like BSD ports, OSX Homebrew, etc.

Jason

[Keshav Kini]

Yup, exactly right. It also fits in well with using Gentoo Prefix, if we
decide to go that way, but is useful even if we don't. Especially nice
is that it brings hacking on package install scripts into the same
development model as hacking on $SAGE_LOCAL/bin or the Sage library,
making it easier to consolidate Sage development into a single
repository, a goal which I know Robert shares (which is why I presume to
speak for him when it comes to this SPKG upstream/downstream
partitioning idea).

> the unmodified (or possibly cleaned-up) upstream source

Indeed, if we went with purely unmodified upstream source (as I was
insisting on earlier in the thread before Jeroen and William changed my
mind), the upstream tarballs wouldn't even need to be stored on the Sage
mirrors - we could just wget them directly from upstream. And we could
still do that, for those packages which don't need their tarballs
cleaned up, only patched.

[William Stein]

I'm confused: when (not if!) the website of sage or one of its components goes down, then nobody can build Sage?   How is this not a recipe for misery?

> -Keshav
>
> ----
> Join us in #sagemath on irc.freenode.net !
>

[Keshav Kini]

Please read again... standard packages will be 100% included within the
Sage distribution tarball.

[William Stein]

Sorry, I read it in a bumpy cab. You wrote "If the package is a
standard package, the upstream tarball is shipped with Sage". By
this do you really mean "If the package is a standard package, then a
modified version of the upstream tarball is shipped with Sage"? I'm
against shipping the exact upstream tarballs with Sage, as explained
above (e.g., what if they contain opaque Windows binaries?).

-- William

>
> -Keshav
>
> ----
> Join us in #sagemath on irc.freenode.net !
>

[Julien Puydt]

Le mercredi 16 mai, William Stein a écrit:

> I'm against shipping the exact upstream tarballs with Sage, as
> explained above (e.g., what if they contain opaque Windows binaries?).

My impression is that the current situations occur (sorted in very fast
decreasing frequency) :
(1) upstream tarball is ready to ship (tarballs obtained from 'make
dist' with autotools generally belong to that category) ;
(2) upstream tarball is in fact obtained by taking a snapshot from a
repository (there something more might be needed, like running some
kind of autogen.sh, removing .hg/.svn/.git/.whatever) ;
(3) upstream tarball is a mess for various reasons.

So there is definitely a need for some script to prepare the shipped
tarball. Let me call it spkg-get-upstream-tarball for the rest.

For (1), that script could just be a one-liner wget (or is that a too
big dependency?).

For (2), that script would be a few lines to checkout and remove the
cruft. Perhaps sage could even have a generic spkg-get-upstream-from-vs
script which would be called something like :
spkg-get-upstream-from-vs --git git://url/to/upstream/foo
that would take care of doing the checkout and removing the obvious
directories ; that way each spkg-get-upstream-tarball would just call
that generic script, then do additional actions.

For (3), that script would do everything by hand.

How many (3) are there anyway?

Snark on #sagemath

[Keshav Kini]

Yes, here by "upstream tarball" I just meant "the tarball containing the
portion of the package which is not written by us, i.e. containing the
source code we need from upstream", not "the tarball provided by the
upstream developers". Sorry, that was a bit confusing.

Of course, in some cases our tarball of upstream stuff may be exactly
the same as the official upstream tarball of their software. In that
case we could either mirror it on the Sage mirrors, or just point our
install scripts to its official download location. If we were modifying
the official upstream tarball, though - say by removing binary blobs or
whatever - then we would of course need to host this tarball on the Sage
mirrors.

[Keshav Kini]

Julien Puydt <julien...@laposte.net> writes:
> Le mercredi 16 mai, William Stein a écrit:
>> I'm against shipping the exact upstream tarballs with Sage, as
>> explained above (e.g., what if they contain opaque Windows binaries?).
>
> My impression is that the current situations occur (sorted in very fast
> decreasing frequency) :
> (1) upstream tarball is ready to ship (tarballs obtained from 'make
> dist' with autotools generally belong to that category) ;
> (2) upstream tarball is in fact obtained by taking a snapshot from a
> repository (there something more might be needed, like running some
> kind of autogen.sh, removing .hg/.svn/.git/.whatever) ;
> (3) upstream tarball is a mess for various reasons.
>
> So there is definitely a need for some script to prepare the shipped
> tarball. Let me call it spkg-get-upstream-tarball for the rest.
>
> For (1), that script could just be a one-liner wget (or is that a too
> big dependency?).

Python has the ability to download files with the urllib2 module. We
don't need wget for this.

> For (2), that script would be a few lines to checkout and remove the
> cruft. Perhaps sage could even have a generic spkg-get-upstream-from-vs
> script which would be called something like :
> spkg-get-upstream-from-vs --git git://url/to/upstream/foo
> that would take care of doing the checkout and removing the obvious
> directories ; that way each spkg-get-upstream-tarball would just call
> that generic script, then do additional actions.

IMO it's not really necessary to have a script which automates this. We
don't upgrade versions of stuff very often, and I think this can easily
be done manually.

> For (3), that script would do everything by hand.

It doesn't make sense to me to have a script that would "do everything
by hand". Why not just... do it by hand?

> How many (3) are there anyway?

Good question.

[Julien Puydt]

Le mercredi 16 mai, Keshav Kini a écrit:

> Julien Puydt <julien...@laposte.net> writes:
> > Le mercredi 16 mai, William Stein a écrit:
> >> I'm against shipping the exact upstream tarballs with Sage, as
> >> explained above (e.g., what if they contain opaque Windows
> >> binaries?).
> >
> > My impression is that the current situations occur (sorted in very
> > fast decreasing frequency) :
> > (1) upstream tarball is ready to ship (tarballs obtained from 'make
> > dist' with autotools generally belong to that category) ;
> > (2) upstream tarball is in fact obtained by taking a snapshot from a
> > repository (there something more might be needed, like running some
> > kind of autogen.sh, removing .hg/.svn/.git/.whatever) ;
> > (3) upstream tarball is a mess for various reasons.
> >
> > So there is definitely a need for some script to prepare the shipped
> > tarball. Let me call it spkg-get-upstream-tarball for the rest.
> >
> > For (1), that script could just be a one-liner wget (or is that a
> > too big dependency?).
>
> Python has the ability to download files with the urllib2 module. We
> don't need wget for this.

Good.

> > For (2), that script would be a few lines to checkout and remove the
> > cruft. Perhaps sage could even have a generic
> > spkg-get-upstream-from-vs script which would be called something
> > like : spkg-get-upstream-from-vs --git git://url/to/upstream/foo
> > that would take care of doing the checkout and removing the obvious
> > directories ; that way each spkg-get-upstream-tarball would just
> > call that generic script, then do additional actions.
>
> IMO it's not really necessary to have a script which automates this.
> We don't upgrade versions of stuff very often, and I think this can
> easily be done manually.

Ok.

> > For (3), that script would do everything by hand.
>
> It doesn't make sense to me to have a script that would "do everything
> by hand". Why not just... do it by hand?

Because the goal is to have something to document what is done by
hand, and the best way to document it is to have the script of
those actions :-)

> > How many (3) are there anyway?
>
> Good question.

Yes, because the main point of the discussion is "We can't do that
because if...".

Snark on #sagemath

[Nils Bruin]

On May 16, 1:08 am, Keshav Kini <keshav.k...@gmail.com> wrote:
[...]

> What I am proposing (and I guess it's similar to what Robert was
> saying):
>
> The package named foo consists of some files and directories created by
> Sage developers, and some files created by upstream developers. The
> files and directories created by Sage developers are shipped as part of
> Sage - not in a tarball, just in the file tree. The files created by
> upstream developers sit in a tarball. That tarball can be shipped with
> Sage, or not. The foo-related files created by Sage developers and
> shipped in the Sage tree include a datum which gives a location of the
> upstream tarball, either as a path in the Sage file tree, or as a URL
> for an external downloadable file.

How would that approach work when the "upstream" needs both cruft
removal and patching? From your description I understand that the
script doing the patching would live somewhere in the sage tree. Where
would we document (either in the form of instruction or preferably in
the form of an automated script that takes a truly upstream tarball/
repository/whatever and extracts from that the tarball we're
interested in) the procedure to produce the tarball? Does this live
(as it does now) as a spkg-make script (etc) and/or as part of a
README inside of the "tarball"? Or do we put TWO procedures in the
sage tree:

1] procedure to produce a tarball from sources truly outside of
sage's control

2] procedure to take the content of the tarball and use that to build
the appropriate libraries, executable, whatever the functionality is
that the spkg provides.
The idea of the split in procedures is that 2] is supposed to be
executed whenever the component is "installed", whereas 1] is only
supposed to be performed for upgrading the component and the result of
it is shipped with sage (for standard) or stored elsewhere (for
optional)

For some operations it seems a bit arbitrary where they would fit. Why
should only cruft removal go into 1] and no patching? Is that just a
judgement call, that step 1] should be kept minimal while still
producing something that is acceptable to ship? If so, is there a
clear reason why step 1] should be minimal?

It's clear that anything that might be architecture- or system-
dependent should be run in 2] (such as running configure, make, and
install), but I don't think we ever patch dependent on architecture.
In that case, you'd write a patch that can be applied unconditionally
and contains the appropriate architecture tests . So for functional
reasons, it seems patching could happen in step 1] too.

Of course this is all just package management, so by all means, do
whatever works best/is most convenient. However, in my experience,
procedures tend to work better if there is some simple conceptual
reasoning behind them.

[Jeroen Demeyer]

On 2012-05-16 21:18, Nils Bruin wrote:
> I don't think we ever patch dependent on architecture.
> In that case, you'd write a patch that can be applied unconditionally
> and contains the appropriate architecture tests .

It's not very relevant to the discussion at hand, but we *do* sometimes
conditionally patch according to architecture. I think MPFR and
Singular are two examples.

[Robert Bradshaw]

Sorry, I was really tired when I wrote that (and a lot of state was
just in my head). Keshav summarized things nicely. The only part I
would add is that if we set it up this way, we could use the (unpacked
and ignored by sage-spkg) src/ directory, if present, to automatically
verify or generate the patch files.

I don't like spkg.txt because it's just a (under-specified) convention
of what to put there when you make manual changes. Better would be to
have some specification file (json?) like

{ upstream = "http://original.project/releaseX.tar",
upstream_sha1 = "...",
excise = [ "windows.exe", "binary.blob" ], # remove from the above
when creating our tarball (below)
sources = "project.X.tar",
sources_sha1 = "...",
patches = [ "fixOne.patch", "fix2.patch" ], # to be applied to
sources, once unpacked
}

Then an spkg could be automatically generated by changing the upstream
url (most common case), and otherwise one could tweak/meddle with src
and automatically create a new patch once you get it working.

This may be a pain to do with shell scripts (how do you parse a json
file?) but perhaps a lot could be cleaned up if we assumed *some*
python as a prerequisite.

- Robert

[Robert Bradshaw]

Is this by necessity, or could the changes be #ifdef guarded and always applied?

- Robert

[Julien Puydt]

Le mercredi 16 mai, Robert Bradshaw a écrit:

> I don't like spkg.txt because it's just a (under-specified) convention
> of what to put there when you make manual changes. Better would be to
> have some specification file (json?) like
>
> { upstream = "http://original.project/releaseX.tar",
> upstream_sha1 = "...",
> excise = [ "windows.exe", "binary.blob" ], # remove from the above
> when creating our tarball (below)
> sources = "project.X.tar",
> sources_sha1 = "...",
> patches = [ "fixOne.patch", "fix2.patch" ], # to be applied to
> sources, once unpacked
> }
>
> Then an spkg could be automatically generated by changing the upstream
> url (most common case), and otherwise one could tweak/meddle with src
> and automatically create a new patch once you get it working.
>
> This may be a pain to do with shell scripts (how do you parse a json
> file?) but perhaps a lot could be cleaned up if we assumed *some*
> python as a prerequisite.

May I suggest that instead of writing JSON then trying to parse it...
just write a shell script (see below for the type of shell script I
have in mind)?

Instead of the "upstream*=" things, just have two lines of script to
download and check the hash.

Instead of and "excise=", just have the needed 'rm -rf' lines.

I don't see what the "sources*=" are about if you already downloaded
and checked the tarball.

What use if there for a "patches=..." list if you have the patches in a
patches/ directory already?

That could be done by a Makefile file with mandatory rules (which
could then be called by a script) with things like :

.PHONY: unpack patch configure build install

patch-stamp:
@echo 'Applying patches'
<patch commands>
@touch patch-stamp

patch: unpack patch-stamp

unpack-stamp:
@echo 'Unpacking sources'
<unpack commands>
@touch unpack-stamp

configure-stamp:
@echo 'Configuring sources'
<configure commands>
@touch configure-stamp

configure: configure-stamp

build: unpack configure build-stamp

You get the idea,

Snark on #sagemath

[Robert Bradshaw]

On Wed, May 16, 2012 at 11:34 PM, Julien Puydt <julien...@laposte.net> wrote:
> Le mercredi 16 mai, Robert Bradshaw a écrit:
>> I don't like spkg.txt because it's just a (under-specified) convention
>> of what to put there when you make manual changes. Better would be to
>> have some specification file (json?) like
>>
>> { upstream = "http://original.project/releaseX.tar",
>>   upstream_sha1 = "...",
>>   excise = [ "windows.exe", "binary.blob" ],  # remove from the above
>> when creating our tarball (below)
>>   sources = "project.X.tar",
>>   sources_sha1 = "...",
>>   patches = [ "fixOne.patch", "fix2.patch" ], # to be applied to
>> sources, once unpacked
>> }
>>
>> Then an spkg could be automatically generated by changing the upstream
>> url (most common case), and otherwise one could tweak/meddle with src
>> and automatically create a new patch once you get it working.
>>
>> This may be a pain to do with shell scripts (how do you parse a json
>> file?) but perhaps a lot could be cleaned up if we assumed *some*
>> python as a prerequisite.
>
> May I suggest that instead of writing JSON then trying to parse it...
> just write a shell script (see below for the type of shell script I
> have in mind)?

Parsing JSON is easy in a proper language with the right libraries,
and easier to manipulate and re-write programmatically (e.g. when
automatically creating the patch files). I could see storing other
metadata (including dependencies) in that file as well that one would
want to reason with. The makefile idea has merit though.

> Instead of the "upstream*=" things, just have two lines of script to
> download and check the hash.

+1

> Instead of and "excise=", just have the needed 'rm -rf' lines.

+1

> I don't see what the "sources*=" are about if you already downloaded
> and checked the tarball.

There's 2 tarballs here, the actual upstream (that we don't typically
ship with Sage) and the purged one (that we do).

> What use if there for a "patches=..." list if you have the patches in a
> patches/ directory already?

They may not commute. I suppose they could be numbered/otherwise
ordered based on their names.

> That could be done by a Makefile file with mandatory rules (which
> could then be called by a script) with things like :
>
> .PHONY: unpack patch configure build install
>
> patch-stamp:
>        @echo 'Applying patches'
>        <patch commands>
>        @touch patch-stamp
>
> patch: unpack patch-stamp
>
> unpack-stamp:
>        @echo 'Unpacking sources'
>        <unpack commands>
>        @touch unpack-stamp
>
> configure-stamp:
>        @echo 'Configuring sources'
>        <configure commands>
>        @touch configure-stamp
>
> configure: configure-stamp
>
> build: unpack configure build-stamp
>
> You get the idea,
>
> Snark on #sagemath
>

[Julien Puydt]

Le jeudi 17 mai, Robert Bradshaw a écrit:

In debian packages, there is :
- a 'control' file which declares what the package is about, including
dependencies ;
- a 'rules' file, which is a Makefile, which can look like what I shown
(it's generally simpler since it can rely on things debian provides)

> There's 2 tarballs here, the actual upstream (that we don't typically
> ship with Sage) and the purged one (that we do).

Well, you could have a 'download' target to download the upstream and a
'sagify' target which would turn it into the other tarball (the one
which would actually be shipped or placed on sage mirrors depending on
mandatory vs optional status).

> > What use if there for a "patches=..." list if you have the patches
> > in a patches/ directory already?
>
> They may not commute. I suppose they could be numbered/otherwise
> ordered based on their names.

In debian, the patches get in a patches/ directory, named
01-foo.patch, 02-bar.patch, etc. And there is a patches/series
file which contains an ordered list of the patches (I haven't figured
out what it's used for since there is already an explicit numbering).

It would definitely help if more sage developers knew how things are
done in debian, fedora or gentoo -- the mail Harald Schilly sent
yesterday and the citation therein are spot on there.

Snark on #sagemath

[Keshav Kini]

Julien Puydt <julien...@laposte.net> writes:
> It would definitely help if more sage developers knew how things are
> done in debian, fedora or gentoo -- the mail Harald Schilly sent
> yesterday and the citation therein are spot on there.

Knowing how things are done in Gentoo is probably the most useful of
these since Gentoo Prefix is actually something we could use as our
packaging system (as has been prototyped by Burcin's lmonade project),
rather than just learning lessons in package management from it.

[Julien Puydt]

Le jeudi 17 mai, Keshav Kini a écrit:

> Julien Puydt <julien...@laposte.net> writes:
> > It would definitely help if more sage developers knew how things are
> > done in debian, fedora or gentoo -- the mail Harald Schilly sent
> > yesterday and the citation therein are spot on there.
>
> Knowing how things are done in Gentoo is probably the most useful of
> these since Gentoo Prefix is actually something we could use as our
> packaging system (as has been prototyped by Burcin's lmonade project),
> rather than just learning lessons in package management from it.

Notice that rebasing sage to use gentoo prefix/lmonade still won't fix
the main issue with sage : that it's not properly available in
distributions but needs to be compiled (duplicating a good chunk of
what's already there in the process...).

There really should be sage-the-software (distribution agnostic) and
sage-the-distribution (builds on everything with minimal dependencies).

Snark on #sagemath

[Keshav Kini]

Julien Puydt <julien...@laposte.net> writes:
> Le jeudi 17 mai, Keshav Kini a écrit:
>> Julien Puydt <julien...@laposte.net> writes:
>> > It would definitely help if more sage developers knew how things are
>> > done in debian, fedora or gentoo -- the mail Harald Schilly sent
>> > yesterday and the citation therein are spot on there.
>>
>> Knowing how things are done in Gentoo is probably the most useful of
>> these since Gentoo Prefix is actually something we could use as our
>> packaging system (as has been prototyped by Burcin's lmonade project),
>> rather than just learning lessons in package management from it.
>
> Notice that rebasing sage to use gentoo prefix/lmonade still won't fix
> the main issue with sage : that it's not properly available in
> distributions but needs to be compiled (duplicating a good chunk of
> what's already there in the process...).

Yes, I agree. But moving to a Prefix build system will be a lot easier
than making Sage packagable by distros.

[Julien Puydt]

Easier, perhaps, but that looks like trading Charybdis for Scylla!

Snark on #sagemath

[Keshav Kini]

Why? Doing one hardly makes it more difficult to do the other, so it's
not like choosing between alternatives.

[Robert Bradshaw]

On Thu, May 17, 2012 at 3:18 AM, Keshav Kini <kesha...@gmail.com> wrote:
> Julien Puydt <julien...@laposte.net> writes:
>> It would definitely help if more sage developers knew how things are
>> done in debian, fedora or gentoo -- the mail Harald Schilly sent
>> yesterday and the citation therein are spot on there.
>
> Knowing how things are done in Gentoo is probably the most useful of
> these since Gentoo Prefix is actually something we could use as our
> packaging system (as has been prototyped by Burcin's lmonade project),
> rather than just learning lessons in package management from it.

+1, for a long term solution.

In the short term I'm thinking of doing something package by package
that still works with the current "distribution" mechanisms.

- Robert