New guidelines for spkgs

[Jean-Pierre Flori]

Dear all,

I'm a little confused by the way spkg should now be updated.
The official doc is now here:
* http://www.sagemath.org/doc/developer/packaging.html

[Jean-Pierre Flori]

Ooops, end of message follows...

My questions are mainly about :
* package-version.txt, should it only be updated when the tarball changes? this is what http://www.sagemath.org/doc/developer/packaging.html#package-versioning suggests, not what the discussion between Jeroen and I at http://trac.sagemath.org/ticket/15123#comment:32 did.
* history in SPKG.txt, http://www.sagemath.org/doc/developer/packaging.html#the-spkg-txt-file says no history anymore, so should we just get rid of the old history when updating? as remarked http://trac.sagemath.org/ticket/15123#comment:24 it is still available in the hg/git history (and I'd better have no info kept in SPKG.txt than just old info).
* spkg-src, http://www.sagemath.org/doc/developer/packaging.html#modified-tarballs states its optional, I would say it shouldn't be, in fact I would even be in favor of having for every standard package even when it just downloads a tarball.
* in the example files for spkg-install/check, there's not the stuff checking we're in a shell with Sage stuff defined, is that on purpose?
* SAGE_FAT_BINARY, maybe it should be mentioned on this page as well.
* --libdir, same same

Best,
JP

[John Cremona]

Here is how I have interprested these questions for the packages I am
involved in maintining:

On 23 January 2014 09:51, Jean-Pierre Flori <jpf...@gmail.com> wrote:
> Ooops, end of message follows...
>
>
> On Thursday, January 23, 2014 10:43:42 AM UTC+1, Jean-Pierre Flori wrote:
>>
>> Dear all,
>>
>> I'm a little confused by the way spkg should now be updated.
>> The official doc is now here:
>> * http://www.sagemath.org/doc/developer/packaging.html
>
>
> My questions are mainly about :
> * package-version.txt, should it only be updated when the tarball changes?
> this is what
> http://www.sagemath.org/doc/developer/packaging.html#package-versioning
> suggests, not what the discussion between Jeroen and I at
> http://trac.sagemath.org/ticket/15123#comment:32 did.

Yes. It's a label for the upstream version. Local changes to what we
do with the tarball are tracked by git anyway.

> * history in SPKG.txt,
> http://www.sagemath.org/doc/developer/packaging.html#the-spkg-txt-file says
> no history anymore, so should we just get rid of the old history when
> updating? as remarked http://trac.sagemath.org/ticket/15123#comment:24 it is
> still available in the hg/git history (and I'd better have no info kept in
> SPKG.txt than just old info).

Correct: the new SPKG.txt does not contain its own update history.

> * spkg-src,
> http://www.sagemath.org/doc/developer/packaging.html#modified-tarballs
> states its optional, I would say it shouldn't be, in fact I would even be in
> favor of having for every standard package even when it just downloads a
> tarball.

I have never seen an spkg-src in the build/pkgs/*/ directory, only
checksums.ini package-version.txt spkg-install SPKG.txt (and
possibly spkg-check)
but this is for spkgs where no changes at all are need to the upstream tarball.

> * in the example files for spkg-install/check, there's not the stuff
> checking we're in a shell with Sage stuff defined, is that on purpose?

Someone else can answer that, and the following.

John

> * SAGE_FAT_BINARY, maybe it should be mentioned on this page as well.
> * --libdir, same same
>
> Best,
> JP
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-release" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-release...@googlegroups.com.
> To post to this group, send email to sage-r...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-release.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jean-Pierre Flori]

To be completely clear, two additional questions:

On Thursday, January 23, 2014 10:57:29 AM UTC+1, John Cremona wrote:

Here is how I have interprested these questions for the packages I am
involved in maintining:

> * package-version.txt, should it only be updated when the tarball changes?
> this is what
> http://www.sagemath.org/doc/developer/packaging.html#package-versioning
> suggests, not what the discussion between Jeroen and I at
> http://trac.sagemath.org/ticket/15123#comment:32 did.

Yes.  It's a label for the upstream version.  Local changes to what we
do with the tarball are tracked by git anyway.

Is this enough to trigger a package rebuild if you for example add a patch?
 

> * history in SPKG.txt,
> http://www.sagemath.org/doc/developer/packaging.html#the-spkg-txt-file says
> no history anymore, so should we just get rid of the old history when
> updating? as remarked http://trac.sagemath.org/ticket/15123#comment:24 it is
> still available in the hg/git history (and I'd better have no info kept in
> SPKG.txt than just old info).

Correct:  the new SPKG.txt does not contain its own update history.

So you actually removed all the old info stored there?

Best,
JP

[Nathann Cohen]

> So you actually removed all the old info stored there?

It also happens there : http://trac.sagemath.org/ticket/9870

Nathann

[John Cremona]

On 23 January 2014 10:13, Jean-Pierre Flori <jpf...@gmail.com> wrote:
> To be completely clear, two additional questions:
>
>
> On Thursday, January 23, 2014 10:57:29 AM UTC+1, John Cremona wrote:
>>
>> Here is how I have interprested these questions for the packages I am
>> involved in maintining:
>> > * package-version.txt, should it only be updated when the tarball
>> > changes?
>> > this is what
>> > http://www.sagemath.org/doc/developer/packaging.html#package-versioning
>> > suggests, not what the discussion between Jeroen and I at
>> > http://trac.sagemath.org/ticket/15123#comment:32 did.
>>
>> Yes. It's a label for the upstream version. Local changes to what we
>> do with the tarball are tracked by git anyway.
>>
> Is this enough to trigger a package rebuild if you for example add a patch?

I don't know. I was working with optional packages and reinstalled
them manually.

>
>>
>> > * history in SPKG.txt,
>> > http://www.sagemath.org/doc/developer/packaging.html#the-spkg-txt-file
>> > says
>> > no history anymore, so should we just get rid of the old history when
>> > updating? as remarked http://trac.sagemath.org/ticket/15123#comment:24
>> > it is
>> > still available in the hg/git history (and I'd better have no info kept
>> > in
>> > SPKG.txt than just old info).
>>
>> Correct: the new SPKG.txt does not contain its own update history.
>>
> So you actually removed all the old info stored there?

Yes.

John

[Jean-Pierre Flori]

Yet another question:
* "upstream" tarballs are stored in the upstream dir without the ".p?" part, but are not really upstream, i.e. can be repackaged, stripped, include additional stuff.
So how to update these tarballs when the actual upstream version is the same, but the sage specific changes do change.
For example, see http://trac.sagemath.org/ticket/15615#comment:14 where the atlas tarball should be updated to include fixes.
Or http://trac.sagemath.org/ticket/14648 where we'd like to completely get rid of GMP within ECL tarball.

[Jean-Pierre Flori]

In particular, if the ".p?"  part of the package version now only concerns the "upstream" tarball as the dev doc and John suggest, then it would make sense to name the upstream tarballs with the ".p?" part.

If not, that is if the ".p?" part is used to trigger package rebuild (let's say if a new patch was added), then we should think of another way of updating "upstream" tarballs.

[John Cremona]

Yes, I was told that upstream tarballs should not contain any Sage
changes, those should be done via patches in the install script. And
hence upstream tarballs should not have any .p in them. I did that
wrongly when I redid the database_stein_watkins optional spkgs for git
and was told to change it by Jeroen.

Someone like Volker as release manager should probably tell use the real story.

John

[Jean-Pierre Flori]

On Thursday, January 23, 2014 12:07:44 PM UTC+1, John Cremona wrote:

Yes, I was told that upstream tarballs should not contain any Sage
changes, those should be done via patches in the install script.  And
hence upstream tarballs should not have any .p in them.  I did that
wrongly when I redid the database_stein_watkins optional spkgs for git
and was told to change it by Jeroen.

Sure, I definitely agree about not "patching" upstream tarballs, except that some "upstream" tarballs are not really vanilla:
* some stuff might be deleted to save space (and I just would like to delete a little more in ECL than used to be),
* some autogenerated additional stuff might get added (see the custom autotools project for ATLAS),
* some addiitonal "binary"  might be added (see the archdef tarballs for ATLAS).
This situation is kind of exceptional but still problematic.

Someone like Volker as release manager should probably tell use the real story.

Sure, just waiting for some feedback to unlock some tickets.

And thanks for your replies!

[Volker Braun]

You *should* not change upstream tarballs. Sometimes, we have to. There is no good way to change the upstream tarball without changing the version number. Arguably, thats a feature. Maybe use package-2014123.tar.gz if you have to.

There is no point in a spkg-src script if the tarball is not modified. Did you forget how to download files? ;-)

Right now, only the package-version.txt timestamp is used in the makefile dependency calculation. So you have to add a .p<X> to the version number there (but not in the tarball) to trigger a rebuild if necessary.  (*)

Feel free to strip out the old changelog from the SPKG.txt file, but please no autogenerated patches that do it globally.

I'm against boilerplate copy-pasta in spkg-check/spkg-install files to check that we are really being called from Sage.  (*)

(*) these really ought to be addressed in future changes to the build system.

[Jeroen Demeyer]

On 2014-01-23 10:51, Jean-Pierre Flori wrote:
> * package-version.txt, should it only be updated when the tarball
> changes?

package-version.txt is what triggers a rebuild of a package, so the .p
level must be increased whenever you make a change to a package which
affects the functioning of the package. Usually, if you add a patch to a
package, you should increase the .p level.

If you only make changes to port a package to a new system, there is no
need to rebuild the package on every system. I this case, there is no
need to increase the .p level.

Jeroen.

[Nathann Cohen]

> (*) these really ought to be addressed in future changes to the build
> system.

By the way, wasn't there some talk at some point of automatically
testing optiona doctests when the corresponding spkg is installed ? I
do not know if anything happened on that front, but it would be
totally cool !!!

Nathann

[John Cremona]

"boilerplate copy-pasta" -- I want some now!

John

>
>
> (*) these really ought to be addressed in future changes to the build
> system.
>

[Jean-Pierre Flori]

Thanks for all the answers, I think I know what I'll do now:
* update package-version.txt if a rebuild is needed on any system where the previous version did build as well.
* update a tarball name with date info if some change is needed without an actual upstream version bump.
* remove old history when an spkg still containing it is being updated.
* don't add the Sage env detection stuff to new install scripts, but don't remove what's already there.
* don't add quite empty spkg-src scripts.