Dropping support for old-style .spkgs?

[Erik Bray]

We still have quite a bit of code around for supporting old-style
packages in .spkg archives, though it is not well tested anymore and
I'm not even 100% sure installing these still works.

We have also noted, since Sage 6.9, that these are deprecated and unsupported.

That seems like long enough to me to rip out all support for them.
I'd especially like to do this since I feel increasingly compelled to
rewrite most or all of the sage-spkg script in Python, and it will
greatly simplify matters to not have to maintain that support.

But even that aside, it seems well past time?

Thanks,
E

[David Roe]

Sage 6.9 was released in October 2015.  Dropping support for old-style spkgs seems reasonable to me.

David

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Jeroen Demeyer]

On 2018-08-08 19:06, Erik Bray wrote:
> That seems like long enough to me to rip out all support for them.
> I'd especially like to do this since I feel increasingly compelled to
> rewrite most or all of the sage-spkg script in Python, and it will
> greatly simplify matters to not have to maintain that support.

Fine for me, as long as you separate the "dropping support the old-style
spkgs" and "rewrite sage-spkg" parts.

[Travis Scrimshaw]

IMO, it would be good to have here a current list of old-style spkgs so we can be explicit about what things we are dropping support for.

Best,

Travis

[Andrey Novoseltsev]

On Wednesday, 8 August 2018 17:19:13 UTC-6, Travis Scrimshaw wrote:

IMO, it would be good to have here a current list of old-style spkgs so we can be explicit about what things we are dropping support for.

And can we please not omit "huge" packages like polytopes_db_4d-1.0.spkg - I do use it and would like to keep it available, but I have no idea where it is located and whether downloading it still works - I use a local copy for many years (it is 8.7G).

[Vincent Delecroix]

Could you kindly upgrade it to a new style package? (or at least open a
ticket with a pointer to the spkg tarball)

Vincent

[Erik Bray]

Of course. I already started work on this and I wanted to do things
this way since dropping support first makes sage-spkg much much
simpler and thus, easier to (eventually) port.

[Erik Bray]

On Thu, Aug 9, 2018 at 6:54 AM Vincent Delecroix

Entirely by coincidence, unrelated to this thread, I just opened:
https://trac.sagemath.org/ticket/26029

I'm glad to know there's at least someone who uses it. I was not able
to find a copy of the package itself on any of the Sage mirrors I
checked. But I think we should decide what to do about it. Keeping
it as a .spkg isn't really helpful, and keeping "huge" packages as a
special case for still supporting .spkgs defeats the purpose of
dropping support.

[Thierry]

Hi,

For the record, note that there is a wiki page that follows the migration,
and that lists the old-style packages that are still of interest
(polytopes_db_4d is one of them)
https://wiki.sagemath.org/Classify%20old-style%20packages

Ciao,
Thierry

[Erik Bray]

Perhaps a good way to find out what really needs attention is to

a) Make the deprecation louder--e.g. break outright and refuse to
install unless an additional flag is added or something like that.

b) See who complains--if anyone--and if not we can drop them.

For all the non-mathematical and experimental packages I would just
leave them if they are not required by Sage.

[Dima Pasechnik]

could you offer this package for download, at least temporarily?
Perhaps we could put it on github...

[Erik Bray]

On Fri, Aug 10, 2018 at 11:46 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> could you offer this package for download, at least temporarily?
> Perhaps we could put it on github...

Where did it come from in the first place? Is this some pre-computed
data? Was it computed by Sage? Something else...?

[Samuel Lelievre]

The package is here:

http://files.sagemath.org/spkg/huge/ 

[Erik Bray]

Out of curiosity, why were .spkg packages deprecated in the first
place? I'm fine with keeping it that way / removing support. It just
wasn't clear to me in the first place. It seems like an entirely
reasonable thing to have, though maybe difficult to control.

I ask because I *am* interested in having pre-built binary packages
that can be dropped into Sage, but that would be slightly different.
I'm motivated, in that case, by the use case of supplying pre-built
binary packages for optional packages on Windows.

[Samuel Lelievre]

The "Environment variables" section of the "Install from source code"

page of the reference manual,

https://github.com/sagemath/sage/blob/master/src/doc/en/installation/source.rst#environment-variables

has this sentence:

> Note that Sage will search the directory

>

> - `SAGE_SERVER/spkg/upstream`

>

> for clean upstream tarballs, and it searches the directories

>

> - `SAGE_SERVER/spkg/standard/`,

> - `SAGE_SERVER/spkg/optional/`,

> - `SAGE_SERVER/spkg/experimental/`,

> - `SAGE_SERVER/spkg/archive/`

>

> for old-style Sage packages.

but this fails to mention the huge/ subdirectory.

Also, the archive/ subdirectory is no longer found at

http://files.sagemath.org/spkg/archive/

but it exists at

http://old.files.sagemath.org/spkg/archive/

[Andrey Novoseltsev]

Data originates from http://hep.itp.tuwien.ac.at/~kreuzer/CY/

The whole list can be generated using PALP (a standard package in Sage), but when they have done it first, it took about a year on multiple computers, so there is sense in just downloading these polytopes. Sage package contains precomputed data for Hodge numbers as well, allowing quick search for particular cases. Volker knows more about exact details, I was just happily using the result.

While we are at it: it would be awesome if installation of this package after upgrade was faster. "Installation" just means unpacking files and copying them somewhere, but it takes something like 45 minutes. Some time ago I asked to use parallel decompressor if it is available in the system, which helped a lot, but then it stopped again.

Thank you!
Andrey

[Erik Bray]

I introduced some regressions a little while back that are probably
making this extremely slow, but am working on fixing it:
https://trac.sagemath.org/ticket/26011

That said, I don't believe it should really be a Sage package in the
first place. At the very least maybe a "script" type package, which
just runs some commands that might download a tarball and unpack it or
something, but is not considered a normal package. Better still, huge
data like this really ought to be accessible through some kind of
online database (though one would still want to be able to download
the full dataset from somewhere too).

[Samuel Lelievre]

However the huge/ directory should be searched to according to

line 396 of the sage-spkg script:

https://github.com/sagemath/sage/blob/master/build/bin/sage-spkg#L396

            for repo in optional experimental huge; do

[Simon King]

Hi Erik,

On 2018-08-10, Erik Bray <erik....@gmail.com> wrote:
> I ask because I *am* interested in having pre-built binary packages
> that can be dropped into Sage, but that would be slightly different.

If I understand correctly, having pre-built packages would be totally
orthogonal to the question whether old style packages should still be
supported.

Originally, I was against changing from old to new style spkgs, and I
still don't see a very compelling reason why the change was made.
Isn't the only difference that everything that used to be in a single
bzip compressed tar ball (spkg-install, spkg-check, SPKG.txt, patches,
the upstream sources) is now more distributed (with upstream sources
in upstream/ and the rest in build/pkgs)?

I do see an advantage of having spkg-* under version control. And I do
of course see that in the long run it isn't very fruitful to support two
different formats.

Best regards,
Simon

[Simon King]

Hi Erik,

On 2018-08-10, Erik Bray <erik....@gmail.com> wrote:

> That said, I don't believe it should really be a Sage package in the
> first place. At the very least maybe a "script" type package, which
> just runs some commands that might download a tarball and unpack it or
> something, but is not considered a normal package. Better still, huge
> data like this really ought to be accessible through some kind of
> online database (though one would still want to be able to download
> the full dataset from somewhere too).

+1.

If I would pack all the results of my group cohomology package into the
package rather than making it web accessible, it would be uselessly huge
as well.

[Michael Orlitzky]

On 08/10/2018 11:50 AM, Erik Bray wrote:
>>
>> While we are at it: it would be awesome if installation of this
>> package after upgrade was faster. "Installation" just means unpacking
>> files and copying them somewhere, but it takes something like 45
>> minutes. Some time ago I asked to use parallel decompressor if it is
>> available in the system, which helped a lot, but then it stopped again.
>

> ...

>
> That said, I don't believe it should really be a Sage package in the
> first place.

If anything, it should be a standalone (system) package. We could then
point sage at the pile of data with a ./configure flag. No need to
"upgrade" the sage copy of the same stuff every time.

[Erik Bray]

I'm not sure what you mean in this case by "system package". As I see
it it's just a bunch of files that could be unpacked anywhere, and any
code that uses it could be pointed to it with an argument and/or
environment variable, perhaps with a sensible default (in fact, this
is more or less what's already done, and the packaging as an "spkg" is
just a convenience to make sure it's unpacked automatically to the
correct default location).

I'm not even against it remaining as a new-style spkg; it would just
have to be repackaged as a .tar.<whatever compression works best>, and
we would need to make sure that sage-download-file can find files
under http://files.sagemath.org/spkg/huge/

[Michael Orlitzky]

On 08/14/2018 12:22 PM, Erik Bray wrote:
>
> I'm not sure what you mean in this case by "system package".
>

Have the upstream authors create a (basically empty) configure.ac file,
call it version 1.0.0, and release a tarball.

[Erik Bray]

Why would they do even that? It's just a data package. For the
purposes of upstream packaging it doesn't need a configure.ac. It can
just be a bundle of files. But I still don't see what that solves.
Good luck getting 30 GB of obscure math data upstreamed to an
arbitrary number of OS packagers...

[Michael Orlitzky]

On 08/14/2018 12:54 PM, Erik Bray wrote:
>
> Why would they do even that? It's just a data package. For the
> purposes of upstream packaging it doesn't need a configure.ac. It can
> just be a bundle of files. But I still don't see what that solves.
> Good luck getting 30 GB of obscure math data upstreamed to an
> arbitrary number of OS packagers...

From the Gentoo perspective: the existence of ./configure means that the
default build process will put everything in the right place without me
having to figure out where that is and copy/paste it. And the "package"
is nothing but a tiny text file until somebody installs it.

[Erik Bray]

I see what you're saying now--it obviates the need for any "custom"
installation instructions. If someone wanted to do that then it's
obviously fine. Though I don't see how that ultimately relates to the
topic of deprecating old-style Sage spkgs.

[Erik Bray]

On Wed, Aug 8, 2018 at 7:06 PM Erik Bray <erik....@gmail.com> wrote:
>
> We still have quite a bit of code around for supporting old-style
> packages in .spkg archives, though it is not well tested anymore and
> I'm not even 100% sure installing these still works.
>
> We have also noted, since Sage 6.9, that these are deprecated and unsupported.
>
> That seems like long enough to me to rip out all support for them.
> I'd especially like to do this since I feel increasingly compelled to
> rewrite most or all of the sage-spkg script in Python, and it will
> greatly simplify matters to not have to maintain that support.
>
> But even that aside, it seems well past time?

The original discussion got a little off-topic. With the exception of
maybe one or two packages for which there is a desire to convert to
new-style packages--which can certainly be done--is there any
objection to dropping support for old-style packages, both from the
code and the documentation?

[E. Madison Bray]

On Thursday, August 16, 2018 at 2:36:25 PM UTC+2, E. Madison Bray wrote:

With apologies for resurrecting an old thread, just wanted to note that I added a note in the ticket about this polytopes_db_4d about the possibility of making it accessible as an online database: https://trac.sagemath.org/ticket/26029#comment:9

Would anyone have interest in this (not necessarily just working on the problem, but actually having access to such a database?)  Or would it not make much sense to?