Is there an online index of the standard packages shipped in Sage?

[Nathann Cohen]

Hello everybody,

I am trying to 'clean' a bit some other part of Sage's documentation.
I found that page, which lists 'some of' the programs shipped in Sage:

http://www.sagemath.org/doc/installation/introduction.html

It is not very clearly presented, nor probably very updated. Plus we
could have links toward all those specific projects. I have two
questions:

1) Is there an updated index of the packages we ship, somewhere in our doc? [1]

2) If we do not, where do you think it belongs?

Thanks,

Nathann

[1] I am aware of one page, but it is only available online:
http://www.sagemath.org/packages/standard/

[leif]

There's also this dead(?) wiki page:

http://wiki.sagemath.org/standard_packages_available_for_SAGE

And this one I kept updating for a while (quite some time ago):

http://wiki.sagemath.org/Sage_Spkg_Tracking


-leif

[Nathann Cohen]

Yo !

> There's also this dead(?) wiki page:
>
> http://wiki.sagemath.org/standard_packages_available_for_SAGE
>
> And this one I kept updating for a while (quite some time ago):
>
> http://wiki.sagemath.org/Sage_Spkg_Tracking

Fine. Seems to confirm that we need something more reliable somewhere.
Ideally, it could be somewhere in our doc, with a doctest to check
that all packages appear.

Do you have any intuition on where it should appear?

Nathann

[leif]

Nathann Cohen wrote:
>> There's also this dead(?) wiki page:
>>
>> http://wiki.sagemath.org/standard_packages_available_for_SAGE
>>
>> And this one I kept updating for a while (quite some time ago):
>>
>> http://wiki.sagemath.org/Sage_Spkg_Tracking
>
> Fine. Seems to confirm that we need something more reliable somewhere.

+1

> Ideally, it could be somewhere in our doc, with a doctest to check
> that all packages appear.

That's perhaps a bit problematic at the moment; also, at least some
tests may require internet access (although we could tag them
accordingly). Should we document/test their versions as well?

Furthermore, in the "git layout" we only have "new-style" packages
(which are currently still a subset of standard+optional), with no
differentiation between standard and optional; there, experimental
packages (and "the" huge one?) are missing completely AFAIK.

> Do you have any intuition on where it should appear?

>>> from __future__ import intuition
SyntaxError: future feature intuition is not defined

$ ls -l sage/src/doc/
total 48
d--xr-xr-x 3 leif sage 4096 May 5 02:50 ca
drwxr-xr-x 4 leif sage 4096 May 11 19:29 common
d-wxr-xr-x 5 leif sage 4096 May 5 02:50 de
d-wxr-xr-x 13 leif sage 4096 May 5 02:50 en
d--xr-xr-x 4 leif sage 4096 May 5 02:50 fr
d--xr-xr-x 3 leif sage 4096 May 5 02:50 hu
d--xr-xr-x 3 leif sage 4096 May 5 02:50 it
-rw-r--r-- 1 leif sage 334 May 5 02:50 Makefile
d-wxr-xr-x 5 leif sage 4096 May 10 14:05 output
d--xr-xr-x 4 leif sage 4096 May 5 02:50 pt
d--xr-xr-x 3 leif sage 4096 May 5 02:50 ru
d--xr-xr-x 3 leif sage 4096 May 5 02:50 tr


-leif

[Nathann Cohen]

> That's perhaps a bit problematic at the moment; also, at least some
> tests may require internet access (although we could tag them
> accordingly). Should we document/test their versions as well?

Technically, the list of standard packages is available in Sage
directly (build/install). Maybe we should have a file whose content is
the list of standard packages (and only that) which build/install
could then load?

Volker, what do you think?

It seems that all lines in build/install are of the form
package_name=`newest_version package_name`. Couldn't we generate this
from the list of packages instead?

> Furthermore, in the "git layout" we only have "new-style" packages
> (which are currently still a subset of standard+optional), with no
> differentiation between standard and optional; there, experimental
> packages (and "the" huge one?) are missing completely AFAIK.

Yep, only those which have been converted into 'new-style' packages
appear there. Though you can get these lists with:
sage -standard
sage -optional
sage -experimental

All of them require an internet connection.

Nathann

[Volker Braun]

IMHO:

* The package type (standard/optional/experimental) should be in a file inside the package directory

* The build system should check that it builds only standard packages

* Old-style spkgs shouldn't be mashed together with new-style packages, that is super confusing if you have both.

* Instead, put all old spkgs in a separate category (sage -old-spkg or so), or list them separately at the end of "sage -optional"

[Thierry]

Hi,

on-line, you can get the list of new-style spkgs at
http://git.sagemath.org/sage.git/tree/build/pkgs

The standard ones should be the ones listed at:
http://git.sagemath.org/sage.git/tree/COPYING.txt

Or, with more preparsing (so less good for a direct link), at:
http://git.sagemath.org/sage.git/tree/build/install


The problem with pages under http://www.sagemath.org/packages is that
they are not up-to-date and there is a conflict with the old-style
versions and the new-style one, for example, if you type ::

sage -standard

it relies on the page http://www.sagemath.org/packages/standard/list, and
you will get something that is inconsistent according to versions, e.g.::

sympy-0.7.3 ............................ installed version: 0.7.6

Instead of writing a new script to deal bith both-old-and-new-style spkgs,
i would be +1 for simply removing old-style spkgs, and migrate the useful
ones as new-style, then such listing would be trivial to write (and not
require internet connection).

The main question is: which old-style spkgs are worth being saved (as for
Sage Debian Live, the only missing is nauty), and how to establish such a
list ?

Ciao,
Thierry

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
>

[Thierry]

Hi,

On Tue, May 12, 2015 at 07:19:07AM -0700, Volker Braun wrote:
> IMHO:
>
> * The package type (standard/optional/experimental) should be in a file
> inside the package directory

+1. Actually, i have a prototype of this for the purpose of checking to
the upstream website if a newerver version is out (i did it mainly for
openssl, but it now covers most packages), and get some other information
by browsing the source code (e.g. which are installed, how many patches,
...).

> * The build system should check that it builds only standard packages

Or even conversely, generate build/deps from such atomic informations, of
course, we should mention both
- the priority (toolchain, base, standard, optional, experimental)
- the dependencies

This would have the benefits over the current system to be able to provide
dependencies not only for standard packages but also for optional ones, we
could also mention pip dependencies there.

If people think it is interesting and has a chance to be merged at some
point, i can update my prototype to that aim. I was shy to propose it
since it is written in Python, and, if it is used to compile the whole
stuff, then Python will be a requirement, which seems currently pretty
controversial.

> * Old-style spkgs shouldn't be mashed together with new-style packages,
> that is super confusing if you have both.

+1.

> * Instead, put all old spkgs in a separate category (sage -old-spkg or so),
> or list them separately at the end of "sage -optional"

+1, see my previous answer, or why not simply just trash old-style, while
migrating interesting ones to new-style ? They should not be much in the
list since those were not updated since a while now, and there was no
complain, so i guess most are unused or do not deal with mathematics (e.g.
trac, nose, beautifulsoup,...).

Ciao,
Thierry

>
>
>
> On Tuesday, May 12, 2015 at 3:57:47 PM UTC+2, Nathann Cohen wrote:
> >
> > > That's perhaps a bit problematic at the moment; also, at least some
> > > tests may require internet access (although we could tag them
> > > accordingly). Should we document/test their versions as well?
> >
> > Technically, the list of standard packages is available in Sage
> > directly (build/install). Maybe we should have a file whose content is
> > the list of standard packages (and only that) which build/install
> > could then load?
> >
> > Volker, what do you think?
> >
> > It seems that all lines in build/install are of the form
> > package_name=`newest_version package_name`. Couldn't we generate this
> > from the list of packages instead?
> >
> > > Furthermore, in the "git layout" we only have "new-style" packages
> > > (which are currently still a subset of standard+optional), with no
> > > differentiation between standard and optional; there, experimental
> > > packages (and "the" huge one?) are missing completely AFAIK.
> >
> > Yep, only those which have been converted into 'new-style' packages
> > appear there. Though you can get these lists with:
> > sage -standard
> > sage -optional
> > sage -experimental
> >
> > All of them require an internet connection.
> >
> > Nathann
> >
>

[leif]

Thierry wrote:
> [snip]

>
> The problem with pages under http://www.sagemath.org/packages is that
> they are not up-to-date and there is a conflict with the old-style
> versions and the new-style one, for example, if you type ::
>
> sage -standard
>
> it relies on the page http://www.sagemath.org/packages/standard/list, and
> you will get something that is inconsistent according to versions, e.g.::
>
> sympy-0.7.3 ............................ installed version: 0.7.6

Not just versions, also their names:

libm4ri-20130416 ....................... not installed
libm4rie-20130416 ...................... not installed

TOPCOM vs. topcom as an optional package is also nice.

"pil" got /replaced/ by "Pillow" (or "pillow", who knows).

> Instead of writing a new script to deal bith both-old-and-new-style spkgs,
> i would be +1 for simply removing old-style spkgs, and migrate the useful
> ones as new-style, then such listing would be trivial to write (and not
> require internet connection).

So 'sage --standard' would just show "installed" for every package which
is considered standard in the current Sage installation, how useful.
You could of course list whether newer versions are available online,
just to tell people they have to upgrade all of Sage (probably to a
devel version) in order to get/install (all of!) them. (The same is
even true for "new-style" *optional* spkgs.)

"Old-style" spkgs have their raison d'être (namely, they're
self-contained); in the long run, we may deprecate them, but we should
at least keep support for installing such (e.g. ones offered elsewhere).

Requiring each and every installable package to have its (exact)
metadata in the git tree at hand is just stupid. (I proposed separate
spkg metadata files* about four or five years ago.)


-leif

___________
* SagePAckageMetadata^TM files, *.spam

[Volker Braun]

libm4ri and libm4rie are two different libraries...

[leif]

Volker Braun wrote:
> libm4ri and libm4rie are two different libraries...

Sure. But their "new-style" names are "m4ri" and "m4rie", hence 'sage
--standard' lists the former ones as "not installed" (and doesn't
mention the latter ones because they're not in 'spkg/standard/' on the
servers).


-leif

> On Tuesday, May 12, 2015 at 8:36:38 PM UTC+2, leif wrote:
> Thierry wrote:
> > The problem with pages under http://www.sagemath.org/packages
> <http://www.sagemath.org/packages> is that
> > they are not up-to-date and there is a conflict with the old-style
> > versions and the new-style one, for example, if you type ::
> >
> > sage -standard
> >
> > it relies on the page
> http://www.sagemath.org/packages/standard/list

> <http://www.sagemath.org/packages/standard/list>, and

> > you will get something that is inconsistent according to versions,
> e.g.::
> >

> > sympy-0.7.3 ............................. installed

[Thierry]

On Tue, May 12, 2015 at 08:36:24PM +0200, leif wrote:
[...]

> "Old-style" spkgs have their raison d'ętre (namely, they're

> self-contained); in the long run, we may deprecate them, but we should
> at least keep support for installing such (e.g. ones offered elsewhere).

Of course, since this is the way third parties can ship their own spkgs.
The problem we are facing here is the mixture of old-and-new officially
supported spkgs.

> Requiring each and every installable package to have its (exact)
> metadata in the git tree at hand is just stupid. (I proposed separate
> spkg metadata files* about four or five years ago.)

Why not, could you please tell more about this ? In particular, if this
metadata is disconnected to the git tree, how to know whether a newer
version of some package could work on an older version of Sage ? Having
admissible intervals of versions would certainly help in "making Sage
distribution friendly", but it will probably require much more patchbots
to ensure this.

Ciao,
Thierry

[leif]

Volker Braun wrote:
> IMHO:
>
> * The package type (standard/optional/experimental) should be in a file
> inside the package directory

Yes. Suggestions?

checksums.ini and package-version.txt don't seem appropriate to add it
there (nor of course SPKG.txt), so we'd need a new file I think.

It would IMHO be better to have a single (extensible) file for each
package with such meta information. But we could change that later.

> * The build system should check that it builds only standard packages

We also have sage.misc.package.install_all_optional_packages() which
should then work again (modulo build errors of course).

> * Old-style spkgs shouldn't be mashed together with new-style packages,
> that is super confusing if you have both.

I'm currently working on making sage-list-packages a bit smarter, such
that it can cope with the current situation.


-leif

[Dima Pasechnik]

On Wednesday, 13 May 2015 17:55:17 UTC+1, leif wrote:

Volker Braun wrote:
> IMHO:
>
> * The package type (standard/optional/experimental) should be in a file
> inside the package directory

hmm...

rather, how about SAGEROOT/build/pkg/standard/ , SAGEROOT/build/pkg/optional/, etc?

that's much more clear than all these lists...

[leif]

Dima Pasechnik wrote:
> On Wednesday, 13 May 2015 17:55:17 UTC+1, leif wrote:
>
> Volker Braun wrote:
> > IMHO:
> >
> > * The package type (standard/optional/experimental) should be in a
> file
> > inside the package directory
>
> hmm...
>
> rather, how about SAGEROOT/build/pkg/standard/
> , SAGEROOT/build/pkg/optional/, etc?
>
> that's much more clear than all these lists...

But changing a single file in case the category changes (from optional
to standard, say) is certainly simpler (or less error-prone) than moving
the whole folder.

And we wouldn't have to change the whole bunch of build and
package-related scripts. (With an additional file, we'd need only minor
changes to treat e.g. optional packages differently when appropriate.)


-leif

[Dima Pasechnik]

emulating a filesystem sucks.

You'd have to write and maintain scripts that parse these custom file

catalogs, etc etc etc.

 

 

-leif

[leif]

We were not talking about a catalog, but an additional file for each
package, located in build/pkgs/$package/, which just contains
"standard", "optional", "experimental" or probably "base", say.

Having to determine (or defining) the category of a package by the
location of its data in the git / filesystem tree is pretty inconvenient
and IMHO more confusing, and only complicates things when its category
changes.

(On the other hand, we did indeed have at least one package where there
was a standard version and one or more optional versions at the same
time, namely GCC. But with the "new-style" packages, or the "new
layout", the option to install a different version than that hardcoded
into the specific Sage version unfortunately vanished anyway. It's not
impossible, but far from straight-forward, and certainly nothing for an
"end user". Same for upgrading just a specific package, from .pN to
.pM, say, which is a typical but now unsupported use case for users with
build problems.)


-leif

[Dima Pasechnik]

Really?! Are you saying that looking at standard/*/ is more inconvenient

than looking at */* and checking that a file there says standard (or have a file

named standard) ?

[Nathann Cohen]

Hello,

> Really?! Are you saying that looking at standard/*/ is more inconvenient
> than looking at */* and checking that a file there says standard (or have a
> file
> named standard) ?

It may not be more convenient indeed if you only want to remember the
'status' (standard/optional/experimental) of the package, though we
may have more 'flags' to add to each folder: for instance the *other*
packages which are dependencies of a given package. At the moment this
is stored in build/deps, which as you may agree is not "as distributed
as it could".

Plus it seems that there are several flavors of standard packages. In
'deps', the packages prereq/bzip2/patch/pkgconf play a special role as
they are needed by the build system itself.

We may have to create a "SuperStandard" directory for them? Or else we
do everything in files, and eventually store their 'type', their
dependencies, and possibly other things too in there.

Nathann

[Dima Pasechnik]

On Thursday, 14 May 2015 07:10:10 UTC+1, Nathann Cohen wrote:

Hello,

> Really?! Are you saying that looking at standard/*/ is more inconvenient
> than looking at */* and checking that a file there says standard (or have a
> file
> named standard) ?

It may not be more convenient indeed if you only want to remember the
'status' (standard/optional/experimental) of the package,

well, isn't it precisely the proposal we are discussing?

 

though we
may have more 'flags' to add to each folder: for instance the *other*
packages which are dependencies of a given package. At the moment this
is stored in build/deps, which as you may agree is not "as distributed
as it could".

I never said that it is convenient (or even possible) to encode build dependencies in directory structure.

[Nathann Cohen]

> We were not talking about a catalog, but an additional file for each
> package, located in build/pkgs/$package/, which just contains
> "standard", "optional", "experimental" or probably "base", say.

Done in #18431 (needs review)

http://trac.sagemath.org/ticket/18431

Nathann

[Julien Puydt]

Hi,

Le 12/05/2015 17:19, Thierry a écrit :
> On Tue, May 12, 2015 at 07:19:07AM -0700, Volker Braun wrote:
>> IMHO:
>>
>> * The package type (standard/optional/experimental) should be in a file
>> inside the package directory
>
> +1. Actually, i have a prototype of this for the purpose of checking to
> the upstream website if a newerver version is out (i did it mainly for
> openssl, but it now covers most packages), and get some other information
> by browsing the source code (e.g. which are installed, how many patches,
> ...).

Is your prototype to check for latest versions of upstream as good as
uscan ? [ see https://wiki.debian.org/debian/watch ]

Snark on #sagemath

[Jean-Pierre Flori]

Great news!
I guess we can finally have a "make donwload" target which only downloads standard packages now.
And maybe a simple way to cleanup the upstream directory, see #16327.