Should sage -i install *dependencies* of packages?

[Jeroen Demeyer]

This question is because of discussion at #18859.

The current behaviour of "sage -i PKG" is to install package PKG (if the
latest version of that package is not already installed), without
looking at the dependencies of PKG.

Now what should the recommended user interface be for installing a
package *with* dependencies?

(A) change "sage -i PKG" to also install dependencies (adding some
option --no-dependencies to keep the old behaviour)
(B) add a new option, say "sage -m PKG" (-m for make) to install a
package with dependencies
(C) "make PKG" (which is currently supported but documented)

* Note that (C) is absolutely no change in functionality, it's just a
documentation change. (B) is a minor change in functionality (a few
lines in src/bin/sage), while (A) adds significant complexity.

* Options (B) and (C) are fully backwards compatible, no changes to
current behaviour are needed.

* An advantage of (A) and (B) is that they can be run from anywhere,
while (C) only works from $SAGE_ROOT.

* The main advantage of (A) is that users do not need to change
behaviour: usually installing packages *with* dependencies is what you want.

* The advantage of (C) is that it is consistent with "sage -b" and "sage
--docbuild FOO" which also do not install dependencies of the Sage
library or the documentation. The corresponding make targets "build" and
"doc" do build the dependencies.

* When choosing (A), we need to define what "sage -i -f PKG" and "sage
-f PKG" should do. I see the following possible variants:
(A1) Disallow "sage -i -f PKG" and "sage -f PKG" without the
--no-depencies option
(A2) The "-f" option implies --no-dependencies
(A3) The "-f" option forces re-installation of the package and all its
dependencies
(A4) Make a difference between "sage -f PKG" and "sage -i -f PKG"

My preference goes to either (B) or (C), mainly because of the added
complexity that (A) would require and because I like to keep a simple
command ("sage -i PKG") to install just package PKG.


What would you like?
Jeroen.

[William Stein]

On Mon, Aug 17, 2015 at 10:00 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
[...]
> What would you like?

What does every other package manager do? (I am being somewhat
rhetorical because I think I know the answer.) Maybe we should do
what *everybody* else does in this case, to minimize surprise and
confusion...?

-- William


--
William (http://wstein.org)

[Jeroen Demeyer]

On 2015-08-17 18:35, William Stein wrote:
> On Mon, Aug 17, 2015 at 10:00 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> [...]
>> What would you like?
>
> What does every other package manager do?

I don't think that's relevant. The question is not whether or not it
should be possible to install packages with dependencies (what a package
manager does), only what the user interface should be.

My question could be rephrased as: should "sage -i" start behaving like
a package manager or should it be just a low-level command to install a
single package?

Jeroen.

[William Stein]

Let's at least establish what people are likely to think "sage -i"
already is (after 10+ years of Sage). I double people think it's
just a low-level command, since "sage --help" lists it at the top
level (no need to use "sage -advanced" to find out about it).

sage --help
...
-advanced -- list all command line options
...
-i [packages] -- install the given Sage packages
...

It's description is "install the given Sage packages".

Wikipedia: "A package manager [...] is a collection of software tools
that automates the process of installing [...] software packages..."

William




>
>
> Jeroen.
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[kcrisman]

How many packages currently have such (additional) dependencies now as opposed to several years ago?  I only expect it to install one package at a time but I also can't remember installing a lot of packages that had dependencies that weren't standard packages.  Maybe we're separating out things more than we used to, so this is causing trouble for that reason?  In which event the default could indeed change to installing dependencies (which I have to say is VERY convenient in R, but is also a zillion times more necessary in R).

[Nathann Cohen]

What would you like?

As your ticket aims at making users install packages in a way that takes dependencies into account, I vote for (A) which does not change the command that they are used to type to install packages.

Nathann

[Jeroen Demeyer]

On 2015-08-17 19:12, kcrisman wrote:
> How many packages currently have such (additional) dependencies now as
> opposed to several years ago?

Currently, the only packages which depend on a package which is not
standard are:

* python_igraph depending on igraph

* qepcad depending on saclib

By a strange coincidence, both are introduced in the latest beta
(6.9.beta2).

> I only expect it to install one package
> at a time but I also can't remember installing a lot of packages that
> had dependencies that weren't standard packages.

Note that "./sage -i" might also be used for standard packages when
doing Sage development, it's not just optional/experimental packages.

> In which event the default could indeed change to installing
> dependencies

That's obvious, the default absolutely must change to installing
dependencies. But, like William's answer, that's just shifting the
question: the default of *which command* should be like that?

Jeroen.

[Jeroen Demeyer]

On 2015-08-17 19:01, William Stein wrote:
> Let's at least establish what people are likely to think "sage -i"
> already is (after 10+ years of Sage).

Personally, I see it as a low-level command, analogous to other
low-level commands like ./sage -b and ./sage --docbuild.

> -i [packages] -- install the given Sage packages
> ...
>
> It's description is "install the given Sage packages".
>
> Wikipedia: "A package manager [...] is a collection of software tools
> that automates the process of installing [...] software packages..."

It's not because "sage -i" installs packages and a package manager
installs packages, that "sage -i" is a package manager.

I'm not saying this to be pedantic, since I think both
install-with-dependencies-like-a-package-manager and
install-just-the-one-package are needed commands for Sage and they both
install packages.

Jeroen.

[John H Palmieri]

On Monday, August 17, 2015 at 2:34:19 PM UTC-7, Jeroen Demeyer wrote:

On 2015-08-17 19:01, William Stein wrote:
> Let's at least establish what people are likely to think "sage -i"
> already is (after 10+ years of Sage).

I guess this is one of the questions: should Sage users be encouraged to switch from "sage -i pkg" to "make pkg" to install packages? As long as "sage -i pkg" continues to work (either as before or including dependency-checking), would "make pkg" be better for the future? I can see an argument for that.

My current vote (although I keep changing my mind):

- document "make pkg" to install packages, not "sage -i pkg"

As far as whether "sage -i pkg" should check dependencies, I don't feel strongly about it. I don't think I want a new option "sage -m pkg" which would just run "make pkg".

--
John

[Nathann Cohen]

> My current vote (although I keep changing my mind):
>
> - document "make pkg" to install packages, not "sage -i pkg"

How do you propose we update the doc for that? Sage is filled with
exceptions which say "type sage -i <pkgname>" in ordert to install the
pacakge you need. We cannot just say "type make <pkgname>", for they
would not be able to guess in which directory they should type that.

Nathann

[Daniel Krenn]

On 2015-08-17 19:00, Jeroen Demeyer wrote:
> (A) change "sage -i PKG" to also install dependencies (adding some
> option --no-dependencies to keep the old behaviour)

+1

Daniel

[Simon King]

Hi,

> On 2015-08-17 19:00, Jeroen Demeyer wrote:
> (A) change "sage -i PKG" to also install dependencies (adding some
> option --no-dependencies to keep the old behaviour)

Since I maintain an optional package that depends on another optional
package that must not be automatically installed because of licence:
Will the automatic installation of dependencies be involve asking the
user if the licence of the dependency is not GPL compatible?

Best regards,
Simon

[Nathann Cohen]

> Since I maintain an optional package that depends on another optional
> package that must not be automatically installed because of licence:
> Will the automatic installation of dependencies be involve asking the
> user if the licence of the dependency is not GPL compatible?

Whatever happens, the spkg-install scripts will still be run. If you
add a (blocking; i.e. ask for a confirmation) warning in this script,
then I'd say that there is no problem.

Nathann

[Thierry]

Hi,

thanks Jeroen for raising this question. Note that `sage -i` appears both
in `sage -h` and `sage -advanced`, which might explain why its status is
not well defined (user interface vs atomic dev tool). So, the question is
about splitting the two features.

From the user point of view, (A) is definitely better than (B) or (C).

Having a high-level, easy to remember syntax will survive to packaging
changes, while the direct use of a makefile might not be natural if we
move to a packaging system like conda (say). Those things should remain
behind the scenes (including having to be in the right directory).

Actually, i do not see why (A) involves more complexity than (B). We
could get both benefits without much work with (D):

- move the current low-level `sage -i` to, say, `sage -p` (which
automatically includes `sage -p -c -f` and remains consistent with
`sage -b` and `sage --docbuild`, so that we keep all the benefits of
(B)). Replace `sage -i` with `sage -p` in `sage -advanced`.
- consider `sage -i` as the user interface to the current packaging
system. For now, it could just call `make`, later it could call some
conda/hashdist/whatever command transparently.

Concerning the "change of behaviour", i guess nobody except a few
developpers know that `sage -i` does only a partial job. Having `sage -i`
installing the dependencies should not be considered as a change of
behaviour, but as an improvement towards the (unchanged) expected
behaviour ("something that just works to install a package", from
anywhere, without bothering about dependencies).

There are easy and hard to remember notations. I want to plead that the
easy ones should be reserved for the end users and high-level stable
interfaces. The `sage -i` command is definitely more natural than `sage
-m` to install a package (`sage -i` is about what is done, `sage -m` is
about how it is done). So, as i explained before, i prefer (A) or (D) from
this point of view, see:

(A) users: `sage -i pkg`
devs : `sage -i --no-dependencies pkg`
todo : boring preparsing
(B) users: `sage -m pkg`
devs : `sage -i pkg`
todo : almost nothing but user documentation
(C) users: `cd $SAGE_ROOT ; make pkg`
devs : `sage -i pkg`
todo : documentation and user support (please join ask.sagemath.org)
(D) users: `sage -i pkg`
devs : `sage -p pkg`
todo : almost nothing but dev documentation

Perhaps some (very few) of us will have to change our scripts to get
adapted to the fact that `sage -i` now installs dependencies, but at least
we are able to to this easily. Explaining to every user how to use make
requires much more work, especially if we have to teach another tool while
eventually changing the packaging system.

Concerning `sage -f`, isn't it more about rebuilding "backward
dependencies" than dependencies ?

Ciao,
Thierry

[Volker Braun]

Whats the use case of attempting to install a package without dependencies? I can't think of any. Whether you are developer or user, you always want to have your dependencies in place before compiling something. The fact that "sage -i" does not is 100% historical limitation. Now we can build packages with dependencies, so we should consider the old behavior as the bug that is has always been.

[John H Palmieri]

On Tuesday, August 18, 2015 at 6:49:47 AM UTC-7, Volker Braun wrote:

Whats the use case of attempting to install a package without dependencies? I can't think of any.

I'm not sure I can come up with very compelling examples, but if I've modified some package on which gcc depends, but I really want to know whether that modification affects some other package, I may just want to run "sage -i <other package>", not wait for gcc to recompile. But your point is pretty convincing to me.
 

Whether you are developer or user, you always want to have your dependencies in place before compiling something. The fact that "sage -i" does not is 100% historical limitation. Now we can build packages with dependencies, so we should consider the old behavior as the bug that is has always been.

--
John

[Dima Pasechnik]

On Monday, 17 August 2015 14:29:01 UTC-7, Jeroen Demeyer wrote:

On 2015-08-17 19:12, kcrisman wrote:
> How many packages currently have such (additional) dependencies now as
> opposed to several years ago?
Currently, the only packages which depend on a package which is not
standard are:

* python_igraph depending on igraph

* qepcad depending on saclib

By a strange coincidence, both are introduced in the latest beta
(6.9.beta2).

There are more dependencies in old-style packages, e.g. see

http://files.sagemath.org/spkg/optional/p_group_cohomology-2.1.4.p1.txt

which depends on database_gap.

[Jeroen Demeyer]

On 2015-08-18 14:49, Volker Braun wrote:
> Whats the use case of attempting to install a package without
> dependencies?

An obvious use-case is that you're switching branches/Sage versions and
you just want to try to compile a single package and you don't want to
be bothered by dependencies. I agree that installing without
dependencies is a "only use this if you know what you're doing" feature,
but it's not totally useless.

For me, it is clear that installing-with-dependencies and
installing-without-dependencies are useful.

But that was not my question: I was just asking what command should be
used to install with dependencies.


Jeroen.

[Volker Braun]

On Tuesday, August 18, 2015 at 9:26:44 PM UTC+2, Jeroen Demeyer wrote:

An obvious use-case is that you're switching branches/Sage versions and
you just want to try to compile a single package and you don't want to
be bothered by dependencies.

I'd call that "working around bugs in our broken build system". E.g. in hashdist you can't install without dependencies, as it would be guaranteed to NOT work. Previous build artifacts must never linger around if you install a package in a package manager that is worth its salt. The only sane way is to cache them separately outside of the install tree.

[Jeroen Demeyer]

On 2015-08-18 12:00, Thierry wrote:
> Actually, i do not see why (A) involves more complexity than (B).

1. We need to define how the various options to -i can be combined, in
particular the -f option is tricky.
2. The "sage -i" command would need to do two completely unrelated
things: act as high-level command to run make and low-level command to
actually install a package.

> We could get both benefits without much work with (D):
>
> - move the current low-level `sage -i` to, say, `sage -p` (which
> automatically includes `sage -p -c -f` and remains consistent with
> `sage -b` and `sage --docbuild`, so that we keep all the benefits of
> (B)). Replace `sage -i` with `sage -p` in `sage -advanced`.
> - consider `sage -i` as the user interface to the current packaging
> system. For now, it could just call `make`, later it could call some
> conda/hashdist/whatever command transparently.

Good for me, since it is very close to my proposal (C) but just with
different option names: (-i, -p) instead of (-m, -i)! I hope the other
people who voted for (A) can accept this solution too.

[Jeroen Demeyer]

On 2015-08-18 20:42, Volker Braun wrote:
> On Tuesday, August 18, 2015 at 9:26:44 PM UTC+2, Jeroen Demeyer wrote:
>
> An obvious use-case is that you're switching branches/Sage versions and
> you just want to try to compile a single package and you don't want to
> be bothered by dependencies.
>
>
> I'd call that "working around bugs in our broken build system".

No, it's called "not wanting to wait for hours until all dependencies
are compiled". The same reason that people use "./sage -b" in favour of
"make": the latter might need minutes just to determine that the
documentation does not need to be rebuilt.

[Jeroen Demeyer]

On 2015-08-18 21:16, Jeroen Demeyer wrote:
> Good for me, since it is very close to my proposal (C)

I meant (B) here (add an option -m to install with dependencies)

[Jeroen Demeyer]

I would simply not put the dependency in
build/pkgs/PKGNAME/dependencies, then your dependency will not be
automatically installed. Of course, you need to add a manual check for
the dependency in spkg-install then and force the user to manually
install your dependency.


Jeroen.

[Volker Braun]

On Tuesday, August 18, 2015 at 10:22:33 PM UTC+2, Jeroen Demeyer wrote:

> I'd call that "working around bugs in our broken build system".
No, it's called "not wanting to wait for hours until all dependencies
are compiled".

There is no need to wait for the dependencies to compile, its the job of the build system to cache that. Fast and 100% reliable incremental builds. Why settle for less? 

[kcrisman]

>
>     An obvious use-case is that you're switching branches/Sage versions and
>     you just want to try to compile a single package and you don't want to
>     be bothered by dependencies.
>
>
> I'd call that "working around bugs in our broken build system".
No, it's called "not wanting to wait for hours until all dependencies
are compiled". The same reason that people use "./sage -b" in favour of
"make": the latter might need minutes just to determine that the
documentation does not need to be rebuilt.

+1 

[Daniel Krenn]

On 2015-08-18 22:16, Jeroen Demeyer wrote:
>> - move the current low-level `sage -i` to, say, `sage -p` (which
>> automatically includes `sage -p -c -f` and remains consistent with
>> `sage -b` and `sage --docbuild`, so that we keep all the benefits of
>> (B)). Replace `sage -i` with `sage -p` in `sage -advanced`.
>> - consider `sage -i` as the user interface to the current packaging
>> system. For now, it could just call `make`, later it could call some
>> conda/hashdist/whatever command transparently.
> Good for me, since it is very close to my proposal (C) but just with
> different option names: (-i, -p) instead of (-m, -i)! I hope the other
> people who voted for (A) can accept this solution too.

To resolve all doubts on what is planned:
Is the planned difference between -i and -p the same as in Debian
systems apt-get/aptitude and dpkg ? If yes, I am fine with this solution.

[Jeroen Demeyer]

Well actually, there would a slight problem with *parallel* builds. If
you build packages A and B at the same time and A needs a confirmation,
then the on-screen output will look like

Are you sure you want to install package A (y/N)? Building package B...
output from building package B...
output from building package B...

[Volker Braun]

Apart from not spewing the entire log to stdout (can anybody read that fast?), we should re-print the prompt on keyboard input thats not "y" or "n"

[mmarco]

I think the solution for that would be to have all checks run before the actual build of any package.

That is:

1) the package manager figures out which packages should be installed, and in which (partial) order.

2) If any of them needs a user confirmation, ask for it

3) after the user has confirmed, start the build/install sequence for the packages.