sage.misc.package considered harmful

[François Bissey]

Hi all,

This is a rare rant cross-posted to sage-packaging about
one of the annoyance faced by the people packaging sage
and not using sage's packaging system.

The functions found in `sage.misc.package` are a result
of the fact that sagemath and its development is closely
linked to its own packaging system.

Stuff from `sage.misc.package` is used in one of three
things:

1) Runtime: testing for the presence or finding info about
a package. Usually by calling "is_package_installed".
Most of this can be all done a different way and will
hopefully be addressed by
https://trac.sagemath.org/ticket/20382

2) Doctest time: currently testing with "-optional=all" relies
on `sage.misc.package` to establish the list of options to be
tested.

3) Build time: it is used to figure if some optional binding
can be build. The way it is done is generally called "automagic",
that is build if detected. This would be better replaced by
some configuration tool.

At the moment only replacement for (1) is actively worked on.
This is the most harmful bit from the distro point of view.
(3) can be dealt with in an "ad hoc" way.
(2) is probably the toughest problem.

In the meantime what I would like to discuss is that from my
point of view we shouldn't be introducing new runtime call
to sage.misc.package if it can be easily avoided before #20382
gets finalized.

I am starting this debate because of discussion I had earlier
in #22670. It was pointed out to me that there was no policy
of avoiding `sage.misc.package` and I would very much want one.


Francois
sage-on-gentoo dev.

[kcrisman]

I am starting this debate because of discussion I had earlier
in #22670. It was pointed out to me that there was no policy
of avoiding `sage.misc.package` and I would very much want one.

This seems very reasonable, given how much work is going into such packaging.  As a first approximation for 2) (we don't want to give up that auto-testing) - is there any easy way of auto-disabling the auto-discovery of installed packages on "distro" Sage versions as opposed to Sage-the-distribution?  (Is there an easy to way to test for which situation one is in?) That would obviously be a hack, but maybe a better hack than the status quo, I truly don't know.

[Francois Bissey]

(2) while being just one use, is probably not replaceable.
Not in this form at the very least.

François

[Nils Bruin]

On Tuesday, April 4, 2017 at 1:00:31 PM UTC-7, François wrote:

(2) while being just one use, is probably not replaceable.
Not in this form at the very least.

How can one get the appropriate information in sage-on-gentoo? If we compare the mechanisms that work in the different scenarios we might be able to see a common strategy or a way to choose which strategy should be followed. On the other hand, if the information about which optional packages are installed, is fundamentally not available then it is futile to try and make --optional=all work at all.

[Francois Bissey]

I currently do not support “—optional=all” in
sage-on-gentoo. It is indeed futile.
Let’s be clear, I could ship a list of possible
optional packages supported in sage-on-gentoo
but any checking of package availability would
have to go through the the distribution package
manager.

Effectively each distro would need to bring in their
plugin. Another thing, currently “—optional=all” check
that not only it is installed but that the version
is the latest one available to sage.
For many binary distro if you are up to date such check
is superfluous (you may not have the current sage version
but that’s what you’ll get from your package manager).

And we don’t even cover the fact that some options may
just be installed system wide (by package manager or
otherwise) and would just work. Apart from key proprietary
software (mathematica, magma…) that case is not covered
by the current framework either. If it doesn’t exist
in the sage sandbox, it doesn’t exist as far as the testing
framework is concerned. And actually that notion of control
and confinement that has to be maintained is probably getting
in the way of thinking of a more general solution.

From sage-on-gentoo point of view having “—optional==all”
working would be nice for the optional packages I support.
But I don’t explicitly support that many anyway. Not sure
about the other packagers.

François

[Nils Bruin]

On Tuesday, April 4, 2017 at 2:24:52 PM UTC-7, François wrote:

Let’s be clear, I could ship a list of possible
optional packages supported in sage-on-gentoo
but any checking of package availability would
have to go through the the distribution package
manager.

Or through the sage doctesting framework. It would probably be quite doable for each known package to write a doctest that succeeds if a certain package is installed (to some reasonable extent) and fails if it is not. Hence, you could just query the sage runtime to see if certain packages seem to be present. The biggest problem with this approach is that "installed but really badly broken" would not be distinguished from "not installed".

[Francois Bissey]

The current system doesn’t care about broken either.
In fact I’d argue that the approach you describe, which
could possibly be hooked with the “feature” system developed
for runtime use, would be better at spotting brokenness than
the current one.

With the current system you could install and then remove
some essential files manually and the doctesting framework
would still try to use it. It is installed according to the
packaging system after all. runtime testing would have a better
chance to avoid broken packages.

François

[Nils Bruin]

On Tuesday, April 4, 2017 at 4:01:50 PM UTC-7, François wrote:

With the current system you could install and then remove
some essential files manually and the doctesting framework
would still try to use it. It is installed according to the
packaging system after all. runtime testing would have a better
chance to avoid broken packages.

Hm, I don't see that as an advantage. A broken package shouldn't be just avoided, it should be reported, so that it can be repaired or uninstalled. Just silently avoiding broken functionality makes it harder to diagnose.

[Francois Bissey]

Ha, I was thinking of a different kind of broken. I was thinking
of user mistake, you are thinking of systemic problems.

I guess the doctesting framework could output a list of software
not tested because it thinks it is absent or not working.
A discrepancy could be spotted but that is not as direct, I agree.

François

[Isuru Fernando]

> I welcome the work on the Conda port because it will allow just
that in the medium term. You could replace the whole sage packaging
with Conda and a few scripts to set things up. Once you do that
sage.misc.package should die (and please do not replace it by calls
to Conda).

FYI, I have a branch here, https://github.com/isuruf/sage/commits/conda2 that installs dependencies using conda if it exists and otherwise uses spkg-install. What it does is it'll install the dependencies using conda if it finds a file named conda-pkg-name in `build/pkgs/<pkg>/`. This is a hack, and it'll take a lot of work to replace the sage package manager with conda.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscribe@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Nicolas M. Thiery]

On Mon, Apr 03, 2017 at 05:48:30PM +1200, François Bissey wrote:
> I am starting this debate because of discussion I had earlier
> in #22670. It was pointed out to me that there was no policy
> of avoiding `sage.misc.package` and I would very much want one.

From following the many discussions on packaging, I am +1 on such a
policy.

Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/