installing optional packages requires running "sage -b"

[William A Stein]

Hi,

I installed Sage-6.3.beta6 on SageMathCloud, then installed a big list
of optional packages, including cryptominisat. I was surprised when
the next day a user reported that CryptoMiniSat didn't work, and when
you attempt to use it, you see

Run "install_package('cryptominisat')" to install it.

I tried the above (as the owner of the sage install), and it did not
fix the problem. What did fix the problem was typing "sage -b".
So... it seems like somebody removed doing "sage -b" after installing
optional packages from the build system (if it was ever there?).
There were several other Cython modules that also weren't built.

Thoughts? Obviously, doing "sage -b" after installing standard
packages wouldn't make sense, e.g, because "sage -b" wouldn't work.

-- William



--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
wst...@uw.edu

[William A Stein]

On Mon, Aug 4, 2014 at 6:17 AM, William A Stein <wst...@uw.edu> wrote:
> Hi,
>
> I installed Sage-6.3.beta6 on SageMathCloud, then installed a big list
> of optional packages, including cryptominisat. I was surprised when
> the next day a user reported that CryptoMiniSat didn't work, and when
> you attempt to use it, you see
>
> Run "install_package('cryptominisat')" to install it.
>
> I tried the above (as the owner of the sage install), and it did not
> fix the problem. What did fix the problem was typing "sage -b".
> So... it seems like somebody removed doing "sage -b" after installing
> optional packages from the build system (if it was ever there?).
> There were several other Cython modules that also weren't built.

Just to clarify -- this is exactly what wasn't built (not several
other modules):

Compiling sage/libs/coxeter3/coxeter.pyx because it changed.
Compiling sage/numerical/backends/coin_backend.pyx because it changed.
Compiling sage/sat/solvers/cryptominisat/cryptominisat.pyx because it changed.
Compiling sage/sat/solvers/cryptominisat/solverconf.pyx because it changed.

[Jeroen Demeyer]

On 2014-08-04 15:17, William A Stein wrote:
> Hi,
>
> I installed Sage-6.3.beta6 on SageMathCloud, then installed a big list
> of optional packages, including cryptominisat. I was surprised when
> the next day a user reported that CryptoMiniSat didn't work, and when
> you attempt to use it, you see
>
> Run "install_package('cryptominisat')" to install it.
>
> I tried the above (as the owner of the sage install), and it did not
> fix the problem. What did fix the problem was typing "sage -b".
> So... it seems like somebody removed doing "sage -b" after installing
> optional packages from the build system (if it was ever there?).

Running "sage -b" after installing packages has always been needed,
nobody removed anything.

I don't consider that a bug, it's consistent with: whenever you change
something to your Sage setup, run "sage -b" or even better "make".

I would not be in favour of automatically running "sage -b" since you
might want to install several packages but run "sage -b" only once, you
might want to do something else like checking out some git branch (after
which you need to do "sage -b" anyway) and it wouldn't work for standard
packages as you said. I would also advice against doing "sage -b" while
Sage is running, so the install_package() Sage command couldn't do it.

[William A Stein]

On Mon, Aug 4, 2014 at 6:30 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2014-08-04 15:17, William A Stein wrote:
>>
>> Hi,
>>
>> I installed Sage-6.3.beta6 on SageMathCloud, then installed a big list
>> of optional packages, including cryptominisat. I was surprised when
>> the next day a user reported that CryptoMiniSat didn't work, and when
>> you attempt to use it, you see
>>
>> Run "install_package('cryptominisat')" to install it.
>>
>> I tried the above (as the owner of the sage install), and it did not
>> fix the problem. What did fix the problem was typing "sage -b".
>> So... it seems like somebody removed doing "sage -b" after installing
>> optional packages from the build system (if it was ever there?).
>
> Running "sage -b" after installing packages has always been needed, nobody
> removed anything.
>

Of all optional packages (that work), it is apparently only needed
after 2 or 3 of them: cryptominisat, coin, coxeter.

> I don't consider that a bug, it's consistent with: whenever you change
> something to your Sage setup, run "sage -b" or even better "make".

Just consider what happens to a new confused user... or even the most
longterm user of Sage (me):

1. Install cryptominisat -- if you do
install_package('cryptominisat'), I think that's broken, so at least I
know to do:

sage -i cryptominisat

2. Start sage and try to run it. That's me doing this -- I never even
considered having to do "sage -b", and I've been using Sage for a
while.

sage: try function using package
boom

Run "install_package('cryptominisat')" to install it.

3. OK, I'll do what Sage *tells me to do*.

sage: install_package('cryptominisat')
boom!
ValueError: Package is already installed. Try
install_package('cryptominisat',force=True)

4. OK, I'll do what Sage tells me to do:

sage: install_package('cryptominisat', force=True)
ValueError: There is no package name starting with 'cryptominisat'.

5. Wow, I'm starting to think Sage is a confused broken mess? An old
forgetful sage... Oh wait, now I remember there was some bug in
install_package that I reported a few months ago, namely that
install_package, which you're told to use, is broken. So I do:

sage -f cryptominisat

and and go to 1. Nothing has been fixed at all.

This is absolutely not how Sage should behave. The fixes I can think of:

(1) Run "sage -b" as part of installing any package that requires it
for the install to finish. Your argument not to run "sage -b" isn't
convincing, because the user is explicitly installing a package
anyways, so they are potentially changing things all over Sage. Or,

(2) Tell people they have to run "sage -b" clearly in the package
install message. This won't work, because after 20 pages of build
output, nobody is likely to read this. Or,

(3) Instead of telling the user to reinstall the missing package (in
minisat say), actually give a useful error message, instead of a
totally wrong one. E.g., check to see if the minisat library is
installed, but the cython module isn't built. I don't like this
since it is fragile.


So (1) is the best solution I can think of, and solves all cases of
this problem for all packages that have it.

William

[Burcin Erocal]

On Mon, 4 Aug 2014 06:57:20 -0700

William A Stein <wst...@uw.edu> wrote:

<snip>

> This is absolutely not how Sage should behave. The fixes I can
> think of:
>
> (1) Run "sage -b" as part of installing any package that requires it
> for the install to finish. Your argument not to run "sage -b" isn't
> convincing, because the user is explicitly installing a package
> anyways, so they are potentially changing things all over Sage. Or,
>
> (2) Tell people they have to run "sage -b" clearly in the package
> install message. This won't work, because after 20 pages of build
> output, nobody is likely to read this. Or,
>
> (3) Instead of telling the user to reinstall the missing package (in
> minisat say), actually give a useful error message, instead of a
> totally wrong one. E.g., check to see if the minisat library is
> installed, but the cython module isn't built. I don't like this
> since it is fragile.
>
>
> So (1) is the best solution I can think of, and solves all cases of
> this problem for all packages that have it.

There is one more option:

(4) include the relevant python/cython code in the package, build
and install it with the package.

This thread is relevant, even though I misinterpreted the original
question and dragged it in this direction:

https://groups.google.com/d/topic/sage-devel/meBnEvuBang/discussion

> >
> > I would not be in favour of automatically running "sage -b" since
> > you might want to install several packages but run "sage -b" only
> > once, you might want to do something else like checking out some
> > git branch (after which you need to do "sage -b" anyway) and it
> > wouldn't work for standard packages as you said. I would also
> > advice against doing "sage -b" while Sage is running, so the
> > install_package() Sage command couldn't do it.

+1.

Running "sage -b" does not guarantee that the module will be built.
Sage source tree might be in a broken state due to user modifications.


Cheers,
Burcin

[Jeroen Demeyer]

On 2014-08-04 15:57, William A Stein wrote:
> (2) Tell people they have to run "sage -b" clearly in the package
> install message. This won't work, because after 20 pages of build
> output, nobody is likely to read this.

How about mentioning this after the build finished? Would that be more
likely to catch the attention?

[Jeroen Demeyer]

On 2014-08-04 16:35, Burcin Erocal wrote:
> There is one more option:
>
> (4) include the relevant python/cython code in the package, build
> and install it with the package.

That's a possibility, although it complicates doctesting: that extra
code wouldn't get doctested.

[William A Stein]

On Mon, Aug 4, 2014 at 8:14 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2014-08-04 16:35, Burcin Erocal wrote:
>>
>> There is one more option:
>>
>> (4) include the relevant python/cython code in the package, build
>> and install it with the package.

I like (4) the best, but it requires a significant change (compared to
1-3, which are each trivial to implement).

> That's a possibility, although it complicates doctesting: that extra code
> wouldn't get doctested.

Does anybody actually doctest code in the Sage library that uses
optional packages? I wrote the "#optional" tags so that this would be
possible, but never really kept up with it.

The install process for the optional package can at least run a test
suite, when the SAGE_CHECK variable is set. This could run the
corresponding doctests, which might be in the package.

[Volker Braun]

(5) have any dependency handling for optional packages. 

Really, this thread boils down to: the Sage library depends on some optional packages. Any kind of hacks around that (like running "sage -b" by hand in scripts) will just lead to race conditions.

[William A Stein]

On Mon, Aug 4, 2014 at 8:35 AM, Volker Braun <vbrau...@gmail.com> wrote:
> (5) have any dependency handling for optional packages.

Would switching to that package manager that you recently rewrote help here?

>
> Really, this thread boils down to: the Sage library depends on some optional
> packages. Any kind of hacks around that (like running "sage -b" by hand in
> scripts) will just lead to race conditions.

I'm going to give up on changing sage for this particular issue, and
just put "sage -b" in my own script for installing optional packages
to SMC.

Probably somebody should disable the install_package command in Sage,
and remove explicit reference to it in sage source code, if there is
no plan to fix it. There are many places in the sage source code that
tell people to use that command:

~/tmp$ sage -grep "install_package"
sage/sat/solvers/cryptominisat/__init__.py: raise
ImportError("Failed to import
'sage.sat.solvers.cryptominisat.CryptoMiniSat'. Run
\"install_package('cryptominisat')\" to install it.")
sage/misc/package.py:Use the ``install_package`` command to install a new
sage/misc/package.py: install_package(pkg, force=force)
sage/misc/package.py:def install_package(package=None, force=False):
sage/misc/package.py: sage: install_package()
sage/misc/package.py: sage: install_package('chomp') # not tested
sage/misc/package.py: print("install_package may not work
correctly under Microsoft Windows")
sage/misc/package.py: print("use the force option to install_package().")
sage/misc/package.py: raise ValueError("Package is
already installed. Try
install_package('{}',force=True)".format(package))
sage/misc/package.py: return any(p.startswith(package) for p in
install_package())
sage/misc/package.py: Use ``install_package(package_name)`` to install or
sage/misc/package.py: .. seealso:: :func:`install_package`, :func:`upgrade`
sage/misc/package.py: Use ``install_package(package_name)`` to install or
sage/misc/package.py: .. seealso:: :func:`install_package`, :func:`upgrade`
sage/misc/package.py: Use ``install_package(package_name)`` to install or
sage/misc/package.py: .. seealso:: :func:`install_package`, :func:`upgrade`
sage/misc/package.py: .. seealso:: :func:`install_package`,
:func:`optional_packages`
sage/misc/package.py: mesg = 'To install the package %s type
install_package("%s")\n'%(package_name, package_name)
sage/misc/all.py:from package import install_package,
is_package_installed, standard_packages, optional_packages,
experimental_packages, upgrade
sage/interfaces/r.py: def install_packages(self, package_name):
sage/interfaces/r.py: sage: r.install_packages('aaMI')
# not tested
sage/rings/polynomial/multi_polynomial_ideal.py: raise
ImportError("""GINV is missing, to install use
"install_package('ginv')".""")
sage/rings/polynomial/multi_polynomial_sequence.py:
raise ValueError('algorithm=exhaustive_search requires the optional
library FES. Run "install_package(\'fes\')" to install it
.')
sage/groups/perm_gps/permgroup_named.py:
``install_package(`database_gap')``.
sage/geometry/triangulation/point_configuration.py: sage:
install_package('TOPCOM') # not tested
sage/geometry/polyhedron/palp_database.py: install it with the
command ``install_package('polytopes_db_4d')``.
sage/graphs/graph_plot_js.py: ``install_package('d3js')`` or by
running ``sage -i d3js`` from the command
sage/graphs/graph_plot_js.py: ``install_package('d3js')`` or by
running ``sage -i d3js`` from the
sage/server/trac/trac.py: print "Try something like
install_package('trac-0.11.5'),"
~/tmp$ sage -grep "install_package"|wc -l
29



-- William

>
>
>
> On Monday, August 4, 2014 2:58:02 PM UTC+1, wstein wrote:
>>
>> (1) Run "sage -b" as part of installing any package that requires it
>> for the install to finish. Your argument not to run "sage -b" isn't
>> convincing, because the user is explicitly installing a package
>> anyways, so they are potentially changing things all over Sage. Or,
>>
>> (2) Tell people they have to run "sage -b" clearly in the package
>> install message. This won't work, because after 20 pages of build
>> output, nobody is likely to read this. Or,
>>
>> (3) Instead of telling the user to reinstall the missing package (in
>> minisat say), actually give a useful error message, instead of a
>> totally wrong one. E.g., check to see if the minisat library is
>> installed, but the cython module isn't built. I don't like this
>> since it is fragile.
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Jeroen Demeyer]

On 2014-08-04 18:07, William A Stein wrote:
> I'm going to give up on changing sage for this particular issue, and
> just put "sage -b" in my own script for installing optional packages
> to SMC.

I think it's even safer to use "make" instead, which not only runs "sage
-b" but also rebuilds the documentation and builds other packages if needed.

[Jeroen Demeyer]

On 2014-08-04 18:07, William A Stein wrote:

> Probably somebody should disable the install_package command in Sage,
> and remove explicit reference to it in sage source code, if there is
> no plan to fix it.

The plan to fix it is now
http://trac.sagemath.org/ticket/16759

[Volker Braun]

IMHO we should try to switch to hashdist for Sage-7.0. This would solve our dependency problems...

[kcrisman]

Does anybody actually doctest code in the Sage library that uses
optional packages?  I wrote the "#optional" tags so that this would be
possible, but never really kept up with it.

Yes, though not as often as one might like.  There are plenty of tickets where (eventually) bugs have been discovered (and sometimes fixed!) because of this.

 

The install process for the optional package can at least run a test
suite, when the SAGE_CHECK variable is set.  This could run the
corresponding doctests, which might be in the package.

This sounds good. 

But I agree with William that this is important (including the `sage -b` issue).  I'm working at a Sage Days prep right now with exactly the kind of people William was talking about in the thread that led to this - people who know quite a bit of math and might want to use some specialized optional package (say, polymake) but would need an awful lot of hand-holding to make it through some of this weird behavior.  Unless you want to change the mission statement to say "viable open-source alternative that requires people know about build systems", we should not assume that everyone using Sage is all up on this kind of stuff.  I just worked with someone who is doing K-L polynomials, I believe, but definitely has been having a heroic struggle to get Sage to work on a pretty standard Ubuntu laptop (to be fair, partly because of network issues).  But this is not the usual "send them to the cloud/sagenb" customer who perhaps can ignore stuff like this.

As to `sage -b`, IIRC this comes up with a number of packages; would it be possible to put the warning about `sage -b` at the BEGINNING of the installation script messages? Something like

"When this finishes, you may still need to rebuild Sage with sage -b" or the like.  Also, how would that work with people with binaries and do not have the toolchain available?  Or is gcc and friends required for using sage -i?

- kcrisman

[Jeroen Demeyer]

On 2014-08-05 02:04, kcrisman wrote:
> Or is gcc and friends required for using sage -i?

With a few exceptions (like pure database or pure Python packages), yes.
Usually installing packages requires some code to be compiled.