Package installation/uninstallation confusion

[John H Palmieri]

I have been testing new versions of Simon King's p-group cohomology package. The current version doesn't work with Python 3, and he has been working (#28414) to fix this. My workflow:

1.  check out his git branch
2.  run ./sage -f -c p_group_cohomology
3.  report any issues
4.  check out develop branch

Here is the problem: if step 2 works (which it has been recently), then `build/make/Makefile` gets modified, adding p_group_cohomology to the list of installed packages. When I switch back to the develop branch, then (since I am using a Python 3 build of Sage), Sage doesn't build any more, because it tries to build the broken version of the p_group_cohomology package. So then I do

• "make p_group_cohomology-clean"

but this does not change `build/make/Makefile` and so it does not fix the problem. I can either edit the Makefile by hand or rerun ./configure. So here are some questions:

• Is "sage -i PKG" or "sage -f PKG" the preferred way to install an optional package? That's what our Developer's Guide says. Or should I be using "make PKG", which does not modify "build/make/Makefile"?
• Assuming I should keep using "sage -f PKG", should there be an analogous command which uninstalls and then rebuilds the Makefile? Or should "make PKG-clean" itself rebuild the Makefile?

--

John

[E. Madison Bray]

The generated file `build/make/Makefile` is output by the
`./configure` script. In fact, that's its main purpose. When
switching branches in this case the best thing to do is to re-run
`./configure`.

[Simon King]

On 2019-11-05, E. Madison Bray <erik....@gmail.com> wrote:
> The generated file `build/make/Makefile` is output by the
> `./configure` script. In fact, that's its main purpose. When
> switching branches in this case the best thing to do is to re-run
> `./configure`.

Concerning the original question how to uninstall stuff: In principle it
should be possible to cleanly uninstall p_group_cohomology, because it
consists of one autotoolized and one pip-installed sub-package.

However, I heard rumours that in order to make a Sage optional package
uninstallable, one needs some new script analogous to spkg-install.

Can someone give me a pointer on what should be done in that script and
what tools (sdh_*) are available for it?

Best regards,
Simon

[E. Madison Bray]

On Wed, Nov 6, 2019 at 7:58 AM Simon King <simon...@uni-jena.de> wrote:
>
> On 2019-11-05, E. Madison Bray <erik....@gmail.com> wrote:
> > The generated file `build/make/Makefile` is output by the
> > `./configure` script. In fact, that's its main purpose. When
> > switching branches in this case the best thing to do is to re-run
> > `./configure`.
>
> Concerning the original question how to uninstall stuff: In principle it
> should be possible to cleanly uninstall p_group_cohomology, because it
> consists of one autotoolized and one pip-installed sub-package.

That's not the issue here; the issue is that when you uninstall the
package files, some of the build system still needs to be reset. This
is indeed a problem, and there should be a better way to handle what
packages are or are not installed without necessarily having to re-run
configure, but I haven't given much thought to it.

> However, I heard rumours that in order to make a Sage optional package
> uninstallable, one needs some new script analogous to spkg-install.
>
> Can someone give me a pointer on what should be done in that script and
> what tools (sdh_*) are available for it?

That's not quite accurate. In general you do *not* need any
additional scripts, with some exception.
These packaging guidelines still need better documentation, but
essentially you need to make sure the package is built for what is
referred in GNU standards as a "DESTDIR install" [1]. This means that
the package is built for installation to some particular prefix (e.g.
`./configure --prefix=$SAGE_LOCAL` in autoconf terms), but can then be
*installed* with some additional path (the "DESTDIR") prepended to the
prefix. This is similar to providing an alternate root (e.g. prefix
is appended to DESTDIR instead of just "/").

This allows the package to be installed to a temporary "staging" area
that makes it easy, in an otherwise agnostic way, to see exactly what
files are going to be installed by the package so that we can generate
a manifest from it.

Any packaging using standard autoconf tools supports DESTDIR. Most
CMake packages do as well I think but I forget if there's something
special one has to do. Pip also supports this. Packages with weird
custom build systems may need to be patched or worked around in other
ways to support this, but fortunately there are not too many like that
in Sage anymore.

In some cases it is necessary to perform additional package
installation steps more complicated than just copying files. In this
case one can write an spkg-postinst script to perform such steps.
This is mentioned here [2]. However, if an "spkg-postinst" installs or
creates any files in $SAGE_LOCAL, although not necessarily required in
all cases it might be good to uninstall those files as well, in which
case one needs to write an "spkg-postrm" script. Normally this isn't
necessary though, and there are only a few examples of this in use.*

[1] https://www.gnu.org/prep/standards/html_node/DESTDIR.html
[2] https://doc.sagemath.org/html/en/developer/packaging.html#build-and-install-scripts

* One bug I am aware of in our packages is that there is a common file
"share/info/dir" which some packages that support "info" modify/append
to during installation. When using DESTDIR install this means a new
share/info/dir file gets created, and then copied over any existing
"share/info/dir". To handle this properly requires postinst/postrm
scripts which handle this infodir for any packages that use it. I had
a branch a long time ago that took care of this but it was part of a
larger patchbomb and never got merged I think. However, it's not a
high-priority issue; I don't think a lot of people are using `info` or
someone would have complained by now, maybe...

[Simon King]

Hi Eric,

On 2019-11-06, E. Madison Bray <erik....@gmail.com> wrote:
>> However, I heard rumours that in order to make a Sage optional package
>> uninstallable, one needs some new script analogous to spkg-install.
>>
>> Can someone give me a pointer on what should be done in that script and
>> what tools (sdh_*) are available for it?
>
> That's not quite accurate. In general you do *not* need any
> additional scripts, with some exception.
> These packaging guidelines still need better documentation, but
> essentially you need to make sure the package is built for what is
> referred in GNU standards as a "DESTDIR install" [1]. This means that
> the package is built for installation to some particular prefix (e.g.
> `./configure --prefix=$SAGE_LOCAL` in autoconf terms), but can then be
> *installed* with some additional path (the "DESTDIR") prepended to the
> prefix. This is similar to providing an alternate root (e.g. prefix
> is appended to DESTDIR instead of just "/").

Then what to do with p_group_cohomology? Its spkg-install script
installs two sub-packages. Each of them, I think, follows the above
scheme. However, Sage has no way of knowing that uninstalling
p_group_cohomology means uninstalling two sub-packages.

Perhaps one possibility is to split the current p_group_cohomology-3.3
into its two sub-packages, and make the first (which is autotoolized)
a dependency for the second (which is a bunch of cython and python
modules linked against the first sub-package.

Say:
- p_group_cohomology-3.4 shall only comprise what currently is the
p/cython part of p_group_cohomology-3.3, and have modular_resolution
as a dependency.
- modular_resolution-1.0 shall only comprise what currently is the
autotoolized c-library part of p_group_cohomology-3.3

Advantage: "make p_group_cohomology" would also install the dependency,
hence, both packages get installed. And both packages can also be
un-installed, I suppose.

Disadvantage: The only purpose of modular_resolution-1.0 would be to
serve as a dependency, it is (to my knowledge at least) of no
independent use.

Best regards,
Simon

[Dima Pasechnik]

sdh_install installs a json file into
SAGE_LOCAL/var/lib/sage/installed/
which are then used by unistallaller

e.g. "make meataxe" produces SAGE_LOCAL/var/lib/sage/installed/meataxe-1.0.p0
{
"package_name": "meataxe",
"package_version": "1.0.p0",
"install_date": "Wed Nov 6 17:24:57 GMT 2019",
"system_uname": "Linux hilbert 5.1.12-gentoo #1 SMP Fri Jun 21
19:27:38 BST 2019 x86_64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
GenuineIntel GNU/Linux",
"sage_version": "SageMath version 9.0.beta4, Release Date: 2019-11-02",
"test_result": "",
"files": [
"bin/cfcomp",
"bin/chop",
...
}

I guess sdh_pip_install does not do this, I don't know.

Anyhow, it would make perfect sense to split the package into two, as
you propose.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/qpuqe5%247598%241%40blaine.gmane.org.

[John H Palmieri]

Simon, you should look at the json file for p_group_cohomology to see if it contains all of the installed files, or if indeed some are not listed. If everything is there, there is no need to split it into two packages, unless I'm missing something.

On Wednesday, November 6, 2019 at 9:33:04 AM UTC-8, Dima Pasechnik wrote:

sdh_install installs a json file into
SAGE_LOCAL/var/lib/sage/installed/
which are then used by unistallaller

e.g. "make meataxe" produces SAGE_LOCAL/var/lib/sage/installed/meataxe-1.0.p0
{
    "package_name": "meataxe",
    "package_version": "1.0.p0",
    "install_date": "Wed Nov  6 17:24:57 GMT 2019",
    "system_uname": "Linux hilbert 5.1.12-gentoo #1 SMP Fri Jun 21
19:27:38 BST 2019 x86_64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
GenuineIntel GNU/Linux",
    "sage_version": "SageMath version 9.0.beta4, Release Date: 2019-11-02",
    "test_result": "",
    "files": [
        "bin/cfcomp",
        "bin/chop",
...
}

I guess sdh_pip_install does not do this, I don't know.

Anyhow, it would make perfect sense to split the package into two, as
you propose.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Dima Pasechnik]

On Wed, Nov 6, 2019 at 7:17 PM John H Palmieri <jhpalm...@gmail.com> wrote:
>
> Simon, you should look at the json file for p_group_cohomology to see if it contains all of the installed files, or if indeed some are not listed. If everything is there, there is no need to split it into two packages, unless I'm missing something.

if you look at its spkg-install, you'd see that shd_install is not
used at some place, instead

make install

is done, something that won't write any JSON files, IMHO...

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f30617cc-9e63-49b2-b097-b10682596c1b%40googlegroups.com.

[Simon King]

Hi Dima,

On 2019-11-06, Dima Pasechnik <dim...@gmail.com> wrote:
> if you look at its spkg-install, you'd see that shd_install is not
> used at some place, instead
>
> make install
>
> is done, something that won't write any JSON files, IMHO...

Good that I leave comments in my code...

The comment says that I used $MAKE install, because "sdh_make install
got broken by trac ticket #24106".

However, I was unaware that apparently "sdh_make install" ought to be
replaced by "sdh_install". So, I guess I should test this.

Or not? I see that far more packages use sdh_make_install (note the
difference: What got broken apparently was sdh_make install with a blank
space) than sdh_install.

Can you point me to man pages, where sdh_install versus
sdh_make_install is discussed?

Best regards,
Simon

[John H Palmieri]

These shell functions are defined and documented in SAGE_ROOT/build/bin/sage-dist-helpers. In my opinion they should also be documented in the developer's guide, but they don't seem to be.

[Simon King]

Hi John,

thank you!

So, I'll try sdh_make_install (sdh_install just copies some files), keep
using sdh_pip_install, and check what jason file results.

Best regards,
Simon

On 2019-11-07, John H Palmieri <jhpalm...@gmail.com> wrote:
> These shell functions are defined and documented in
> SAGE_ROOT/build/bin/sage-dist-helpers. In my opinion they should also be
> documented in the developer's guide, but they don't seem to be.
>
>
> On Thursday, November 7, 2019 at 6:35:05 AM UTC-8, Simon King wrote:
>>
>> Hi Dima,
>>

[Simon King]

Hi Dima,

On 2019-11-06, Dima Pasechnik <dim...@gmail.com> wrote:
> sdh_install installs a json file into
> SAGE_LOCAL/var/lib/sage/installed/
> which are then used by unistallaller

> ...
> I guess sdh_pip_install does not do this, I don't know.

No, I just checked: Those files of p_group_cohomology that got installed
by sdh_pip_install are listed in the json file. Those installed by
"$MAKE install" of course aren't.

> Anyhow, it would make perfect sense to split the package into two, as
> you propose.

If sdh_make_install and sdh_pip_install can share a single json file,
then there would be no need to split the package. I.e., the question is
whether "sdh_pip_install" will override the json file created by
"sdh_make_install", or whether it will append to it.

Best regards,
Simon

[E. Madison Bray]

On Wed, Nov 6, 2019 at 6:33 PM Dima Pasechnik <dim...@gmail.com> wrote:
>
> sdh_install installs a json file into
> SAGE_LOCAL/var/lib/sage/installed/
> which are then used by unistallaller
>
> e.g. "make meataxe" produces SAGE_LOCAL/var/lib/sage/installed/meataxe-1.0.p0
> {
> "package_name": "meataxe",
> "package_version": "1.0.p0",
> "install_date": "Wed Nov 6 17:24:57 GMT 2019",
> "system_uname": "Linux hilbert 5.1.12-gentoo #1 SMP Fri Jun 21
> 19:27:38 BST 2019 x86_64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
> GenuineIntel GNU/Linux",
> "sage_version": "SageMath version 9.0.beta4, Release Date: 2019-11-02",
> "test_result": "",
> "files": [
> "bin/cfcomp",
> "bin/chop",
> ...
> }
>
> I guess sdh_pip_install does not do this, I don't know.

It does. As I wrote in my previous message all SPKGs should be
installed using a DESTDIR scheme, and pip supports this (it's part of
what sdh_pip_install does--ensuring that the correct DESTDIR scheme is
used).

> Anyhow, it would make perfect sense to split the package into two, as
> you propose.

I'm not sure. It depends on what "package" means in this case. From
the perspective of Sage it shouldn't matter.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq22%3DK%2Bss9jhCj2L57iBOe5i9_5iUE9Kdg-ZQO_SxwM50Q%40mail.gmail.com.

[E. Madison Bray]

On Thu, Nov 7, 2019 at 6:26 PM Simon King <simon...@uni-jena.de> wrote:
>
> If sdh_make_install and sdh_pip_install can share a single json file,
> then there would be no need to split the package. I.e., the question is
> whether "sdh_pip_install" will override the json file created by
> "sdh_make_install", or whether it will append to it.

Just to clarify, once more (and please reread my earlier message in
this thread regarding what a "DESTDIR install" is and why that is
used), these commands do not write *anything* to any JSON file.

Their purpose is to wrap commands like "make" and "pip" with
additional flags that are needed for building/installing Sage SPKGs.
In the case of sdh_make_install, for example, as you can see in its
definition [1]. This allows sage-spkg, the script that actually
handles building/installing SPKGs, to look in the $SAGE_DESTDIR
directory that is created as a temporary installation root for the
package, and generate a list of files that is installed by the package
[2]. This happens after the package's spkg-install script has been
run, so this is entirely agnostic as to how the files got into
$SAGE_DESTDIR, whether it was via some `make install`, `pip install`,
`install`, `cp`, or some arbitrary combination of all of the above and
more.

I do believe this should be better documented and I don't appreciate
being told "read the code" anymore than anyone else does. But as
there is a lack of documentation currently that's the best thing one
can do rather than speculate.

Best,
Erik

[1] https://gitlab.com/sagemath/sage/blob/b90b558fc93ab78b4736d73535baf1f8053999df/build/bin/sage-dist-helpers#L190
[2] https://gitlab.com/sagemath/sage/blob/b90b558fc93ab78b4736d73535baf1f8053999df/build/bin/sage-spkg#L886

[Simon King]

Hi Erik,

On 2019-11-08, E. Madison Bray <erik....@gmail.com> wrote:
> Just to clarify, once more (and please reread my earlier message in
> this thread regarding what a "DESTDIR install" is and why that is
> used),

This does not clarify it to me at all, I'm afraid. To start with,
in my very simple mind, I can not intuitively grasp that an
spkg-install script can possibly support a DESTDIR install if the
word "DESTDIR" does not appear in the spkg-install script.

> ... This allows sage-spkg, the script that actually

> handles building/installing SPKGs, to look in the $SAGE_DESTDIR
> directory that is created as a temporary installation root for the
> package, and generate a list of files that is installed by the package
> [2].

Do I understand correctly that sage-spkg creates a temporary directory
and assigns the environment variable DESTDIR to it, then spkg-install
does its job, and *if* all installation steps done in spkg-install take
the DESTDIR environment variable into account then the installation
is correctly staged there before it is transferred from there to its
final destination? And the content of the json file is obtained from
the contents of the stage area before it is erased?

If that's the case then I'm confident that it should work to replace "$MAKE
install" with "sdh_make_install".

While we are at it: Currently, for building the package's documentation,
I use "$MAKE html" followed by some "cp -r" to copy the resulting html
files to $SAGE_SHARE/doc/p_group_cohomology. I guess this needs to be
fixed (in the sense of "rather use sdh_*) as well

I opened #28711 for it.

Best regards,
Simon

[John H Palmieri]

I created https://trac.sagemath.org/ticket/28721 to add documentation for these to the developer's guide. Needs review.

  John

[E. Madison Bray]

On Fri, Nov 8, 2019 at 11:06 PM Simon King <simon...@uni-jena.de> wrote:
>
> Hi Erik,
>
> On 2019-11-08, E. Madison Bray <erik....@gmail.com> wrote:
> > Just to clarify, once more (and please reread my earlier message in
> > this thread regarding what a "DESTDIR install" is and why that is
> > used),
>
> This does not clarify it to me at all, I'm afraid. To start with,
> in my very simple mind, I can not intuitively grasp that an
> spkg-install script can possibly support a DESTDIR install if the
> word "DESTDIR" does not appear in the spkg-install script.

I don't think you have a very simple mind; I agree it's not obvious.
But if you look in most spkg-install scripts you can see (as it
appears you have below) that there are helper functions with names
like "sdh_make_install", so I would think it would follow to look at
what those do and how they are implemented if it's apparent that
DESTDIR is not referenced explicitly in most spkg-install scripts.

> > ... This allows sage-spkg, the script that actually
> > handles building/installing SPKGs, to look in the $SAGE_DESTDIR
> > directory that is created as a temporary installation root for the
> > package, and generate a list of files that is installed by the package
> > [2].
>
> Do I understand correctly that sage-spkg creates a temporary directory
> and assigns the environment variable DESTDIR to it, then spkg-install
> does its job, and *if* all installation steps done in spkg-install take
> the DESTDIR environment variable into account then the installation
> is correctly staged there before it is transferred from there to its
> final destination? And the content of the json file is obtained from
> the contents of the stage area before it is erased?
>
> If that's the case then I'm confident that it should work to replace "$MAKE
> install" with "sdh_make_install".

Yep, you've understood correctly.

> While we are at it: Currently, for building the package's documentation,
> I use "$MAKE html" followed by some "cp -r" to copy the resulting html
> files to $SAGE_SHARE/doc/p_group_cohomology. I guess this needs to be
> fixed (in the sense of "rather use sdh_*) as well

You can also use sdh_install which is a (partial) replacement for the
"install" program, which does a few things in addition to just "cp -r"
and tries to take care of platform-specific differences, if any.

[Simon King]

Dear Erik,

Concerning "$MAKE html": Is "sdh_make html" the right thing, or did I
miss something? And I'll have a look into sdh_install.

Thank you!
Simon

[E. Madison Bray]

I don't know what `make html` does in this case so it's hard to say
for certain, but as you can see sdh_make doesn't do much other than
call $MAKE and then call sdh_die if it fails :)