distro packaging, docbuilding and modularisation

[François Bissey]

Hi all,

So today I will be talking about subject that I in sage-on-gentoo have
brushed under the carpet for years focusing on helping to be able to
use more and more components of the system and ultimately get “sagelib”
to be one such component.

Sagemath as a meta distribution of stuff has bad habits in regards to
treating “sagelib” as a regular package. In short it doesn’t.

A normal software package usually should only contains it own sources,
vendoring as it is called should be kept at a minimum and some other
best practise are that
* the package can be tested after being built and before being installed
* the package documentation be built usually after the package itself,
but before it is installed and the documentation should be installed
at the same time as the rest of the package.

Some project offer pre-build documentation, as part of the package if
it is small, or downloadable tarballs if it is large. It is also
the way to go for particularly difficult to build documentation.
I’d say Sagemath will eventually go that way.

So, we have made progress on vendoring, sage-on-gentoo has been building
Sagelib and the documentation as a regular well behaved package for years
and I have only pestered about the testing. Modularisation may force some
things about the testing and offer solutions to an unmentioned problem
so far, optional dependencies.

But my situation about being able to build the documentation is under
threat and is hack-ish in any case.

So how are we building, installing and testing the elements of “vanilla" sage
at the moment:
* sagelib is built and installed
* using the sagelib install the documentation is built. It is not supposed to be
possible to build the doc without installing sage first, which is why sage-on-gentoo’s
ability is really a hack
* testing is done on the installed elements, that includes running tests inside the documentation

Instead of trying to fix the problem that you should be building the documentation
before installing, the push to modularisation is currently used to enshrine the
current situation. #30010 actually separate the bits needed to build the sage
documentation in a completely separate package.
So the situation is now:
* install package A
* install package B
* make a package C that takes B and applies it to the install of A and install
the documentation of A (not C).

Can anyone point me to some other system that do that? Does anyone else thinks
that’s a bit insane?

#30010 is not all bad. Doc building elements have been buried inside sage_setup
for a while and that wasn’t a good thing. sage_setup is mostly useless at
runtime and shouldn’t really be installed unless some downstream packages can
make use of it. But the doc building elements are already used by downstream
packages (p_cohomology_group does), so installing them has some value.

My own idea of modularisation would be that each package should be able to build
its own documentation and be testable on its own, before install. Not to split
the whole documentation as a separate package.

I want to repeat #30010 is not without value. Having a helper package to build
the documentation is not a bad idea in itself so long as it applies in the following
order:
* install B (the doc building helper)
* build A, build A’s documentation using B
* install A and its documentation
No C please.

If you read this far, thank you. That’s long enough for such a rant.

François

[Dima Pasechnik]

numpy does this:

https://numpy.org/devdocs/docs/howto_build_docs.html

you can only build numpy docs after numpy is installed.

#30010 is not all bad. Doc building elements have been buried inside sage_setup
for a while and that wasn’t a good thing. sage_setup is mostly useless at
runtime and shouldn’t really be installed unless some downstream packages can
make use of it. But the doc building elements are already used by downstream
packages (p_cohomology_group does), so installing them has some value.

My own idea of modularisation would be that each package should be able to build
its own documentation and be testable on its own, before install. Not to split
the whole documentation as a separate package.

I want to repeat #30010 is not without value. Having a helper package to build
the documentation is not a bad idea in itself so long as it applies in the following
order:
* install B (the doc building helper)
* build A, build A’s documentation using B
* install A and its documentation
No C please.

If you read this far, thank you. That’s long enough for such a rant.

François

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/BE94013F-0363-4E78-94AD-93390E8669EA%40gmail.com.

[Antonio Rojas]

El miércoles, 10 de marzo de 2021 a las 10:21:07 UTC+1, François Bissey escribió:

Instead of trying to fix the problem that you should be building the documentation
before installing, the push to modularisation is currently used to enshrine the
current situation. #30010 actually separate the bits needed to build the sage
documentation in a completely separate package.
So the situation is now:
* install package A
* install package B
* make a package C that takes B and applies it to the install of A and install
the documentation of A (not C).

Hi,

 As long as everything is still shipped in the Sage source tarball, isn't it just a matter of changing a path in the build script? I've always run the doc build directly from the Sage source, without installing anything, and I hope to be able to keep doing it that way (after a quick test, it still seems to be possible after #30010). Of course, if the long term plan is to split this to a separate git repo, then I completely share your concerns. 

[Matthias Koeppe]

On Wednesday, March 10, 2021 at 1:21:07 AM UTC-8 François Bissey wrote:

[...] building the documentation [...] 

In https://trac.sagemath.org/ticket/29868 I sketch a design that is in line with modern Python packaging, in particular build isolation. I welcome discussions based on this.

[Matthias Koeppe]

On Wednesday, March 10, 2021 at 4:10:19 AM UTC-8 Antonio Rojas wrote:

 Of course, if the long term plan is to split this to a separate git repo, then I completely share your concerns. 

No, the plan for modularization does not involve splitting the repository. See https://trac.sagemath.org/ticket/29705 for an overview.

 

[Matthias Koeppe]

On Wednesday, March 10, 2021 at 1:21:07 AM UTC-8 François Bissey wrote:

Sagemath as a meta distribution of stuff has bad habits in regards to
treating “sagelib” as a regular package. In short it doesn’t.

That's no longer true.

sagelib is now a regular Python distribution called "sagemath-standard".

An sdist can be built using the script build/pkgs/sagelib/spkg-src

(in fact, the sdist is already available at https://pypi.org/project/sagemath-standard/)

(As of 9.3.beta8, it is still missing Python dependency metadata. This is rectified by https://trac.sagemath.org/ticket/30913, which is waiting for review...)

You can update your downstream distribution packaging so it installs sagelib like any other Python package.

[Matthias Koeppe]

On Wednesday, March 10, 2021 at 4:10:19 AM UTC-8 Antonio Rojas wrote:

 As long as everything is still shipped in the Sage source tarball, isn't it just a matter of changing a path in the build script?

That's right. As far as changes in src/ are concerned, what #30010 did was move src/sage_setup/setup_docbuild to src/sage_docbuild.

The renamed modules sage_docbuild.* can just be imported without installing anything.

In fact, this is a key design principle of the modularization approach of https://trac.sagemath.org/ticket/29705: The overall structure of the src tree is not changed. If you want to ignore questions of Python packaging, you can and just look at src/ as a monolithic source tree containing the top-level package directories sage, sage_setup, sage_docbuild.

I've always run the doc build directly from the Sage source, without installing anything, and I hope to be able to keep doing it that way (after a quick test, it still seems to be possible after #30010).

Absolutely.

 

[François Bissey]

> On 10/03/2021, at 23:50, Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Wed, 10 Mar 2021, 09:21 François Bissey, <frp.b...@gmail.com> wrote:
> So the situation is now:
> * install package A
> * install package B
> * make a package C that takes B and applies it to the install of A and install
> the documentation of A (not C).
>
> Can anyone point me to some other system that do that? Does anyone else thinks
> that’s a bit insane?
>
> numpy does this:
> https://numpy.org/devdocs/docs/howto_build_docs.html
>
> you can only build numpy docs after numpy is installed.
>

And Gentoo doesn’t even try to build the doc we use their provided tarball.

As far as I see it, the way sage sage is going I won’t build or test the documentation
anymore and rely on you producing tarballs. Which means there won’t be version bump
of sage in Gentoo until those exists when that happens.

François

[François Bissey]

I’d like to talk to you on another channel, I have to do a small
patch currently. It may reflect different packaging strategy or
something I could improve.

François

[François Bissey]

build isolation is nice. If python packaging says that what numpy does
and sage is about to do is right, they are completely diverging from
traditional packaging. May be packagers have been wrong all along.

In a way, it mimics the way commercial closed packages tend to behave and
be distributed. And I don’t like that imitation one bit. It rubs me the
wrong way. Reading myself, I hope I am not turning into a “Richard Stallman”
but I am certainly opinionated and not afraid of it.

François

[Nathan Dunfield]

On Wednesday, March 10, 2021 at 4:50:41 AM UTC-6 Dima wrote:

numpy does this:

https://numpy.org/devdocs/docs/howto_build_docs.html

you can only build numpy docs after numpy is installed.

Of course, with numpy "installed" doesn't necessarily mean installed in the main site-packages, you can use a virtualenv.  Then you delete the virtualenv once the docs are built and install the module and the docs for real.  It's hard to avoid something like this if using Sphinx's autodoc features that actually import the Python module and build the docs from the docstrings it finds.

Sage is likely too complicated for this to work, but you can skip the virtualenv step by having Sphinx work off the copy of the in "build/lib.macosx-10.9-x86_64-3.9" or similar.  This is the trick we use with SnapPy:  https://github.com/3-manifolds/SnapPy/blob/master/doc_src/conf.py

Nathan

[Antonio Rojas]

El miércoles, 10 de marzo de 2021 a las 21:13:15 UTC+1, François Bissey escribió:

I’d like to talk to you on another channel, I have to do a small
patch currently. It may reflect different packaging strategy or
something I could improve.

Here is my current PKGBUILD (after some cleanup I did today) https://github.com/archlinux/svntogit-community/blob/packages/sagemath-doc/trunk/PKGBUILD

I'm building the docs in a separate package for practical purposes (sage requires frequent rebuilds for soname bumps in dependencies, and I don't want to have to build the docs every time), hence the 'rm -r src/sage' line so that it doesn't pick up the uncompiled sage module from the source tree, this shouldn't be necessary if you build sagelibs and the docs together.

After #30010, simply changing sage_setup.docbuild to sage_docbuild works for me. 

[Matthias Koeppe]

On Thursday, March 11, 2021 at 11:36:18 AM UTC-8 Antonio Rojas wrote:

Here is my current PKGBUILD (after some cleanup I did today) https://github.com/archlinux/svntogit-community/blob/packages/sagemath-doc/trunk/PKGBUILD

I'm building the docs in a separate package for practical purposes (sage requires frequent rebuilds for soname bumps in dependencies, and I don't want to have to build the docs every time) [...]

Yes, this is an important insight. Sage documentation and Sage library have sharply different dependencies - both at build time and at installation time. The design in https://trac.sagemath.org/ticket/29868 models the Python package dependencies (PEP 518 build system requirements, and install requires) to reflect this. 

[Matthias Koeppe]

On Wednesday, March 10, 2021 at 8:18:11 PM UTC-8 Nathan Dunfield wrote:

On Wednesday, March 10, 2021 at 4:50:41 AM UTC-6 Dima wrote:

numpy does this:

https://numpy.org/devdocs/docs/howto_build_docs.html

you can only build numpy docs after numpy is installed.

Of course, with numpy "installed" doesn't necessarily mean installed in the main site-packages, you can use a virtualenv.  Then you delete the virtualenv once the docs are built and install the module and the docs for real.  

Just a quick comment that you get this type of build isolation for free if you make the documentation a separate distribution according to PEP 517 (pyproject.toml), for which you declare the package as a build-system requirement.

[François Bissey]

For the record, I used to have a separate package for documentation years ago
and I walked away from that. Source distributions have different set of parameters
to play with.

That being said, building the sage documentation is a relatively heavy task so for
releases I build a documentation set that can be used instead of building it from
scratch. That set doesn’t change even if I revise the package one way or another
(unless I fix a documentation specific issue).

For beta and rc release, my expectation is that if you want documentation you’ll have
to build it. If you follow beta and rc I am expecting you want the “fast” paced changes
and to have the capacity for a doc build if you wish it.

I personally do CI from Volker merge tree with my install, so I build and test as
things gets merged, that includes the documentation. If doing the documentation
gets too difficult to put in the framework I use, I will just ditch it in favour to
whatever version is last available upstream. No more testing of that from me.

[François Bissey]

I have been wondering for a while why you want to have those kinds of requirements.
In the autotools world there is no issues with building documentation as part of the
building process even if, potentially like sage, you want some content to be dynamically
generated by executing the just built package.

And that’s when it hits me. In a lot of python packages, building the documentation,
when it is available, a different build system is used than when you where
building the package itself.
In a typical case, you build with distutils/setuptools and then invoke make to start
sphinx to build the doc.

The python build systems in wide use have no hooks or provision to latch on to stuff
you use to build the documentation. If they do they are not in wide use, or may be
too primitive.

In that context, since you often use a completely different way of building stuff,
those PEPs literally mandate a separate package. The only alternative is to include
provision for documentation building in stuff like setuptools. I don’t see that happening
anytime soon.

[Matthias Koeppe]

On Thursday, March 11, 2021 at 1:28:12 PM UTC-8 François Bissey wrote:

> On 12/03/2021, at 09:46, Matthias Koeppe <matthia...@gmail.com> wrote:
>  you get this type of build isolation for free if you make the documentation a separate distribution according to PEP 517 (pyproject.toml), for which you declare the package as a build-system requirement.

I have been wondering for a while why you want to have those kinds of requirements.
In the autotools world there is no issues with building documentation as part of the
building process even if, potentially like sage, you want some content to be dynamically
generated by executing the just built package.

Well, autotools does not even have a mechanism to advertise dependencies/requirements -- other than "configure" exiting with an error if they are not satisfied. You as a downstream package maintainer declare and update the requirements based on (in the best case) package documentation.

 

And that’s when it hits me. In a lot of python packages, building the documentation,
when it is available, a different build system is used than when you where
building the package itself.
In a typical case, you build with distutils/setuptools and then invoke make to start

sphinx to build the doc. [...] since you often use a completely different way of building stuff,

those PEPs literally mandate a separate package.

Yes, pretty much. In the past (pre PEP-517/518) one could get away with treating docbuild etc. as something depending on "optional dependencies", like using Sphinx if Sphinx happens to be installed. This is no longer a thing in the much more structured modern Python build system world. One gains reproducible builds and overall (hopefully) more robustness.

And then -- of course likely not relevant for gentoo -- there is also the distinction between source and binary distributions (wheels): For the package itself, many users have a good reason to build it from source; but for a separate documentation package, most users do not really benefit from building it from source - they can as well install a prebuilt wheel providing the documentation. These users will then not have a need to install Sphinx at all.

[François Bissey]

> On 12/03/2021, at 11:20, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> And then -- of course likely not relevant for gentoo -- there is also the distinction between source and binary distributions (wheels): For the package itself, many users have a good reason to build it from source; but for a separate documentation package, most users do not really benefit from building it from source - they can as well install a prebuilt wheel providing the documentation. These users will then not have a need to install Sphinx at all.

I guess it is my fault for doing CI, including doc, straight from git.
Some may say I am abusing the package manager in doing so. But someone
really put some thoughts in making the tools available for it. If there wasn’t
a couple more kinks here and there, I could test branches attached to tickets
directly from it.
I’d really like to continue to do stuff the way I have been doing them.
But if they have to go extinct, that’ll just be that.

[Matthias Koeppe]

On Thursday, March 11, 2021 at 2:33:18 PM UTC-8 François Bissey wrote:

I guess it is my fault for doing CI, including doc, straight from git.

No, there's nothing wrong with it; this is the normal way to use the repository now and in the planned future. 

After running ./bootstrap, the source tree contains several self-contained Python distribution package source trees in build/pkgs/*/src/, which can be built and installed normally (setup.py install, setup.py bdist_wheel, etc.). Right now * is "sagelib", "sage_docbuild", and "sage_sws2rst". Modularization will add more.

https://trac.sagemath.org/ticket/29868 will add "sagemath_doc_html" -- your custom script to build the documentation will reduce to running "setup.py install" for this package.

[François Bissey]

> On 12/03/2021, at 11:20, Matthias Koeppe <matthia...@gmail.com> wrote:
>

> Well, autotools does not even have a mechanism to advertise dependencies/requirements -- other than "configure" exiting with an error if they are not satisfied. You as a downstream package maintainer declare and update the requirements based on (in the best case) package documentation.
>

Yup, it is just a build system. Languages that come with package management
are a relatively new in a way [OK, R had one for a long time, I am sure there
are others].
It took some time for it to mature to the present state in python.

[François Bissey]

> On 12/03/2021, at 11:48, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> No, there's nothing wrong with it; this is the normal way to use the repository now and in the planned future.
> After running ./bootstrap, the source tree contains several self-contained Python distribution package source trees in build/pkgs/*/src/, which can be built and installed normally (setup.py install, setup.py bdist_wheel, etc.). Right now * is "sagelib", "sage_docbuild", and "sage_sws2rst". Modularization will add more.
> https://trac.sagemath.org/ticket/29868 will add "sagemath_doc_html" -- your custom script to build the documentation will reduce to running "setup.py install" for this package.
>

When are we going to start seeing separate sdist tarballs? That’s really the point at which
I am going to switch the way I do things.

[Matthias Koeppe]

this one? https://pypi.org/project/sagemath-standard/#files

 

[François Bissey]

Even for betas? I was thinking there would only be some for proper releases.
Well, I should get ready to use that stuff directly rather than the sage
git tarball with stuff that I do not use and that requires special settings
to figure out where the sources are located.
9.3 may very well be my last “classic" release of sage. Although I may still
consider to have a special branch to do the work I currently do directly
on git for a while.

[E. Madison Bray]

You can also run `./setup.py develop` to have all extension modules
copied into the main package source, so that it can be directly
importable. Then in Sphinx's conf.py you put something like (in
sage's case):

sys.path.insert(0, '../src')

and now sage is importable from there without having to be "installed".

I have suggested this in the past over the way Sage has often been
developed by always re-installing the package into site-packages
before testing changes, but some people have been hostile to this out
of a distaste of having build artifacts cluttering the source tree.
And I sympathize with that distaste, though personally I don't mind
it. I've argued in the past that both styles can work simultaneously
without much if any compromise.

[Matthias Koeppe]

On Friday, March 12, 2021 at 2:38:31 AM UTC-8 erik....@gmail.com wrote:

You can also run `./setup.py develop` to have all extension modules
copied into the main package source, so that it can be directly
importable. 

Good news: The next beta of Sage will support this - see https://trac.sagemath.org/ticket/31377

 

[Matthias Koeppe]

On Thursday, March 11, 2021 at 3:23:03 PM UTC-8 François Bissey wrote:

> On 12/03/2021, at 12:14, Matthias Koeppe <matthia...@gmail.com> wrote:
> this one? https://pypi.org/project/sagemath-standard/#files

Even for betas? I was thinking there would only be some for proper releases.

Yes, pip handles this fine. It will prefer stable releases over newer beta versions unless `--pre` is used.

Well, I should get ready to use that stuff directly rather than the sage
git tarball with stuff that I do not use and that requires special settings
to figure out where the sources are located.

Sure, makes sense if that is easier for you in the context of the gentoo packaging scripts.

From the side of Sage, there's not really a difference.

 

I've written a bit about the source layout in I've written a bit about the source layout in https://wiki.sagemath.org/ReleaseTours/sage-9.3#Modularization_and_packaging_of_sagelib -- feedback is very welcome!

9.3 may very well be my last “classic" release of sage. 

Well, thanks for all your work that has helped make Sage easier to package!