How to deal with cross-references to Sphinx documentation of other projects

[Markus Wageringel]

This came up in #29231, and #27164 is related.

Several packages included with Sage use Sphinx for their documentation (such as numpy, scipy, sympy, matplotlib, pplpy, cypari, cysignals). For the quality of our documentation, it would be nice to have more cross-references to the online documentation of some of those projects. Sphinx supports this via the intersphinx extension, which we already use for Python cross-references, for example, so that markup like :func:`map` is turned into a hyperlink to the corresponding Python documentation page.

The way this works is that every Sphinx project provides an inventory file `objects.inv` containing mappings of objects that can be cross-referenced. Those inventory files are then inspected during our documentation build in order to resolve missing references. For Python and pplpy, the inventory files exist locally in Sage, which are the only ones used by Sage so far, but for other projects they can only be found online. Intersphinx can be configured to use local inventory files or to download them.

It was brought up that building Sage should in principle work without an internet connection, so that downloading inventory files during the documentation build is suboptimal. At the same time, Sage is trying to move away from Sage-the-distribution, so that packaging third-party inventory files for offline use in Sage is undesirable as well, not to mention the maintenance overhead this would entail.

This raises the question of how Sage should deal with these cross-references. My suggestion would be to do one of the following:

1. Allow the documentation build to trigger internet connections to download inventory files. If a download fails, we could either raise an error, or ignore it and just not resolve cross-links. The former is the approach that projects like numpy, scipy and matplotlib take.

2. Add a docbuild option to optionally enable downloading inventory files, similar to the `--no-plot` docbuild option for example. This way, the documentation would by default build without internet connection and without resolving cross-references to other projects. However, the option could be enabled for building the documentation files that are put on our website, which is presumably where most people read the Sage documentation.

Comments and other suggestions are very welcome. Thanks.

[Kwankyu Lee]

 At the same time, Sage is trying to move away from Sage-the-distribution, so that packaging third-party inventory files for offline use in Sage is undesirable as well, not to mention the maintenance overhead this would entail.

3. Ask by default before downloading missing inventory files including python3.inv and adding a docbuild option `--yes-download` to prevent asking. 

[Michael Orlitzky]

On 2/26/20 2:46 PM, Markus Wageringel wrote:
>
> This raises the question of how Sage should deal with these
> cross-references.

Presumably those objects.inv files are generated when the documentation
is built? If the sage documentation is going to link to the numpy
documentation (for example) and if we want it to work offline, then we
should build the numpy documentation too. Then we'll have an objects.inv
file handy when it comes time to build the sage docs, and everything
will work out.

If we don't build the numpy documentation (and don't get it from a
system package), then linking to it won't work offline anyway. Spending
even more time building the numpy documentation would suck, but we
should ultimately be convincing people to use their distribution
packages for things like numpy, and building the documentation is
essentially free in that case.

[Markus Wageringel]

This is an interesting suggestion. How would this work with non-interactive builds like on the patchbots or in CI? Would the user be asked right before downloading, i.e. during the docbuild phase, or perhaps when running configure, so it can fail early?

Am 27.02.2020 um 03:30 schrieb Kwankyu Lee <ekwa...@gmail.com>:

 At the same time, Sage is trying to move away from Sage-the-distribution, so that packaging third-party inventory files for offline use in Sage is undesirable as well, not to mention the maintenance overhead this would entail.

3. Ask by default before downloading missing inventory files including python3.inv and adding a docbuild option `--yes-download` to prevent asking. 

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/h3jkMGhKjBs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/35dacff1-b715-445f-ae89-6aa4da22c212%40googlegroups.com.

[Markus Wageringel]

Yes, the objects.inv files are generated by Sphinx. This is the approach that Sage currently follows with pplpy. The documentation is compiled and installed by the spkg-postinst of pplpy.

Applying this approach to other packages however is not easily doable. When building the numpy documentation for example, numpy will simply download missing inventory files and thus connect to the internet.

I am not sure how other distributions work, but Homebrew for example does not include documentation files. Packages on PyPI also do not seem to include the objects.inv file. So in these cases, resolving cross-references can only work if we download missing inventory files or include them with Sage.

> --
> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/h3jkMGhKjBs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8deec2f8-a389-63f2-722d-5c4a55798b69%40orlitzky.com.

[Michael Orlitzky]

On 2/27/20 1:39 PM, Markus Wageringel wrote:
> Yes, the objects.inv files are generated by Sphinx. This is the approach that Sage currently follows with pplpy. The documentation is compiled and installed by the spkg-postinst of pplpy.
>
> Applying this approach to other packages however is not easily doable. When building the numpy documentation for example, numpy will simply download missing inventory files and thus connect to the internet.

In this case, numpy has prebuilt documentation containing the
objects.inv file:

https://numpy.org/doc/1.17/numpy-html.zip

Are there any others that we couldn't build?

> I am not sure how other distributions work, but Homebrew for example
> does not include documentation files. Packages on PyPI also do not
> seem to include the objects.inv file. So in these cases, resolving
> cross-references can only work if we download missing inventory files
> or include them with Sage.

I'm sure it's technically possible to build the docs with homebrew if
somebody asks. They're optionally included in Gentoo for most packages,
and with Debian/Arch/Fedora it's just a matter of asking them to include
a prebuilt *-doc package. Debian and Arch already package the numpy
docs, from what I can tell. Conda/nix should be relatively easy too
since they build from source.

In any case, missing docs on some distro wouldn't be a blocker. Having
the distro provide the docs makes your life easier, but you can always
build the spkg, or just have broken cross-references in your sage docs
if you don't care.

[Kwankyu Lee]

On Friday, February 28, 2020 at 3:37:52 AM UTC+9, Markus Wageringel wrote:

This is an interesting suggestion. How would this work with non-interactive builds like on the patchbots or in CI?

So add a docbuild option `--yes-download` to prevent asking. And `--no-download` might be useful too...

 

Would the user be asked right before downloading, perhaps when running configure, so it can fail early?

I imagined asking download permission from the user at one time both for packages and inventory files at start of making. If no download is needed because they were already downloaded, no asking happens.

I am not sure if this can happen at configure time.