building 4.7.alpha4 from source with SAGE_SPKG_INSTALL_DOCS=yes fails
Dan Drake
noticed...)
On Mon, 11 Apr 2011 at 03:45PM +0200, Jeroen Demeyer wrote:
> ear Sage lovers,
>
> We're releasing Sage 4.7.alpha4.
Hrm. There is a problem with numpy when SAGE_SPKG_INSTALL_DOCS is set.
Here's the relevant part of the log:
make[2]: Entering directory `/home/drake/s/sage-4.7.alpha4/spkg/build/numpy-1.5.1.p0/src/doc'
mkdir -p build
touch build/generate-stamp
mkdir -p build/html build/doctrees
LANG=C sphinx-build -b html -d build/doctrees source build/html
/bin/sh: sphinx-build: not found
make[2]: *** [html] Error 127
make[2]: Leaving directory `/home/drake/s/sage-4.7.alpha4/spkg/build/numpy-1.5.1.p0/src/doc'
Error building numpy docs.
This is a fresh build from source. It looks like we need to alter the
build dependencies to get Sphinx built early. Or perhaps the spkg
installation script could check to see if the INSTALL_DOCS variable is
set, and if so, check that Sphinx is built?
The tricky thing here seems to be that the build deps are different if
SAGE_SPKG_INSTALL_DOCS is set.
I installed the Sphinx spkg and restarted the build, but now I get:
make[2]: Entering directory `/home/drake/s/sage-4.7.alpha4/spkg/build/numpy-1.5.1.p0/src/doc'
mkdir -p build
touch build/generate-stamp
mkdir -p build/html build/doctrees
LANG=C sphinx-build -b html -d build/doctrees source build/html
Traceback (most recent call last):
File "/home/drake/s/sage-4.7.alpha4/local/bin/sphinx-build", line 6, in <module>
import sage.all
ImportError: No module named sage.all
make[2]: *** [html] Error 1
make[2]: Leaving directory `/home/drake/s/sage-4.7.alpha4/spkg/build/numpy-1.5.1.p0/src/doc'
Error building numpy docs.
Yikes. Do we need the Sage library before we can build any Sphinx
documents? Do we need to build documents *after* the usual build process
is done?
Dan
--
--- Dan Drake
----- http://mathsci.kaist.ac.kr/~drake
-------
Jeroen Demeyer
> There is a problem with numpy when SAGE_SPKG_INSTALL_DOCS is set.
Unmerging #10826 because of this...
Jason Grout
>
> Yikes. Do we need the Sage library before we can build any Sphinx
> documents? Do we need to build documents *after* the usual build process
> is done?
Interesting. It looks like in 2008, Mike Hansen "Made sure that
sphinx-build.py imported sage.all at the beginning", from the sphinx
SPKG.txt. I'm not sure why; it seems that sage.all doesn't appear
anywhere in the spkg file other than the changelog note and the single
"import sage.all" in the patched sphinx-build.py. Furthermore, I wonder
if the dependency chain in the spkg deps file really enforces that sage
is installed before sphinx. I traced it a little bit, but didn't come
to any conclusion (it certainly wasn't an obvious dependency).
However, since we have this problem with the python package itself,
maybe using the same approach we plan on executing there would work
here. That is: after everything is done, maybe when we are building the
sage documentation anyway, we attempt to rebuild an spkgs that have docs
to build. In those spkgs that have docs to build, we wrap the current
spkg-install in the following outer loops (pseudocode here):
if SAGE_SPKG_INSTALL_SPKG:
...build and install source...
...just like things are now...
if SAGE_SPKG_INSTALL_DOCS:
...build and install docs...
Then in the initial install phase, SAGE_SPKG_INSTALL_DOCS is temporarily
set to false and SAGE_SPKG_INSTALL_SPKG is set to true, and spkgs are
installed as normal. After everything is installed, then the
SAGE_SPKG_INSTALL_DOCS is reverted to whatever it was, and
SAGE_SPKG_INSTALL_SPKG is set to false. The spkgs with possible docs to
build are then "installed" again, but this time all they do is build
their docs.
That way, docs (including sage's docs) all get built at the same time,
so if there is just a problem with installing docs, or if the user
doesn't want to wait and cancels the build while building docs, it
doesn't prevent a sage build. It does mean that the spkgs with docs to
build are extracted twice, but if someone wants to build the docs,
presumably the extraction twice won't be that much more time.
What do people think?
Jason
Dan Drake
> However, since we have this problem with the python package itself,
> maybe using the same approach we plan on executing there would work
> here. That is: after everything is done, maybe when we are building
> the sage documentation anyway, we attempt to rebuild an spkgs that
> have docs to build. In those spkgs that have docs to build, we wrap
> the current spkg-install in the following outer loops (pseudocode
> here):
>
> if SAGE_SPKG_INSTALL_SPKG:
>
> ...build and install source...
> ...just like things are now...
>
> if SAGE_SPKG_INSTALL_DOCS:
>
> ...build and install docs...
>
> Then in the initial install phase, SAGE_SPKG_INSTALL_DOCS is
> temporarily set to false and SAGE_SPKG_INSTALL_SPKG is set to true,
> and spkgs are installed as normal. After everything is installed,
> then the SAGE_SPKG_INSTALL_DOCS is reverted to whatever it was, and
> SAGE_SPKG_INSTALL_SPKG is set to false. The spkgs with possible
> docs to build are then "installed" again, but this time all they do
> is build their docs.
That will certainly work, and it does seem like we need to do two passes
-- build everything, then do documentation. How about, in addition to
spkg-install and spkg-check, we have spkg-install-docs? We would move
the logic that checks the SAGE_SPKG_INSTALL_DOCS variable to the
build/install script, which would run the script if the env var is set.
This would also allow the build process to determine if a spkg has
documentation that can be built, using something roughly like
tar jtf foo.spkg | grep spkg-install-docs
As a data point, doing that for all spkgs in 4.7.alpha4 (except the Sage
library, which we handle separately) takes about 70 seconds on a
mildly-loaded sagenb.kaist.ac.kr, which is an 8-core Xeon machine.
Thoughts?
Jeroen Demeyer
That's probably the best solution. I'm only slightly worried about
efficiency. If all spkgs have to be unpacked twice (once for install,
once for install-docs) that creates considerable I/O overhead.
Jeroen.
Jason Grout
Even with the I/O overhead, it sounds like the best solution to date. I
really like the idea of an spkg-install-docs instead of trying to mess
with the spkg-install file.
I've made http://trac.sagemath.org/sage_trac/ticket/11197 to track this.
Jason