Compiling the doc...

[Florent Hivert]

Dear All,

I've some trouble compiling the doc:

tomahawk-*e/sage-3.4/devel $ sage -clone doc
[...]
tomahawk-*e/sage-3.4/devel $ cd sage-doc
tomahawk-*4/devel/sage-doc $ sage -b doc
[...]
tomahawk-*4/devel/sage-doc $ sage -docbuild reference html
Traceback (most recent call last):
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 668, in <module>
getattr(get_builder(name), type)()
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 349, in _wrapper
self.write_auto_rest_file(module_name.replace(os.path.sep, '.'))
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 529, in write_auto_rest_file
title = self.get_module_docstring_title(module_name)
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 498, in get_module_docstring_title
__import__(module_name)
ImportError: No module named quaternion_order
tomahawk-*4/devel/sage-doc $ sage -docbuild reference pdf
Traceback (most recent call last):
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 668, in <module>
getattr(get_builder(name), type)()
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 210, in pdf
self.latex()
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 349, in _wrapper
self.write_auto_rest_file(module_name.replace(os.path.sep, '.'))
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 529, in write_auto_rest_file
title = self.get_module_docstring_title(module_name)
File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 498, in get_module_docstring_title
__import__(module_name)
ImportError: No module named quaternion_order

Am I doing something wrong ?

Cheers,

Florent

[mabshoff]

On Mar 16, 8:09 am, Florent Hivert <florent.hiv...@univ-rouen.fr>
wrote:

>       Dear All,
>
>    I've some trouble compiling the doc:

<SNIP>

>   File "/usr/local/sage/sage/devel/sage/doc/common/builder.py", line 498, in get_module_docstring_title
>     __import__(module_name)
> ImportError: No module named quaternion_order
>
> Am I doing something wrong ?

This issue was fixed in the final 3.4 tarball - you should check if
you have multiple heads in case you are truly running the final 3.4.

> Cheers,
>
> Florent

Cheers,

Michael

[Florent Hivert]

Dear Michael,

Strange !!! My install is a brand new one from the following tar I downloaded
from the French mirror if I didn't mix things up... Is it possible that the
mistake reach this tarball ?

tomahawk-/usr/local/sage $ md5sum sage-3.4.tar
2613a702a388aba701c95598cdd29705 sage-3.4.tar

Cheers,

Florent

[mabshoff]

On Mar 16, 10:30 am, Florent Hivert <florent.hiv...@univ-rouen.fr>
wrote:
>       Dear Michael,

Hi Florent,

> > This issue was fixed in the final 3.4 tarball - you should check if
> > you have multiple heads in case you are truly running the final 3.4.
>
> Strange !!! My install is a brand new one from the following tar I downloaded
> from the French mirror if I didn't mix things up... Is it possible that the
> mistake reach this tarball ?
>
> tomahawk-/usr/local/sage $ md5sum sage-3.4.tar
> 2613a702a388aba701c95598cdd29705  sage-3.4.tar

This is the correct tarball and I checked and the fix I mentioned is
in:

mabshoff@sage:/scratch/mabshoff/sage-3.4.1.alpha0/devel/sage$ hg
export 11798
# HG changeset patch
# User mabs...@sage.math.washington.edu
# Date 1236752650 25200
# Node ID ca39c0a70fabe3d73377523ed0cfdbb0ca23bde1
# Parent 71fa9d6a2232cd23c4c71d10fae6091d7d54d13b
Remove no longer existing Quaternion files from list of files included
into the ReST documentation

diff -r 71fa9d6a2232 -r ca39c0a70fab doc/en/reference/
quat_algebras.rst
--- a/doc/en/reference/quat_algebras.rst Tue Mar 10 22:18:13 2009
-0700
+++ b/doc/en/reference/quat_algebras.rst Tue Mar 10 23:24:10 2009
-0700
@@ -9,8 +9,3 @@
sage/algebras/quaternion_algebra
sage/algebras/quaternion_algebra_element

- sage/algebras/quaternion_order
- sage/algebras/quaternion_order_element
-
- sage/algebras/quaternion_order_ideal
- sage/algebras/quaternion_order_ideal_element
\ No newline at end of file

But strangely enough when I run the documentation build I run into the
same issue you described. I am poking around to see what is
happening ...

> Cheers,
>
> Florent

Cheers,

Michael

[mabshoff]

On Mar 16, 10:37 am, mabshoff <mabsh...@googlemail.com> wrote:

<SNIP>

> But strangely enough when I run the documentation build I run into the
> same issue you described. I am poking around to see what is
> happening ...

An observation:

"
pickling environment... done
checking consistency... WARNING: /scratch/mabshoff/sage-3.4.1.alpha0/
devel/sage-main/doc/en/reference/sage/algebras/quaternion_order.rst::
document isn't included in any toctree
done
"

There is a quaternion_order.rst in devel/sage/doc/en/reference/sage/
algebras that seems to be autogenerated.

After nuking the output directory rebuilding all documentation
repeatedly works.

Mike?

Cheers,

Michael

[Carl Witty]

This would happen if you pushed/applied a patch that added
quaternion_order.pyx (and a reference to quaternion_order.rst from one
of the main reference manual files), built the documentation, and then
popped/reverted the patch.

Maybe the right fix is to find the code that generates the WARNING
message, and make it just delete the .rst file instead.

Carl

[mabshoff]

On Mar 16, 11:08 am, Carl Witty <carl.wi...@gmail.com> wrote:

> On Mon, Mar 16, 2009 at 10:46 AM, mabshoff <mabsh...@googlemail.com> wrote:
>
> > On Mar 16, 10:37 am, mabshoff <mabsh...@googlemail.com> wrote:
>
> > <SNIP>
>
> >> But strangely enough when I run the documentation build I run into the
> >> same issue you described. I am poking around to see what is
> >> happening ...
>
> > An observation:
>
> > "
> > pickling environment... done
> > checking consistency... WARNING: /scratch/mabshoff/sage-3.4.1.alpha0/
> > devel/sage-main/doc/en/reference/sage/algebras/quaternion_order.rst::
> > document isn't included in any toctree
> > done
> > "
>
> > There is a quaternion_order.rst in devel/sage/doc/en/reference/sage/
> > algebras that seems to be autogenerated.
>
> > After nuking the output directory rebuilding all documentation
> > repeatedly works.
>
> This would happen if you pushed/applied a patch that added
> quaternion_order.pyx (and a reference to quaternion_order.rst from one
> of the main reference manual files), built the documentation, and then
> popped/reverted the patch.

Well, the strange thing is that is is re-generated even though I
deleted the output directory and the string "quaternion_order" does
not appear anywhere inside devel/sage/ expect in output directories:

mabshoff@sage:/scratch/mabshoff/sage-3.4.1.alpha0$ grep -r
quaternion_order devel/sage-main/*
devel/sage-main/doc/output/html/en/reference/sage/algebras/
quaternion_order.html: <li><a href="../../_sources/sage/
algebras/quaternion_order.txt">Show Source</a></li>
Binary file devel/sage-main/doc/output/doctrees/en/reference/
environment.pickle matches
Binary file devel/sage-main/doc/output/doctrees/en/reference/sage/
algebras/quaternion_order.doctree matches

There are also the following files:

mabshoff@sage:/scratch/mabshoff/sage-3.4.1.alpha0$ find devel/sage-
main/ -name "quaternion_order*"
devel/sage-main/doc/en/reference/sage/algebras/quaternion_order.rst
devel/sage-main/doc/output/html/en/reference/_sources/sage/algebras/
quaternion_order.txt
devel/sage-main/doc/output/html/en/reference/sage/algebras/
quaternion_order.html
devel/sage-main/doc/output/doctrees/en/reference/sage/algebras/
quaternion_order.doctree

> Maybe the right fix is to find the code that generates the WARNING
> message, and make it just delete the .rst file instead.

Something along those lines.

I now deleted the empty doc/en/reference/sage/algebras/
quaternion_order.rst file, nuked the output directory to get rid of
the pickles and did rebuild the documentation from scratch.
quaternion_order was not in the list of files where ReST strings were
exported from. And low and behold, the other files are gone:

mabshoff@sage:/scratch/mabshoff/sage-3.4.1.alpha0$ find devel/sage-
main/ -name "quaternion_order*"
mabshoff@sage:/scratch/mabshoff/sage-3.4.1.alpha0$

> Carl

It might be a good idea to have some kind of procedure that describes
how to get rid of files that have been added to the documentation even
though it should be a rare occurrence.

Cheers,

Michael

[Florent Hivert]

Hi

> There is a quaternion_order.rst in devel/sage/doc/en/reference/sage/
> algebras that seems to be autogenerated.
>
> After nuking the output directory rebuilding all documentation
> repeatedly works.

I tried this one... But on my computation server:

sage/algebras/steenrod_algebra_element sage/calculus/calculus sage/calculus/equations Exception occurred:
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/subprocess.py", line 1003, in _execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
The full traceback has been saved in /tmp/sphinx-err--kaK6F.log, if you want to report the issue to the author.
Please also report this if it was a user error, so that a better error message can be provided next time.
Send reports to sphin...@googlegroups.com. Thanks!
Build finished. The built documents can be found in /home/combi/hivert/Sage/sage/devel/sage/doc/output/html/en/reference

Is this really a bug of sphinx ?
Here is the complete traceback:

massena-*el/sage-main/doc $ cat /tmp/sphinx-err--kaK6F.log

Traceback (most recent call last):

File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/cmdline.py", line 151, in main
app.build(all_files, filenames)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/application.py", line 138, in build
self.builder.build_update()
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/builder.py", line 250, in build_update
'out of date' % len(to_build))
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/builder.py", line 290, in build
self.write(docnames, updated_docnames, method)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/builder.py", line 328, in write
self.write_doc(docname, doctree)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/builder.py", line 522, in write_doc
self.docwriter.write(doctree, destination)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/writers/__init__.py", line 76, in write
self.translate()
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/htmlwriter.py", line 37, in translate
self.document.walkabout(visitor)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 176, in walkabout
if child.walkabout(visitor):
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 176, in walkabout
if child.walkabout(visitor):
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 176, in walkabout
if child.walkabout(visitor):
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 176, in walkabout
if child.walkabout(visitor):
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 176, in walkabout
if child.walkabout(visitor):
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 176, in walkabout
if child.walkabout(visitor):
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 176, in walkabout
if child.walkabout(visitor):
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 168, in walkabout
visitor.dispatch_visit(self)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/docutils/nodes.py", line 1535, in dispatch_visit
return method(node)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/ext/pngmath.py", line 181, in html_visit_math
fname, depth = render_math(self, '$'+node['latex']+'$')
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/site-packages/Sphinx-0.5.1-py2.5.egg/sphinx/ext/pngmath.py", line 115, in render_math
p = Popen(ltx_args, stdout=PIPE, stderr=PIPE)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/subprocess.py", line 594, in __init__
errread, errwrite)
File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/subprocess.py", line 1003, in _execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
massena-*el/sage-main/doc $

Florent

[mabshoff]

On Mar 16, 11:25 am, Florent Hivert <florent.hiv...@univ-rouen.fr>
wrote:

>     Hi
>
> > There is a quaternion_order.rst in devel/sage/doc/en/reference/sage/
> > algebras that seems to be autogenerated.
>
> > After nuking the output directory rebuilding all documentation
> > repeatedly works.
>
> I tried this one... But on my computation server:
>
>  sage/algebras/steenrod_algebra_element sage/calculus/calculus sage/calculus/equations Exception occurred:
>   File "/home/combi/hivert/Sage/sage-3.4/local/lib/python2.5/subprocess.py", line 1003, in _execute_child
>     errpipe_read, errpipe_write = os.pipe()
> OSError: [Errno 24] Too many open files
> The full traceback has been saved in /tmp/sphinx-err--kaK6F.log, if you want to report the issue to the author.
> Please also report this if it was a user error, so that a better error message can be provided next time.
> Send reports to sphin...@googlegroups.com. Thanks!
> Build finished.  The built documents can be found in /home/combi/hivert/Sage/sage/devel/sage/doc/output/html/en/reference
>
> Is this really a bug of sphinx ?

It might be, but this is the error reported: "OSError: [Errno 24] Too
many open files"

Can you check with ulimit -a what your current limit is (it seems to
be 1024 on sage.math for example) and increase it and try again? The
sphinx build works for me with 256 max open files on OSX, so the limit
itseself seems unlikely to be the culprit. Maybe you have some other
code running on the box that leaks file handles?

Cheers,

Michael

[Florent Hivert]

> It might be, but this is the error reported: "OSError: [Errno 24] Too
> many open files"
>
> Can you check with ulimit -a what your current limit is (it seems to
> be 1024 on sage.math for example) and increase it and try again? The
> sphinx build works for me with 256 max open files on OSX, so the limit
> itseself seems unlikely to be the culprit. Maybe you have some other
> code running on the box that leaks file handles?

massena-*el/sage-main/doc $ ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 268288
-n: file descriptors 1024
-l: locked-in-memory size (kb) 32
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 268288
-q: bytes in POSIX msg queues 819200
-e: max nice 0
-r: max rt priority 0

massena-*el/sage-main/doc $ ulimit -n 1025
ulimit: value exceeds hard limit

I can't give it more...

Is there a way to know it some process leaks file handles ? From a regular
account / from root ? I tried lsof but it only answered my file 192 of them (I
don't have a root account).

Cheers,

Florent

[mabshoff]

On Mar 16, 11:43 am, Florent Hivert <florent.hiv...@univ-rouen.fr>
wrote:

Ok.

> Is there a way to know it some process leaks file handles ? From a regular
> account / from root ? I tried lsof but it only answered my file 192 of them (I
> don't have a root account).

I am not aware of anything in Sage that leaks file handles but the gmp-
ecm interface at the moment.

It seems that the issue pops up when building the LaTeX bits of the
documentation, so building the doc via

sage -docbuild --jsmath all html

i.e. using jsmath might work around the problem.

> Cheers,
>
> Florent

Cheers,

Michael

[davidloeffler]

I also came across the problem with quaternion_order_ideal.py, but for
me it went away when I re-built from a clean tarball (rather than
upgrading from 3.4.rc0 as I had done before). But even the final 3.4
tarball generates some Sphinx error messages when you do a docbuild,
from some slightly mis-formatted docstrings in quaternion_algebra.py.
I've made a trac ticket for this -- #5531 -- and posted a patch. Any
volunteers for a review?

David

[mabshoff]

On Mar 16, 1:43 pm, davidloeffler <dave.loeff...@gmail.com> wrote:

Hi David,

> I also came across the problem with quaternion_order_ideal.py, but for
> me it went away when I re-built from a clean tarball (rather than
> upgrading from 3.4.rc0 as I had done before). But even the final 3.4
> tarball generates some Sphinx error messages when you do a docbuild,
> from some slightly mis-formatted docstrings in quaternion_algebra.py.
> I've made a trac ticket for this -- #5531 -- and posted a patch. Any
> volunteers for a review?

Yeah, that was unfortunate, but 3.4 needed to get out the door :)

I will review your ReST fix later on today, i.e. it should be in
3.4.1.a0.

> David

Cheers,

Michael