How to use Sphinx to create documentation for a project that uses Sage?

[Paul Leopardi]

Hello,
I am trying to create HTML documentation for my Boolean-Cayley-graphs project ( https://github.com/penguian/Boolean-Cayley-graphs ) that uses Sage.
If I try to just use Sphinx, following the instructions at https://codeandchaos.wordpress.com/2012/07/30/sphinx-autodoc-tutorial-for-dummies/ it is no clear to me how to tell Sphinx where to find everything in Sage that it might need to import.

The Sage documentation itself says how to generate the Sage manuals ( http://doc.sagemath.org/html/en/developer/sage_manuals.html ).
I can't find anything that says how to generate documentation for code that uses Sage without incorporating that code into Sage proper.

I need step by step instructions, with examples. Do these exist somewhere? Have I overlooked something? Or is it just assumed that I should fork Sage itself and incorporate my code there, just so that I can generate documentation?
All the best, Paul

[Jeroen Demeyer]

On 2017-05-28 14:33, Paul Leopardi wrote:
> Hello,
> I am trying to create HTML documentation for my Boolean-Cayley-graphs
> project ( https://github.com/penguian/Boolean-Cayley-graphs ) that uses
> Sage.
> If I try to just use Sphinx, following the instructions at
> https://codeandchaos.wordpress.com/2012/07/30/sphinx-autodoc-tutorial-for-dummies/

Good! Since you are sending this email, I assume that something is not
working for you. What is the problem?

[Paul Leopardi]

On Tuesday, 30 May 2017 18:16:51 UTC+10, Jeroen Demeyer wrote:

Good! Since you are sending this email, I assume that something is not
working for you. What is the problem?

Please bear with me as I retrace my steps and show you the result.

 1. Following https://codeandchaos.wordpress.com/2012/07/30/sphinx-autodoc-tutorial-for-dummies/ I ran sphinx-quickstart:

leopardi@catawba:~/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc$ sphinx-quickstart
Welcome to the Sphinx 1.5.3 quickstart utility.

Please enter values for the following settings (just press Enter to
accept a default value, if one is given in brackets).

Enter the root path for documentation.
> Root path for the documentation [.]:

You have two options for placing the build directory for Sphinx output.
Either, you use a directory "_build" within the root path, or you separate
"source" and "build" directories within the root path.
> Separate source and build directories (y/n) [n]:

Inside the root directory, two more directories will be created; "_templates"
for custom HTML templates and "_static" for custom stylesheets and other static
files. You can enter another prefix (such as ".") to replace the underscore.
> Name prefix for templates and static dir [_]:

The project name will occur in several places in the built documentation.
> Project name: Boolean-Cayley-graphs
> Author name(s): Paul Leopardi

Sphinx has the notion of a "version" and a "release" for the
software. Each version can have multiple releases. For example, for
Python the version is something like 2.5 or 3.0, while the release is
something like 2.5.1 or 3.0a1.  If you don't need this dual structure,
just set both to the same value.
> Project version []: 0.0
> Project release [0.0]: 0.0.1

If the documents are to be written in a language other than English,
you can select a language here by its language code. Sphinx will then
translate text that it generates into that language.

For a list of supported codes, see
http://sphinx-doc.org/config.html#confval-language.
> Project language [en]:

The file name suffix for source files. Commonly, this is either ".txt"
or ".rst".  Only files with this suffix are considered documents.
> Source file suffix [.rst]:

One document is special in that it is considered the top node of the
"contents tree", that is, it is the root of the hierarchical structure
of the documents. Normally, this is "index", but if your "index"
document is a custom template, you can also set this to another filename.
> Name of your master document (without suffix) [index]:

Sphinx can also add configuration for epub output:
> Do you want to use the epub builder (y/n) [n]:

Please indicate if you want to use one of the following Sphinx extensions:
> autodoc: automatically insert docstrings from modules (y/n) [n]: y
> doctest: automatically test code snippets in doctest blocks (y/n) [n]: y
> intersphinx: link between Sphinx documentation of different projects (y/n) [n]:
> todo: write "todo" entries that can be shown or hidden on build (y/n) [n]:
> coverage: checks for documentation coverage (y/n) [n]: y
> imgmath: include math, rendered as PNG or SVG images (y/n) [n]:
> mathjax: include math, rendered in the browser by MathJax (y/n) [n]: y
> ifconfig: conditional inclusion of content based on config values (y/n) [n]:
> viewcode: include links to the source code of documented Python objects (y/n) [n]:
> githubpages: create .nojekyll file to publish the document on GitHub pages (y/n) [n]:

A Makefile and a Windows command file can be generated for you so that you
only have to run e.g. `make html' instead of invoking sphinx-build
directly.
> Create Makefile? (y/n) [y]:
> Create Windows command file? (y/n) [y]: n

Creating file ./conf.py.
Creating file ./index.rst.
Creating file ./Makefile.

Finished: An initial directory structure has been created.

You should now populate your master file ./index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
   make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.

2. Next, I ran make html

leopardi@catawba:~/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc$ make html
Running Sphinx v1.5.3
making output directory...
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                                                                                                                                                               
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                                                                                                                                                                
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in _build/html.

3. So far the index.html page did not show any content, as I had not told Sphinx where to find it. So, still following the instructions, I edited the generated conf.py:

leopardi@catawba:~/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc$ diff -ub conf.py.orig conf.py
--- conf.py.orig        2017-05-30 23:02:34.694951241 +1000
+++ conf.py     2017-05-30 23:09:42.787327564 +1000
@@ -16,9 +16,9 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+sys.path.insert(0, os.path.abspath('../sage-code/boolean_cayley_graphs'))
 
 4. I ran make html again:

leopardi@catawba:~/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc$ make html
Running Sphinx v1.5.3
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.

Build finished. The HTML pages are in _build/html.

5. There is no change to the HTML. Still following https://codeandchaos.wordpress.com/2012/07/30/sphinx-autodoc-tutorial-for-dummies/ I then edited index.rst
leopardi@catawba:~/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc$ diff -ub index.rst.orig index.rst
--- index.rst.orig      2017-05-30 23:02:34.694951241 +1000
+++ index.rst   2017-05-30 23:16:43.611120666 +1000
@@ -10,7 +10,10 @@
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: bent_function_cayley_graph_classification
 
+.. autoclass:: BentFunction
+    :members:
 
 Indices and tables

6. I then ran make html again, and saw the following output, including error messages.

leopardi@catawba:~/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc$ make html
Running Sphinx v1.5.3
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] index                                                                                                                                                                                                                                               
/home/leopardi/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc/index.rst:13: WARNING: autodoc: failed to import module u'bent_function_cayley_graph_classification'; the following exception was raised:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/leopardi/sync/src/sage-sandbox/Boolean-Cayley-graphs/sage-code/boolean_cayley_graphs/bent_function_cayley_graph_classification.py", line 48, in <module>
    from sage.arith.srange import xsrange
ImportError: No module named sage.arith.srange
/home/leopardi/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc/index.rst:15: WARNING: don't know which module to import for autodocumenting u'BentFunction' (try placing a "module" or "currentmodule" directive in the document, or giving an explicit module name)
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                                                                                                                                                                
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 2 warnings.

Build finished. The HTML pages are in _build/html.

7. The error message "No module named sage.arith.srange" tells me that make can't find the Sage Python modules.
How do I tell Sphinx enough about Sage to be able to run make html without errors? E.g. is there a way of invoking Sphinx from inside a Sage session? Would that solve the problem?

Do you understand the problem yet, or do I need to explain it again in greater detail?

[Jeroen Demeyer]

On 2017-05-30 15:24, Paul Leopardi wrote:
> Do you understand the problem yet, or do I need to explain it again in
> greater detail?

I understand the problem, but you are focusing too much on the detail of
Sphinx. The problem is unrelated to Sphinx; it has to do with your
Sage/Python installation in general.

Sphinx needs to be able to import your modules to document them. There
is nothing Sphinx-specific about this, it's just usual Python imports. I
am assuming that your package works within Sage but not outside of Sage.
That means that you need to run Sphinx also within the Sage environment.

The easiest way to do this is running a Sage shell:

$ sage --sh

And now do everything again within this environment.

[Paul Leopardi]

On Tuesday, 30 May 2017 23:35:37 UTC+10, Jeroen Demeyer wrote:

Sphinx needs to be able to import your modules to document them. There
is nothing Sphinx-specific about this, it's just usual Python imports. I
am assuming that your package works within Sage but not outside of Sage.
That means that you need to run Sphinx also within the Sage environment.

The easiest way to do this is running a Sage shell:

$ sage --sh

And now do everything again within this environment.

Thanks. I was not aware of sage -sh. This gets me a little further, but still doesn't work.

(sage-sh) leopardi@catawba:doc$ make html
Running Sphinx v1.4.4

loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] index                                                                                                                                                                                                                                               
/home/leopardi/sync/src/sage-sandbox/Boolean-Cayley-graphs/doc/index.rst:13: WARNING: autodoc: failed to import module u'bent_function_cayley_graph_classification'; the following exception was raised:
Traceback (most recent call last):

  File "/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 518, in import_object

    __import__(self.modname)
  File "/home/leopardi/sync/src/sage-sandbox/Boolean-Cayley-graphs/sage-code/boolean_cayley_graphs/bent_function_cayley_graph_classification.py", line 48, in <module>
    from sage.arith.srange import xsrange

  File "sage/rings/integer.pxd", line 7, in init sage.arith.srange (/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/src/build/cythonized/sage/arith/srange.c:9410)
    cdef class Integer(EuclideanDomainElement):
  File "sage/rings/rational.pxd", line 8, in init sage.rings.integer (/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/src/build/cythonized/sage/rings/integer.c:50835)
    cdef class Rational(sage.structure.element.FieldElement):
  File "sage/rings/rational.pyx", line 66, in init sage.rings.rational (/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/src/build/cythonized/sage/rings/rational.c:40571)
  File "sage/rings/integer_ring.pyx", line 60, in init sage.rings.integer_ring (/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/src/build/cythonized/sage/rings/integer_ring.c:15199)
  File "/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/local/lib/python2.7/site-packages/sage/structure/factorization.py", line 190, in <module>
    from sage.misc.all import prod
  File "/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/local/lib/python2.7/site-packages/sage/misc/all.py", line 88, in <module>
    from .functional import (additive_order,
  File "/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/local/lib/python2.7/site-packages/sage/misc/functional.py", line 32, in <module>
    from sage.rings.complex_double import CDF
  File "sage/rings/real_double.pxd", line 8, in init sage.rings.complex_double (/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/src/build/cythonized/sage/rings/complex_double.c:23863)
    cdef class RealDoubleField_class(Field):
  File "sage/rings/real_double.pyx", line 63, in init sage.rings.real_double (/home/leopardi/sync/src-downloaded/sage/sage-7.6-Ubuntu_16.04-x86_64/src/build/cythonized/sage/rings/real_double.c:26478)
ImportError: cannot import name ZZ

[Jeroen Demeyer]

I think you need to import sage.all first, before importing anything
else from sage.

[Paul Leopardi]

On Wednesday, 31 May 2017 19:39:32 UTC+10, Jeroen Demeyer wrote:

I think you need to import sage.all first, before importing anything
else from sage.

Thanks, in which file do you mean, every Python file that uses Sage, or some specific file?
 

[Paul Leopardi]

On Wednesday, 31 May 2017 19:39:32 UTC+10, Jeroen Demeyer wrote:

I think you need to import sage.all first, before importing anything
else from sage.

Thanks, I think I figured out what you mean. I updated conf.py to contain the following:
...
# If extensions (or modules to document with autodoc) are in another directory,

# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#

import os
import sys
sys.path.insert(0, os.path.abspath('../sage-code'))
sys.path.insert(0, os.path.abspath('../sage-code/boolean_cayley_graphs'))
import sage.all
...
After that, make html worked.
 

[Jeroen Demeyer]

On 2017-05-31 12:55, Paul Leopardi wrote:
> Thanks, I think I figured out what you mean. I updated conf.py to
> contain the following:

That's not what I meant since I said that the issues you are having are
unrelated to Sphinx. So by "fixing" it in Sphinx you are working around
the problem, not actually fixing the problem.

[Jeroen Demeyer]

Well, you just have to make sure that the statement "import sage.all" is
executed at least once before importing anything from Sage. Now, in
practice, the easiest way to achieve that is to put "import sage.all" in
the top-level __init__.py file of your package. Note that this will
cause Sage imported when importing anything from your package (this may
or may not be what you want).

[mforets]

Hi Paul,
let me mention that there is a "sage sample project":   https://github.com/sagemath/sage_sample#documentation
which you may use. although in that project, it is implicit that you have .py files (instead of .sage files).
some indications are given in this FAQ:
http://doc.sagemath.org/html/en/faq/faq-contribute.html#i-wrote-some-sage-code-and-i-want-it-to-be-integrated-into-sage-however-after-renaming-my-file-a-sage-to-a-py-i-got-syntax-errors-do-i-have-to-rewrite-all-my-code-in-python-instead-of-sage