Section headings in docstrings
Oben
I have a question on the autodoc extension.
It seems sections are not allowed in docstrings, at least I get this
error messages when I use the autodoc extensions for modules,
classes, ... which contain a section heading:
Unexpected section title or transition
Is this an error on my side or are sections not allowed by design?
In my workflow I really prefer to have all documentation in the source
file and thus would benefit from sections in docstrings.
Thanks,
Oben
Guenter Milde
> In my workflow I really prefer to have all documentation in the source
> file and thus would benefit from sections in docstrings.
You might consider using PyLit (http://pylit.berlios.de).
Günter
Pauli Virtanen
Another option is to use the autodoc-process-docstring hook [1], and
mangle all sections to ".. rubric::", except for automodule::. This of
course loses TOC information.
.. [1] http://sphinx.pocoo.org/ext/autodoc.html#docstring-preprocessing
--
Pauli Virtanen
Oben
now). However, a better solution would be to have an extension which
integrates sections in docstrings into the final TOC. Maybe one day
I'll have some time to work on this .. and maybe until then someone
else has done this already ;)
But seriously, if reST and Sphinx are supposed to be the standard for
Python documentation, isn't backwards to force shifting API
documentation out of the docstrings? AFAIK the official Python
documentation is written mostly outside the source code and I see the
benefits of this approach as it avoids API generator look-a-like
documentation. OTH I would guess many projects in the Python world are
happy with simple API generation, consider the JavaDoc workflow as an
example. Well, there are tools like epydoc but I would prefer to
follow the "official" Python documentation conventions and tools. I'm
quite surprised Sphinx does not spend much attention to
straightforward API doc generation.
Roberto Alsina
> But seriously, if reST and Sphinx are supposed to be the standard for
> Python documentation, isn't backwards to force shifting API
> documentation out of the docstrings? AFAIK the official Python
> documentation is written mostly outside the source code and I see the
> benefits of this approach as it avoids API generator look-a-like
> documentation. OTH I would guess many projects in the Python world are
> happy with simple API generation, consider the JavaDoc workflow as an
> example. Well, there are tools like epydoc but I would prefer to
> follow the "official" Python documentation conventions and tools. I'm
> quite surprised Sphinx does not spend much attention to
> straightforward API doc generation.
Why not use the doc generator to create sphinx's source files? The best of
both worlds ;-)
--
("\''/").__..-''"`-. . Roberto Alsina
`9_ 9 ) `-. ( ).`-._.`) KDE Developer (MFCH)
(_Y_.)' ._ ) `._`. " -.-' http://lateral.netmanagers.com.ar
_..`-'_..-_/ /-'_.' The 6,855th most popular site of Slovenia
(l)-'' ((i).' ((!.' according to alexa.com (27/5/2007)
Fernando Perez
On Sat, Oct 3, 2009 at 2:13 PM, Oben <oben...@googlemail.com> wrote:
> But seriously, if reST and Sphinx are supposed to be the standard for
> Python documentation, isn't backwards to force shifting API
> documentation out of the docstrings? AFAIK the official Python
> documentation is written mostly outside the source code and I see the
> benefits of this approach as it avoids API generator look-a-like
> documentation. OTH I would guess many projects in the Python world are
> happy with simple API generation, consider the JavaDoc workflow as an
> example. Well, there are tools like epydoc but I would prefer to
> follow the "official" Python documentation conventions and tools. I'm
> quite surprised Sphinx does not spend much attention to
> straightforward API doc generation.
I'm also a bit puzzled by this, and I think we'd greatly benefit from
wider adoption of real reST for all docstrings, including in the
stdlib (honestly the state of docstrings in the stlib is atrocious for
the most part, so almost anything there will be an improvement). The
numpy/scipy projects adopted what I think is a very sensible docstring
standard, basically plain reST with some conventions, and so far the
nipy family of projects and ipython at least have also adopted it:
http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines
This lets you write simple docstrings like the one in the example:
http://svn.scipy.org/svn/numpy/trunk/doc/example.py
To parse these you need an extension, which is available with numpy as well:
http://projects.scipy.org/numpy/browser/trunk/doc/sphinxext/numpydoc.py
Since I personally do like fully automatic API coverage (in addition
to well-written, humanly tuned docs), in nipy we have a little API
generator that you are welcome to grab. It's two files, a short
script you tailor to your project and the module that does the real
work, that should be completely project-independent:
http://bazaar.launchpad.net/~nipy-developers/nipy/trunk/annotate/head%3A/tools/apigen.py
It's used by a little bit of code in the 'api' rule of the makefile:
http://bazaar.launchpad.net/~nipy-developers/nipy/trunk/annotate/head%3A/doc/Makefile
I've posted this here before and others have found it useful, it lets
a project produce full API docs automatically. With numpy markup in
the docstrings, they are very readable in raw text (such as with
ipython's foo?) and yet they produce quite civilized html/pdf in the
end, e.g:
http://nipy.sourceforge.net/nipy/api/generated/nipy.algorithms.statistics.rft.html
The numpy doc standard may not be perfect, but many of us find that
it's a very good compromise between plaintext readability and final
doc generation quality (the :param: style of docstring is very
annoying for users to read in plaintext).
HTH,
f
Oben
> ...
> ...
> The numpy doc standard may not be perfect, but many of us find that
> it's a very good compromise between plaintext readability and final
> doc generation quality (the :param: style of docstring is very
> annoying for users to read in plaintext).
the numpy doc style and nipy's generator a try, it looks promising.
Concerning Python/reST in general, does anyone know a project which
uses Sphinx' default reST markup (i.e. not numpy doc) in docstrings
extensively?
Mike Hansen
On Sun, Oct 4, 2009 at 8:57 PM, Oben <oben...@googlemail.com> wrote:
> Concerning Python/reST in general, does anyone know a project which
> uses Sphinx' default reST markup (i.e. not numpy doc) in docstrings
> extensively?
Sage (http://www.sagemath.org/doc/) does use Sphinx's default markup
(although not idiomatically in many cases due to autoconversion from a
previous markup.) However, a patch was just accepted to allow $...$
to delimit math mode.
--Mike
Fernando Perez
> Sage (http://www.sagemath.org/doc/) does use Sphinx's default markup
> (although not idiomatically in many cases due to autoconversion from a
> previous markup.) However, a patch was just accepted to allow $...$
> to delimit math mode.
But sage doesn't use the :param: style for parameters, does it? It
uses all-caps for informal section delimiting and regular bullet lists
(with dashes) for parameters (this is from the 'factor' docstring):
INPUT:
- ``n`` - an nonzero integer
- ``proof`` - bool or None (default: None)
I make the precision because I interpreted the OP's question as asking
about a project that would be using all the :param: & friends
machinery, not just avoiding post-processing (like numpydoc).
Speaking of the $x$ patch, with Ondrej we're maintaining a few useful
such things on a public github repo:
http://github.com/fperez/lacquer
we hope that as these things stabilize and get cleaned up, we can
upstream them (I'm using that extension a lot, as :math:`x` inline is
simply intolerable for extensive markup).
Cheers,
f
Mike Hansen
On Mon, Oct 5, 2009 at 3:32 AM, Fernando Perez <fpere...@gmail.com> wrote:
> But sage doesn't use the :param: style for parameters, does it? It
> uses all-caps for informal section delimiting and regular bullet lists
> (with dashes) for parameters (this is from the 'factor' docstring):
>
> INPUT:
>
> - ``n`` - an nonzero integer
>
> - ``proof`` - bool or None (default: None)
>
> I make the precision because I interpreted the OP's question as asking
> about a project that would be using all the :param: & friends
> machinery, not just avoiding post-processing (like numpydoc).
Yep, this is what I was talking about since the previous format just
used a LaTeX \begin{itemize} and so it autoconverted to the above when
we switched over to Sphinx. It'd be better to have something more
standard, but the work just needs to be done.
> Speaking of the $x$ patch, with Ondrej we're maintaining a few useful
> such things on a public github repo:
>
> http://github.com/fperez/lacquer
>
> we hope that as these things stabilize and get cleaned up, we can
> upstream them (I'm using that extension a lot, as :math:`x` inline is
> simply intolerable for extensive markup).
Cool.
--Mike
Oben
On Oct 5, 4:23 am, Mike Hansen <mhan...@gmail.com> wrote:
> > But sage doesn't use the :param: style for parameters, does it? It
> > uses all-caps for informal section delimiting and regular bullet lists
> > (with dashes) for parameters (this is from the 'factor' docstring):
>
> > INPUT:
>
> > - ``n`` - an nonzero integer
>
> > - ``proof`` - bool or None (default: None)
>
> > I make the precision because I interpreted the OP's question as asking
> > about a project that would be using all the :param: & friends
> > machinery, not just avoiding post-processing (like numpydoc).
>
> Yep, this is what I was talking about since the previous format just
> used a LaTeX \begin{itemize} and so it autoconverted to the above when
> we switched over to Sphinx. It'd be better to have something more
> standard, but the work just needs to be done.
mentioned by Fernando (INPUT ..). Do I get you right that you plan to
convert these to :param:... markup in the future? Finally, from a long
term perspective, I just wonder what would be the best way to format
docstrings. :param: and friends are good for Sphinx but until now I've
not seen them in the wild occurring in docstings.
@Fernando: Do you know if integrating the numpy doc extension into
Sphinx is in discussion or already in progress?
Cheers,
Oben
Fernando Perez
On Mon, Oct 5, 2009 at 1:47 PM, Oben <oben...@googlemail.com> wrote:
>
> @Fernando: Do you know if integrating the numpy doc extension into
> Sphinx is in discussion or already in progress?
I don't think it's happened yet, though I only joined this list very
recently. But I'm pretty sure nobody in numpy would be opposed to it,
we've always talked informally about it being nice if all the various
extensions we have floating around got upstreamed, it's just a matter
of making it happen.
If there's interest from the sphinx devs, I'm happy to drop a line on
the numpy list about it, to query in case someone has any more plans
or thoughts on the matter.
Honestly I think the numpy standard is quite acceptable, and I'd love
it if more and more projects adopted it. It came to be after a fair
amount of discussion on the numpy list, so even if not perfect
(nothing is), I think it's a very reasonable solution to the problem,
and one that's already at least used by: numpy, scipy, ipython
(partly, we still have a bunch of epydoc leftovers), nipy (including
the core nipy code and the 'children' projects nipype and nitime). I
haven't checked, but I wouldn't be surprised if several of the scikits
also use it.
Cheers,
f
Oben
current situation of docstrings in the wild :)
John H Palmieri
> I browsed a bit in Sage's doc and so far only found the style
> mentioned by Fernando (INPUT ..). Do I get you right that you plan to
> convert these to :param:... markup in the future?
markup and also to use it in one portion of the documentation. I
think that it would be nice to eventually convert everything to
the :param: markup, but it's pretty low on the priority list for Sage
developers.
Regards,
John
Guenter Milde
A main advantage of reST is the nice and intelligable look in plain-text
format. Using too many roles and directives, this feature is lost.
The docstring should be easy to read and understand also if I get it via
help(somefun) in a console or x-terminal. Lots of :strange:`markup` can be
distracting there. Choosing the right default role might help.
Günter
Max Battcher
The discussion here, to my knowledge, is primarily about using reST
field lists in doc strings (which predate Sphinx, actually, and have
been used by tools such as epydoc), rather than the roles that you are
assuming. An example:
def fox_speed(size, weight, age):
"""
Return the maximum speed for a fox.
:param size: The size of the fox (in meters)
:param weight: The weight of the fox (in stones)
:type weight: float
:param age: The age of the fox (in years)
:type age: int
"""
#[...]
Or better yet, using what epydoc calls "consolidated fields" (because
epydoc was built for javadoc syntax, and reST has some nicer markup
constructs built-in to take advantage of):
def fox_speed(size, weight, age):
"""
Return the maximum speed for a fox.
:Parameters:
size
The size of the fox (in meters)
weight : float
The weight of the fox (in stones)
age : int
The age of the fox (in years)
"""
#[...]
I think this last example is very readable in its ASCII form there, and
will produce nicely formatted results in Sphinx (or epydoc for that matter).
--
--Max Battcher--
http://worldmaker.net
Georg Brandl
Hash: SHA1
Max Battcher schrieb:
> On 10/8/2009 1:53, Guenter Milde wrote:
>>
>> On 2009-10-07, John H Palmieri wrote:
>>
>>> On Oct 5, 1:47 pm, Oben<obenso...@googlemail.com> wrote:
>>
>>>> I browsed a bit in Sage's doc and so far only found the style
>>>> mentioned by Fernando (INPUT ..). Do I get you right that you plan to
>>>> convert these to :param:... markup in the future?
>>
>>> A few days ago, I opened a trac ticket for Sage to document this
>>> markup and also to use it in one portion of the documentation. I
>>> think that it would be nice to eventually convert everything to
>>> the :param: markup, but it's pretty low on the priority list for Sage
>>> developers.
>>
>> A main advantage of reST is the nice and intelligable look in plain-text
>> format. Using too many roles and directives, this feature is lost.
>>
>> The docstring should be easy to read and understand also if I get it via
>> help(somefun) in a console or x-terminal. Lots of :strange:`markup` can be
>> distracting there. Choosing the right default role might help.
Agreed.
> The discussion here, to my knowledge, is primarily about using reST
> field lists in doc strings (which predate Sphinx, actually, and have
> been used by tools such as epydoc), rather than the roles that you are
> assuming. An example:
[...]
I added support for these field lists precisely because people were
already using them for epydoc and considered using Sphinx (instead or
additionally). I'm not particularly enthusiastic about them...
cheers,
Georg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
iEYEARECAAYFAkr3AGAACgkQN9GcIYhpnLD3fACeJWtIMZCtE19SnHvP96+iIpXp
fNAAoKNSr5GQdk4RtFsFfBjwnfctwdvk
=0wpD
-----END PGP SIGNATURE-----
Georg Brandl
Hash: SHA1
Oben schrieb:
> Thanks for the hints (unfortunately berlios.de seems to be down right
> now). However, a better solution would be to have an extension which
> integrates sections in docstrings into the final TOC. Maybe one day
> I'll have some time to work on this .. and maybe until then someone
> else has done this already ;)
>
> But seriously, if reST and Sphinx are supposed to be the standard for
> Python documentation, isn't backwards to force shifting API
> documentation out of the docstrings? AFAIK the official Python
> documentation is written mostly outside the source code and I see the
> benefits of this approach as it avoids API generator look-a-like
> documentation. OTH I would guess many projects in the Python world are
> happy with simple API generation, consider the JavaDoc workflow as an
> example. Well, there are tools like epydoc but I would prefer to
> follow the "official" Python documentation conventions and tools. I'm
> quite surprised Sphinx does not spend much attention to
> straightforward API doc generation.
Well, Sphinx was never intended to be "the official documentation
standard" for all Python world. It was created to be the successor to
the LaTeX based toolchain the old Python docs were written for, and
since that wasn't used extensively by other projects, I didn't assume
Sphinx to become so popular either.
The Python docs have never used any docstring extraction, all content
is in the source files. Docstrings have always been maintained separately.
While this is a lot of extra burden on the maintainers, it ensures that
both docstrings and the docs have the "right amount" of content; i.e.
docstrings are concise and only cover very important special cases,
the docs are much more in-detail and can also digress in places.
(There are some people who'd like to change that situation in Python, mainly
because they're seeing how it is comparatively easy to integrate docstrings
into a still separately written body of documentation. I'm not sure we'll
ever make that change, but it is a possibility. That something needs to
be done about many modules' docstrings is unquestioned though.)
So much for history... it explains why automatic generation of content was
not planned in Sphinx from the beginning. After a while I understood the
need for more and more automatic features, and added autodoc, autosummary,
etc. A full API doc generation may be inevitable :)
In that, I'm glad for any help -- there are just so many different ways
it could be done.
Georg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
iEYEARECAAYFAkr3EJsACgkQN9GcIYhpnLBkIQCeKJu23CuOyxxQgkP9K0i7/Yt3
qDcAoJRGW72deq6r/azJPcoB7xJY5mK8
=ivcm
-----END PGP SIGNATURE-----
Eli Collins
A few years ago I found this thread trying solve the same problem. I ended up coding my own Sphinx extension which monkeypatches Sphinx so that sections nested inside docstrings are supported.
It's distributed at part of a package containing a custom sphinx theme, but the extension should be usuable independantly of the theme (and I'm certainly open to contributing it back to Sphinx if there's interest).
The documentation is at https://pythonhosted.org/cloud_sptheme/lib/cloud_sptheme.ext.autodoc_sections.html, and the source code for the extension itself can be viewed at https://bitbucket.org/ecollins/cloud_sptheme/src/tip/cloud_sptheme/ext/autodoc_sections.py.
The only drawbacks to this extension are 1) it may require some custom css styling in order for docstring sections to layout in a desirable manner, 2) sections within docstrings aren't currently added into the TOC, and 3) it works through extensive monkeypatching and callstack inspection of sphinx.
Hopefully someone will still find it helpful, at least as a starting point for a more proper implementation :)
- Eli