Broken links in Doc
Florent Hivert
Thanks to the newly added '--warn-links' option to sage doc builder,
several patch are currently working to remove broken links in the doc. See
eg. #12810, #12767. However, there is still the problem of private methods
(the one starting with "_"). I understand that we don't want to include them
systematically, because they don't always interest the user, they render the
documentation much too big and make it unreadable. However, in several places
of the doc, in particular in files dealing with infrastructure we definitely
want to include some private members. This was asked to me at least by three
different people.
When you are in a module, there is a standard Sphinx way (an attribute called
"__all__") which tells which member of the module you want to appear in the
documentation. It is only used in Sphinx in a couple of places.
Unfortunately, it only works in a module but not in a class. I could easily
make it work in class by patching "sage_autodoc.py" but I don't think this is
exactly what we want. I think what we really want is a specific sage private
attribute to classes say "_included_private_doc_" which lists the private
method we want always see included in the doc. The non-private methods will be
included as usual.
Note that this could be done in a more standard ways: we could edit the
autogenerated ".rst" file and tell sphinx which members we want to include. I
don't think this would be as easy as the solution I suggested before: Sphinx
asks to write every member we want to include (not only the private ones).
I'm Ok to provide solution; I've a prototype on the sage-combinat queue which
seems to be working. I've to test it more before inclusion in Sage. We just
need to decide if the solution I'm suggestion is Ok and to agree on the name
(I'm currently using '__all__' which seems bad to me).
Any idea or better suggestion ?
Cheers,
Florent
David Roe
If not, I think having "_included_private_doc_" class attribute as a list of strings giving the private functions to include sounds like a fine solution.
David
--
To post to this group, send an email to sage-...@googlegroups.com
To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Florent Hivert
On Sun, Apr 08, 2012 at 12:15:32PM -0700, David Roe wrote:
> I assume there's no way to automatically add only those private functions
> that are referred to within Sphinx?
There certainly is a way to do that, after all Sphinx is written in
Python. Now, knowing the pain I had to make Sphinx aware to all.py to resolve
its links, I'm not volunteering to do that. Maybe I did it the wrong way but I
had to dive way too much in Docutils/Sphinx internals. So if someone come up
with a solution, I gladly review it but I don't think this is easily doable,
without hacking deep into the beasts.
> If not, I think having "_included_private_doc_" class attribute as a list
> of strings giving the private functions to include sounds like a fine
> solution.
This is much easier than the other solution you suggested.
Waiting for more reaction.
Cheers,
Florent
Andrey Novoseltsev
I think "_included_private_doc_" attribute would be great. Does it
have to be a list of strings, or can it be a list of class members,
i.e. without quotes?
Thank you!
Andrey
Nicolas M. Thiery
> I think "_included_private_doc_" attribute would be great.
Of course, as Florent and David I'd much prefer the approach of
including a private method if and only if it's referred to somewhere,
but unless we have a volunteer to implement that, the attribute is fine.
I am not very happy with the name of that attribute but don't have a
really better suggestion so ... Maybe "_documented_private_methods_"?
A potential issue/feature about that attribute: it will be inherited.
While we are speaking about documentation for ``._*`` methods it, a
related question. I consider Python's (``.__*__``) and Sage's
(``._*_``) special methods as public, and would be very much in favor
of including them by default in the Sphinx documentation.
What do you think?
Cheers,
Nicolas
--
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/
Andrey Novoseltsev
On Apr 9, 2:51 pm, "Nicolas M. Thiery" <Nicolas.Thi...@u-psud.fr>
wrote:
> > I think "_included_private_doc_" attribute would be great.
>
> Of course, as Florent and David I'd much prefer the approach of
> including a private method if and only if it's referred to somewhere,
> but unless we have a volunteer to implement that, the attribute is fine.
>
> I am not very happy with the name of that attribute but don't have a
> really better suggestion so ... Maybe "_documented_private_methods_"?
>
> A potential issue/feature about that attribute: it will be inherited.
>
> While we are speaking about documentation for ``._*`` methods it, a
> related question. I consider Python's (``.__*__``) and Sage's
> (``._*_``) special methods as public, and would be very much in favor
> of including them by default in the Sphinx documentation.
>
> What do you think?
>
does not show up in the documentation!
> Cheers,
> Nicolas
> --
> Nicolas M. Thi�ry "Isil" <nthi...@users.sf.net>http://Nicolas.Thiery.name/
David Roe
David
John H Palmieri
On Monday, April 9, 2012 2:14:02 PM UTC-7, David Roe wrote:
I agree that it would be nice if __*__ and _*_ showed up in the documentation.
David
Then run "sage --docbuild -u reference html". According to "sage --docbuild help":
-u, --underscore include variables prefixed with '_' in reference
manual; may be slow, may fail for PDF output
(It should probably say something other than "variables" here.") This increases the size of the reference manual, slows down the build (as noted), and will likely produce many warnings, since docstrings for underscore methods haven't received as much attention as for non-private methods, so many of them are misformatted.
--
John
John H Palmieri
On Monday, April 9, 2012 1:51:43 PM UTC-7, Nicolas M. Thiéry wrote:
While we are speaking about documentation for ``._*`` methods it, a
related question. I consider Python's (``.__*__``) and Sage's
(``._*_``) special methods as public, and would be very much in favor
of including them by default in the Sphinx documentation.
Can you post some timings and sizes for building the reference manual the default way vs. with the "-u" flag?
--
John
Florent Hivert
> On Mon, Apr 09, 2012 at 10:06:32AM -0700, Andrey Novoseltsev wrote:
> > I think "_included_private_doc_" attribute would be great.
>
> Of course, as Florent and David I'd much prefer the approach of
> including a private method if and only if it's referred to somewhere,
> but unless we have a volunteer to implement that, the attribute is fine.
>
> I am not very happy with the name of that attribute but don't have a
> really better suggestion so ... Maybe "_documented_private_methods_"?
>
> A potential issue/feature about that attribute: it will be inherited.
I currently look it into the Class.__dict__. So no inheritance.
> While we are speaking about documentation for ``._*`` methods it, a
> related question. I consider Python's (``.__*__``) and Sage's
> (``._*_``) special methods as public, and would be very much in favor
> of including them by default in the Sphinx documentation.
>
> What do you think?
When I was speaking of private methods, I was refering to those case of
methods as well. The question you are rising here is if SAGE_DOC_UNDERSCORE
should be True by defaults. This is a different question. I though it was
settled.
Cheers,
Florent
Florent Hivert
> I think "_included_private_doc_" attribute would be great. Does it
> have to be a list of strings, or can it be a list of class members,
> i.e. without quotes?
If we want to be consistent with __all__, it should be a list (or more
generally iterable) of string. DO you have a precise usecase for having a list
of members ?
Cheers,
Florent
Andrey Novoseltsev
considerations certainly override it.
Thank you,
Andrey
Nicolas M. Thiery
Not quite: I only want the .__*__ and ._*_ methods. Not the other ._*,
for which it was indeed settled that there were too many of them.
Cheers,
Nicolas
--
John H Palmieri
On Tuesday, April 10, 2012 3:45:01 PM UTC-7, Nicolas M. Thiéry wrote:
On Mon, Apr 09, 2012 at 05:03:08PM -0700, John H Palmieri wrote:
> I agree that it would be nice if __*__ and _*_ showed up in the
> documentation.
> David
>
> Then run "sage --docbuild -u reference html". According to "sage
> --docbuild help":
>
> -u, --underscore include variables prefixed with '_' in reference
> manual; may be slow, may fail for PDF output
>
> (It should probably say something other than "variables" here.") This
> increases the size of the reference manual, slows down the build (as
> noted), and will likely produce many warnings, since docstrings for
> underscore methods haven't received as much attention as for non-private
> methods, so many of them are misformatted.Not quite: I only want the .__*__ and ._*_ methods. Not the other ._*,
for which it was indeed settled that there were too many of them.
if 'SAGE_DOC_UNDERSCORE' in os.environ:
if name.split('.')[-1].startswith('_'):
return False
-- "False" means that we should *not* skip these methods when building the reference manual -- I changed the second line to
if name.split('.')[-1].startswith('_') and name.split('.')[-1].endswith('_'):
Then I built the reference manual with the -u flag. It took a little longer, not much (13 minutes vs. 11 minutes). The html documentation was a bit larger (215M vs. 177M). Sphinx produced 145 warnings. Except for the warnings, this is not too bad. If someone (not me!) wants to fix these warnings, I wouldn't mind including these methods (etc.) in the reference manual.
(I didn't bother trying to build the pdf version of the reference manual, because isn't it too unwieldy already, even without these extra methods?)
--
John
Florent Hivert
In the process of trying to help fixing broken links in the documentation, I'd
like to share some info and call for a vote:
> > > I think "_included_private_doc_" attribute would be great.
I was writing a patch to allows for including private methods through an
attribute "_included_private_doc_". I just discovered that adding an
"automethod" directive does the job. For example:
.. automethod:: _foo
.. automethod:: _bar
to the end of the main doc of a class document those two special
functions. There is a drawback: The documentation of those methods is placed
here in the given order. So maintaining alphabetic order is at the doc writer
charge. Still I think that this is better than relies on a special sage
attribute. I'm Ok to add this trick to sage devel doc if chosen. So I see two
choices here. Please vote:
[ ] use the Sphinx automethod standard trick.
[ ] use a special Sage attribute "_included_private_doc_"
> > While we are speaking about documentation for ``._*`` methods it, a
> > related question. I consider Python's (``.__*__``) and Sage's
> > (``._*_``) special methods as public, and would be very much in favor
> > of including them by default in the Sphinx documentation.
> >
> > What do you think?
This is a second question: It seems that several Sage developers consider
Python's (``.__*__``) and Sage's (``._*_``) *special* methods as public so
that they should be included in the doc by default. On the opposite, *private*
methods starts by ``_`` but doesn't end with one. I'd like to call a vote for
including them by default in the doc. Some informations (thanks to John H
Palmieri): Compiling the doc with those special functions, take a little
longer, not much (13 minutes vs. 11 minutes). The html documentation was a bit
larger (215M vs. 177M). Sphinx produced 145 warnings which should be fixed.
I see the following choices:
[ ] leave things as they are and rely on the preceding "automethod" or
"_included_private_doc_" trick to document special methods.
[ ] include __*__ and _*_ method by default in the doc.
Thanks for helping having a better doc,
Cheers,
Florent
Keshav Kini
> [ ] use the Sphinx automethod standard trick.
> [ ] use a special Sage attribute "_included_private_doc_"
No opinion.
> [ ] leave things as they are and rely on the preceding "automethod" or
> "_included_private_doc_" trick to document special methods.
> [ ] include __*__ and _*_ method by default in the doc.
The latter.
My two cents, for what they're worth...
-Keshav
----
Join us in #sagemath on irc.freenode.net !
Andrey Novoseltsev
"_included_private_doc_" trick to document special methods.
With the second answer to the second question, it is even more the
first one for the first one ;-) I don't recall if I ever wanted to
document "private-non-special" methods, while there were cases when
having containment or conversion methods documented would be nice.
Thank you!
Andrey
Florent Hivert
This means that the trivial methods __init__, __repr__, __hash__ and the like
will be included as well. They are usually not very well documented.
Cheers,
Florent
Andrey Novoseltsev
distinction between public and private methods with regard to
documentation. As I understand the main difference of private methods
is that we are free to change them without any deprecation periods or
compatibility - if users use them and their code gets broken, it is
not our problem. Special methods like __init__ or __getitem__
definitely cannot change their behaviour freely.
For __init__ it is currently recommended to put everything into the
class documentation and just make a link. So they actually should be
"well-documented" in that sense. __repr__ and __hash__ are indeed
trivial in most cases, but then it means that they will not take much
space in the documentation anyway.
Andrey
leif
> This is a second question: It seems that several Sage developers consider
> Python's (``.__*__``) and Sage's (``._*_``) *special* methods as public so
> that they should be included in the doc by default. On the opposite, *private*
> methods starts by ``_`` but doesn't end with one. I'd like to call a vote for
> including them by default in the doc. Some informations (thanks to John H
> Palmieri): Compiling the doc with those special functions, take a little
> longer, not much (13 minutes vs. 11 minutes).
already takes an hour, that's +11 minutes.
> The html documentation was a bit
> larger (215M vs. 177M).
multiplies with the number of Sage installations.
> I see the following choices:
>
> [ ] leave things as they are and rely on the preceding "automethod" or
> "_included_private_doc_" trick to document special methods.
> [ ] include __*__ and _*_ method by default in the doc.
talking about is even included in so-called binary distributions.)
IMHO, if someone needs information on "special" or private methods, he/
she can always (and will presumably anyway) look at the source code,
which in our case also contains the documentation, in more or less
readable reST format though (but next to the implementation).
I'd vote for a least having the option to *exclude* them.
Adding docstrings to such functions (or improving existing ones) is
orthogonal to including them into the HTML or PDF documentation in
general.
My 0.2 ct,
-leif
Jeroen Demeyer
> Same here. I wouldn't say that +38 MB (or +21.5%) is "a bit"; it also
> multiplies with the number of Sage installations.
Currently (i.e. without this change), Sage needs about 2.5GB to build
the documentation. I think this is already too much, it certainly
shouldn't be made worse.
Sage itself builds with just 1GB of memory and almost all doctests pass
with 2GB of RAM. It seems crazy to me that building the documentation
is the part of Sage which needs the most memory.
(all this refers to a 64-bit x86_64 Linux system and virtual memory
limited with ulimit -v)
Andrey Novoseltsev
> Adding docstrings to such functions (or improving existing ones) is
> orthogonal to including them into the HTML or PDF documentation in
> general.
necessary memory are orthogonal to (not) adding special methods to the
documentation. I am not saying that they are irrelevant, but if we hit
limits on these the solution is perhaps to change how documentation is
built, not just throw away stuff from the documentation. Maybe solving
these issues can be prerequisites to merging such a change. E.g.
docbuilding is definitely faster if it can be done in parallel, does
file splitting for parallel documentation also affect the minimal
amount of memory? Otherwise we still will increase the compile time,
increase the size, and increase the memory requirements, just do it
slower by adding new regular methods, it is just a matter of time.
Hitting LaTeX limits on http://trac.sagemath.org/sage_trac/ticket/9128
was the same issue - it surfaces when we improve documentation, but if
we don't do it - the issue is not going away, it just remains
submerged for a few more months.
I had people asking me how to check if an element is in a set and
complaining that it is not written in the documentation. Well, the
reasonable search on a doc page is for "contains" and if
"__contains__" was included, it would show up. It may be obvious to
people on this list that this is the case and that it is easy to look
at the source code, but that can be quite different for new or just
"non-developing" users.
Thank you,
Andrey
Florent Hivert
> I'd also say that docbuild time, documentation size, and amount of
> necessary memory are orthogonal to (not) adding special methods to the
> documentation.
I totally agree.
> I am not saying that they are irrelevant, but if we hit
> limits on these the solution is perhaps to change how documentation is
> built, not just throw away stuff from the documentation. Maybe solving
> these issues can be prerequisites to merging such a change. E.g.
> docbuilding is definitely faster if it can be done in parallel, does
> file splitting for parallel documentation also affect the minimal
> amount of memory?
I'm currently working on that too ! We should aim to have #6495 in Sage as
soon as possible. From what concerns compile time, using some more
improvement, I cut down the compile time of the ref man in a little more that
10 min on my laptop (a fast I7 2Gh, 8GB).
> Otherwise we still will increase the compile time,
> increase the size, and increase the memory requirements, just do it
> slower by adding new regular methods, it is just a matter of time.
> Hitting LaTeX limits on http://trac.sagemath.org/sage_trac/ticket/9128
> was the same issue - it surfaces when we improve documentation, but if
> we don't do it - the issue is not going away, it just remains
> submerged for a few more months.
+1
> I had people asking me how to check if an element is in a set and
> complaining that it is not written in the documentation. Well, the
> reasonable search on a doc page is for "contains" and if
> "__contains__" was included, it would show up. It may be obvious to
> people on this list that this is the case and that it is easy to look
> at the source code, but that can be quite different for new or just
> "non-developing" users.
+1 too
Florent
leif
> On Apr 23, 3:22 pm, leif <not.rea...@online.de> wrote:
>
> > Adding docstrings to such functions (or improving existing ones) is
> > orthogonal to including them into the HTML or PDF documentation in
> > general.
>
> I'd also say that docbuild time, documentation size, and amount of
> necessary memory are orthogonal to (not) adding special methods to the
> documentation. I am not saying that they are irrelevant, but if we hit
> limits on these the solution is perhaps to change how documentation is
> built, not just throw away stuff from the documentation. Maybe solving
> these issues can be prerequisites to merging such a change. E.g.
> docbuilding is definitely faster if it can be done in parallel, does
> file splitting for parallel documentation also affect the minimal
> amount of memory? Otherwise we still will increase the compile time,
> increase the size, and increase the memory requirements, just do it
> slower by adding new regular methods, it is just a matter of time.
> Hitting LaTeX limits on http://trac.sagemath.org/sage_trac/ticket/9128
> was the same issue - it surfaces when we improve documentation, but if
> we don't do it - the issue is not going away, it just remains
> submerged for a few more months.
flame>
> I had people asking me how to check if an element is in a set and
> complaining that it is not written in the documentation. Well, the
> reasonable search on a doc page is for "contains" and if
> "__contains__" was included, it would show up. It may be obvious to
> people on this list that this is the case and that it is easy to look
> at the source code, but that can be quite different for new or just
> "non-developing" users.
directly (at least not by an "end user"), so IMHO documenting
__contains__() doesn't really make sense, in contrast, it might
confuse people more than it helps them [to learn Python for example*].
In more complex cases than set membership at least, one could simply
add an alias method (without underscores), also documenting the
natural syntax. (Examples using the operator form should also be
contained in the class' docstring.)
-leif
________
* I don't really know, but I'm pretty sure "e in S" is documented in
some tutorial or introductory text to Sage (and it's quite similar to
how one writes this in LaTeX btw.).
Florent Hivert
> flame>
> > I had people asking me how to check if an element is in a set and
> > complaining that it is not written in the documentation. Well, the
> > reasonable search on a doc page is for "contains" and if
> > "__contains__" was included, it would show up. It may be obvious to
> > people on this list that this is the case and that it is easy to look
> > at the source code, but that can be quite different for new or just
> > "non-developing" users.
>
> Good example, as these "special" methods are not supposed to be called
> directly (at least not by an "end user"), so IMHO documenting
> __contains__() doesn't really make sense, in contrast, it might
> confuse people more than it helps them [to learn Python for example*].
First of all, it is definitely "Learning Python" to learn that
- "a in A" is translated to "A.__contains__(a)"
- "a + b" is translated to "a.__add__(b)"
- "f(x)" is translated to "f.__call__(x)"
Hiding this is just hiding Python.
Now, if the doc of contains gives the two equivalent examples with the
comment::
- a in A # implicit call
- A.__contains__(a) # explicit call
I think it will be perfectly explicit to the user.
> In more complex cases than set membership at least, one could simply
> add an alias method (without underscores), also documenting the
> natural syntax. (Examples using the operator form should also be
> contained in the class' docstring.)
1 - it hide one fundamentals of Python (special methods)
2 - it teach a bad syntax and usage to the users.
3 - it require that the developer write a lot of unused boilerplate code.
So I don't think this is a good idea.
Florent
leif
leif
> > flame>
>
> Or rewrite it in Cython !!! Please Do !!!
> > > I had people asking me how to check if an element is in a set and
> > > complaining that it is not written in the documentation. Well, the
> > > reasonable search on a doc page is for "contains" and if
> > > "__contains__" was included, it would show up. It may be obvious to
> > > people on this list that this is the case and that it is easy to look
> > > at the source code, but that can be quite different for new or just
> > > "non-developing" users.
>
> > Good example, as these "special" methods are not supposed to be called
> > directly (at least not by an "end user"), so IMHO documenting
> > __contains__() doesn't really make sense, in contrast, it might
> > confuse people more than it helps them [to learn Python for example*].
>
> I don't agree with that.
>
> First of all, it is definitely "Learning Python" to learn that
> - "a in A" is translated to "A.__contains__(a)"
> - "a + b" is translated to "a.__add__(b)"
> - "f(x)" is translated to "f.__call__(x)"
(You don't need to know it for writing basic scripts or functions
either. When you /later/ dive deeper into Python, and start to write
your own classes, it's certainly the first thing you'll learn.)
> Hiding this is just hiding Python.
people new to Sage (and Python).
And there are reasons these functions carry underscores.
-leif