Decorators degrading documentation

[Rob Beezer]

(Say that subject three times fast.)

I'm looking at a ticket whose main purpose is to apply the
@rename_keyword decorator in lots of places, outside of the plot
module, where it was first employed.

When I build the HTML version of the reference manual, it would appear
these functions are now listed simply and universally as foo(*args,
**kwds) and any automatic information about the exact nature of the
real arguments is lost. If the docstrings were to list this
information carefully, at least it would be there, but the docstrings
are not always so careful.

I believe there is a similar behavior with the @options decorator,
though I have not investigated as thoroughly. Again, these options
are not always listed in the docstring. I think this explains why
sometimes it hard to tell just what plotting options are available.

In the case of @rename_keyword there is automatically a deprecation
warning. So someday, the decorator will be pulled and the
documentation will go back to a more informative version.

I find the generic version of the function definitions less than
satisfactory. I'd guess it would be had to make Sphinx pickup the
more detailed info in these situations? I'd also guess the decorators
could maybe manipulate the docstring and inject some information based
on the arguments of the decorator? Either way, could the effect of
these decorators on the documentation be improved?

Rob

[Tom Boothby]

> I find the generic version of the function definitions less than
> satisfactory.  I'd guess it would be had to make Sphinx pickup the
> more detailed info in these situations?  I'd also guess the decorators
> could maybe manipulate the docstring and inject some information based
> on the arguments of the decorator?  Either way, could the effect of
> these decorators on the documentation be improved?

Yes, that is possible -- decorators already have to "steal"
documentation from whatever they decorate. I think that we should
require decorators to document their existence and effect.

--tom

[John H Palmieri]

I don't know if this is the same issue, but I think I've also seen the
@CachedFunction decorator hide documentation: if you have

@CachedFunction
def bozo(...):

Then bozo doesn't appear in the reference manual. I've considered
doing

def bozo_(...):

bozo = CachedFunction(bozo_)

Then bozo_ appears in the ref manual and bozo is what you actually
call. But it's annoying.

--
John

[Tim Joseph Dumol]

> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
>

Perhaps changing the decorators to make use of the decorator package
(http://micheles.googlecode.com/hg/decoratior/documentation.html and
http://pypi.python.org/pypi/decorator), which preserve call signatures
and docstrings, would help?

--
Tim Joseph Dumol <tim (at) timdumol (dot) com>
http://timdumol.com

[Rob Beezer]

Thanks for the pointer, Tim. That looks very promising - I'll try
some experiments when I get a chance.

And I'd forgotten all about John's struggles with @CachedFunction.
The documentation for the decorator package has an example that is a
"memoize" decorator that seems to have an identical purpose as Sage's
caching decorators. Maybe the example would provide some insight, or
be a simple verbatim test case to see if it comes through into the
documentation.

Rob

On Sep 16, 8:11 am, Tim Joseph Dumol <t...@timdumol.com> wrote:
> Perhaps changing the decorators to make use of the decorator package

> (http://micheles.googlecode.com/hg/decoratior/documentation.htmlandhttp://pypi.python.org/pypi/decorator), which preserve call signatures
> and docstrings, would help?

[cousteau]

On 16 sep, 17:06, John H Palmieri <jhpalmier...@gmail.com> wrote:
> I don't know if this is the same issue, but I think I've also seen the
> @CachedFunction decorator hide documentation: if you have
>
>   @CachedFunction
>   def bozo(...):
>
> Then bozo doesn't appear in the reference manual.  I've considered
> doing
>
>   def bozo_(...):
>
>   bozo = CachedFunction(bozo_)
>
> Then bozo_ appears in the ref manual and bozo is what you actually
> call.  But it's annoying.

It's just a workaround, but what about

def bozo(...):
...
bozo = CachedFunction(bozo)

This is, you define a function and assign it the "bozo" identifier,
and then set the "bozo" identifier to point to the cached version of
the old "bozo".

[John H Palmieri]

Unfortunately, this makes any entry for "bozo" disappear from the
reference manual.

--
John

[Simon King]

Hi!

On 18 Sep., 00:00, John H Palmieri <jhpalmier...@gmail.com> wrote:
> ...

> Unfortunately, this makes any entry for "bozo" disappear from the
> reference manual.

Why is this? For my group cohomology spkg, I wrote some decorator
"temporary_result" that caches methods of a ring approximation (i.e.,
generators and relations are only known out to a certain degree), but
clears the cache if more generators or relations are found in higher
degree.

So, in the code, I have, e.g.:
@temporary_result
def poincare_series(self):
"""
Return the Poincar\\'e series...
...
"""
...

"poincare_series" DOES appear in the package's manual (see
http://sage.math.washington.edu/home/SimonKing/Cohomology/cohomology.html#pGroupCohomology.cohomology.COHO.poincare_series).
Moreover, the decorator also manages to decorate the given docstring,
so that it becomes
Temporarily cached method: Return the Poincar’e series...

The decorator is a callable class, and a decorated method is an
instance of this class. This instance can apparently borrow the name
and the docstring (even adding a modification) from the method.

Since the decorators in Sage (more precisely: the cached method
decorator) were my source of inspiration, I am surprised that
@CachedFunction is problematic for the documentation.

Cheers,
Simon

[Johan S. R. Nielsen]

Hi

> I find the generic version of the function definitions less than
> satisfactory. I'd guess it would be had to make Sphinx pickup the
> more detailed info in these situations? I'd also guess the decorators
> could maybe manipulate the docstring and inject some information based
> on the arguments of the decorator? Either way, could the effect of
> these decorators on the documentation be improved?

I agree that this is completely unacceptable. I have created Trac 9976
and
written a patch for Sage's custom version of Sphinx. This essentially
checks if
a function/method to be documented is a decorator by looking for the
attribute
"_sage_decorating". If a function/method is actually a decorated
version of
another callable, then this callable should be referenced to in
_sage_decorating. With my patch, Sphinx then constructs the signature
from this
callable instead.

All decorators (on documented functions/methods) should then set this
attribute; in particular, sage_wraps does so (with my patch), and I
guess that
every decorator aimed at documented functions/methods should use
sage_wraps.
Ideally, noone should therefore again have to know about this
workaround.

This seems to work for every function and method I have looked at.
However, it
is a bit tedious to force Sphinx to do the rebuild. It seems that you
have to
manually touch all py-files, do a complete rebuild (sage -ba) and then
rebuild
the documentation.

I hope that you agree this solution is sensible.

Simon King wrote:
> Why is this? For my group cohomology spkg, I wrote some decorator
> "temporary_result" that caches methods of a ring approximation (i.e.,
> generators and relations are only known out to a certain degree), but
> clears the cache if more generators or relations are found in higher
> degree.
>
> So, in the code, I have, e.g.:
> @temporary_result
> def poincare_series(self):
> """
> Return the Poincar\\'e series...
> ...
> """
> ...
>
> "poincare_series" DOES appear in the package's manual (see

> http://sage.math.washington.edu/home/SimonKing/Cohomology/cohomology....).

The problem is not that the functions won't appear but that they will
have the
signature (argument list) of the decorator; for decorators which can
decorate a
function with any argument, this will often be the generic signature
"<function-name>(*args, **kwargs)". In your case, your method does not
take any
arguments (except for self), and neither does the decorator, so you
will see no
difference in the documentation.

> Moreover, the decorator also manages to decorate the given docstring,
> so that it becomes
> Temporarily cached method: Return the Poincar’e series...
>
> The decorator is a callable class, and a decorated method is an
> instance of this class. This instance can apparently borrow the name
> and the docstring (even adding a modification) from the method.
>
> Since the decorators in Sage (more precisely: the cached method
> decorator) were my source of inspiration, I am surprised that
> @CachedFunction is problematic for the documentation.

The docstring and name is inherited by the decorator by using Python's
"wraps"
decorator inside the decorator; isn't that what you did? We need a
similar
mechanism for inheriting the signature. My patch does this by
requiring
decorators to set the attribute _sage_decorating, which is
automatically done
by using Sage's "sage.misc.decorators.sage_wraps" decorator (see also
Trac
#9907).

Cheers,
Johan

[Jason Grout]

On 9/23/10 7:13 AM, Johan S. R. Nielsen wrote:
> Hi
>
>> I find the generic version of the function definitions less than
>> satisfactory. I'd guess it would be had to make Sphinx pickup the
>> more detailed info in these situations? I'd also guess the decorators
>> could maybe manipulate the docstring and inject some information based
>> on the arguments of the decorator? Either way, could the effect of
>> these decorators on the documentation be improved?
>
> I agree that this is completely unacceptable. I have created Trac 9976
> and
> written a patch for Sage's custom version of Sphinx. This essentially
> checks if
> a function/method to be documented is a decorator by looking for the
> attribute
> "_sage_decorating". If a function/method is actually a decorated
> version of
> another callable, then this callable should be referenced to in
> _sage_decorating. With my patch, Sphinx then constructs the signature
> from this
> callable instead.

In the case of an @options decorator, it would be really cool if the
function signature could be modified to show the options, or in the case
of a @suboptions decorator, show the suboptions. Is there an easy way
to have a decorator change the signature in the docs? For example,
could we make a _sage_signature attribute that the decorator could
change, and use that for the docs?

Thanks,

Jason

[Johan S. R. Nielsen]

First of all, sorry for the terrible word-wrapping in my previous
post; it seems Google groups wraps at less than 80 characters.

Thats a very nice extension; I guess all sorts of uses for this could
be imagined.

With the current patch, there is no easy way for the signature to be
changed. One would have to define a new function with the appropriate
signature inside the decorator and assign to _sage_decorating, as
_sage_decorating contains only a reference to a callable. Originally,
I wanted to use a _sage_signature like you suggest instead, but the
problem was in constructing such a signature. The most practical thing
would probably be to construct a signature as a named tuple such as
that returned by inspect.getargspec and then let Sphinx format this to
text. However, as can be seen by the code in
doc.common.sage_autodoc.py surrounding my patches, it takes Sphinx
quite a bit of logic to extract the signature, and I wanted to avoid
having to copy that. Especially references to custom C-function
signature-extractors scared me a bit.

There is, of course, the possibility to have both custom attributes,
and let _sage_signature override _sage_decorating in case both are
defined. It still leaves the problem of having to construct these
signatures in the general case.

I think the idea is good, but I'm not completely sure how to do it
elegantly.

Cheers,
Johan

[Johan S. R. Nielsen]

On Sep 23, 2:57 pm, Jason Grout <jason-s...@creativetrax.com> wrote:

Hi Jason
Ok, I succeeded in writing a slightly less intuitive version of the
patch which allows for something like the suggested for the @options
and @suboptions decorators. For example, the Sphinx-built argument
line for the function sage.plot.contour_plot.contour_plot becomes

sage.plot.contour_plot.contour_plot(f, xrange, yrange, axes=False,
linestyles=None, frame=True, labels=False, plot_points=100,
linewidths=None, colorbar=False, contours=None, fill=True, **options)

However, this does not fix everything at once. For example, the
function sage.plot.contour_plot.implicit_plot accepts two arguments
which are defaulted to None (linewidth and linestyle), so the author
did not care to add them to the options-decorator. Therefore, they
will not show up in any documentation.

If this sounds like what you wanted, I can clean up the code, test it
some more and add a new patch to #9976.

Cheers, Johan

[jason...@creativetrax.com]

That sounds great! I've been looking at the decorator patches over the
last few days, but haven't had time to do a complete review yet. I hope
to review them by the deadline for 4.6.

Thanks,

Jason