100% doctesting question

[Ondrej Certik]

Hi,

one of the rule for getting code into Sage is 100% doctesting --- what
does it mean exactly?
At least one doctest per function/method? Is there some tool to check
that? I think I remember there was some script for it, but I can't
find it now.

However, at least to me, just one doctest per function is not enough,
I like to have the code 100% tested, which is a completely different
thing (but so far I didn't have much luck with automatic testing tools
to test it, like figleaf or coverage, I'll try to look into it again,
but I was curious if any of you have more experience with this). Also,
should all tests go to the docstring? I remember asking William about
it on IRC some time ago and he said it should (e.g. the TESTS section
is not shown in the docstring in ipython with sage patches). I can't
find the TESTS section here:

http://sagemath.org/doc/developer/conventions.html

but I thought I saw it somewhere (correct me if I am wrong).

In sympy, for example for the Order() class (holding the order in the
series expansion, e.g. the "O" term) has about 100 lines of tests. So
in Sage you would just put all of them into the docstring?

I however like that each function has at least one doctest showing the
actual usage of the function, so once I write (or borrow from Sage)
some script to automatically check that, I am going to impose that for
all new code going to sympy (note again, that this is about the
examples, the actual tests were always required). Because this shows
very nice in the sphinx documentation.

I am asking here, because you have the experience with holding all
tests in the docstring itself, so I am curious about your opinion of
it.

Also a related question --- where exactly should the tests in the file
"calculus/test_sympy.py" go?

Thanks,
Ondrej

[Rob Beezer]

Ondrej,

I believe the tools you want are

sage -coverage <files>
sage -coverageall

which you can find again listed when you do

sage -advanced

Rob

[William Stein]

On Mon, Jun 22, 2009 at 1:41 AM, Ondrej Certik<ond...@certik.cz> wrote:
>
> Hi,
>
> one of the rule for getting code into Sage is 100% doctesting --- what
> does it mean exactly?
> At least one doctest per function/method?

Yes.

> Is there some tool to check
> that?

Yes.

> I think I remember there was some script for it, but I can't
> find it now.

sage -coverage file.py

I wrote the first version of that script during a docday so I could
figure out what to doctest. It subsequently got traction.

> However, at least to me, just one doctest per function is not enough,
> I like to have the code 100% tested, which is a completely different
> thing (but so far I didn't have much luck with automatic testing tools
> to test it, like figleaf or coverage, I'll try to look into it again,
> but I was curious if any of you have more experience with this).

That is a worthy goal. But it should come after getting 100% of
functions tested *at all*, which is also a worthy goal. Note that
when people referee patches and look at the doctests, they of course
often complain if the tests don't test "everything". For example,
when refereeing if I look at a function foo that takes as input
parameters bar and xyz, and the doctests don't have "bar" and "xyz" in
them, I immediately complain (maybe the script should too).

> Also,
> should all tests go to the docstring? I remember asking William about

I think they should for Sage, yes. For other projects, e.g., twisted,
this would make no sense at all.

> it on IRC some time ago and he said it should (e.g. the TESTS section
> is not shown in the docstring in ipython with sage patches). I can't
> find the TESTS section here:
>
> http://sagemath.org/doc/developer/conventions.html
>
> but I thought I saw it somewhere (correct me if I am wrong).

Yes, there is a TESTS section. We don't treat it differently yet, but
*plan* to.
The TESTS section is just part of the docstring.

> In sympy, for example for the Order() class (holding the order in the
> series expansion, e.g. the "O" term) has about 100 lines of tests. So
> in Sage you would just put all of them into the docstring?

There are also lots of tests that aren't in doctrings. There's whole
modules that create randomized tests. People run there for fun every
so often. Also -- and this is important -- there are doctests in
docstrings that call the randomized tests just to make sure they still
work.

> I however like that each function has at least one doctest showing the
> actual usage of the function, so once I write (or borrow from Sage)
> some script to automatically check that, I am going to impose that for
> all new code going to sympy (note again, that this is about the
> examples, the actual tests were always required). Because this shows
> very nice in the sphinx documentation.

+1 Thanks!

> I am asking here, because you have the experience with holding all
> tests in the docstring itself, so I am curious about your opinion of
> it.

I think it is an excellent first step.

> Also a related question --- where exactly should the tests in the file
> "calculus/test_sympy.py" go?
>
> Thanks,
> Ondrej
>
> >
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Ondrej Certik]

On Sun, Jun 21, 2009 at 5:53 PM, Rob Beezer<goo...@beezer.cotse.net> wrote:
>
> Ondrej,
>
> I believe the tools you want are
>
> sage -coverage <files>
> sage -coverageall

Yes, thanks! It's in local/bin/sage-coverage. Here is an example:

$ sage -coverage devel/sage/sage/calculus/calculus.py
----------------------------------------------------------------------
devel/sage/sage/calculus/calculus.py
SCORE devel/sage/sage/calculus/calculus.py: 83% (15 of 18)

Missing documentation:
* _find_var(name):
* _find_func(name):
* symbolic_expression_from_string(s, syms=None, accept_sequence=False):

----------------------------------------------------------------------

so it only looks if the function has at least one docstring.

Ondrej

[Simon King]

Dear William,

On Jun 22, 2:20 am, William Stein <wst...@gmail.com> wrote:
> sage -coverage file.py
>
> I wrote the first version of that script during a docday so I could
> figure out what to doctest. It subsequently got traction.

Thank you, that comes at the right time for me...

Fortunately, it also seems to work for pyx-files.

It doesn't work perfectly, though (at least on my pyx-files):
- I have a method ngens, and in the doc string one can read
sage: H.ngens()
4
However, the coverage script complains that the name of the function
does not occur in the doc tests. This also happens for various other
methods.
- The script misunderstood one line in my code: It believes that it is
a doc test, but in fact it is not in a doc string but in a line of
code; it is a string that I pass to the singular interface. By
consequence, it believes it sees a function definition, but in fact it
is a definition for Singular.

However, it is a good tool that will help me to bring the doc test
coverage of my package to 100%.

Cheers,
Simon

[William Stein]

It's not a proper parser. It's just a quick "hack" that goes through
the text file and grabs """ (triple quoted strings). It even would
break if you use ''' instead of """. Somebody, maybe you, will make
it better. Precise bug reports would help a lot too, actually.

William

[Simon King]

Hi William,

On Jun 22, 6:33 pm, William Stein <wst...@gmail.com> wrote:
> It's not a proper parser. It's just a quick "hack" that goes through
> the text file and grabs """ (triple quoted strings). It even would
> break if you use ''' instead of """.

I hope you didn't take offense in what I wrote!

> Somebody, maybe you, will make
> it better.

If I find the time. Currently I am concerned with a relocation to
another country.

Hum. I already have a function that recursively lists all things
contained in an importable thing (e.g., you give it a package foo, and
it returns the docstrings for all methods of all classes/class
instances in all modules of foo).

So, this could be a building brick for something more precise. It
would work on the level of importable things, but not on the level of
source files, though. And it fails to find cdef-methods, since it
relies on Python's introspection.

> Precise bug reports would help a lot too, actually.

Here is the thing with ngens:
def ngens (self):
"""
Return the number of generators of self

NOTE:

We consider the scalar ``1`` as a generator.

EXAMPLES:

We use a temporary root directory, that will be
removed when quitting Sage.
::

sage: from pGroupCohomology.cohomology import COHO
sage: tmp_root = tmp_filename()
sage: H=COHO(8,3,root=tmp_root)
sage: H.make()
sage: H.ngens()
4
sage: len(H.gens())
4

"""
return len(self.Gen)+1

Running sage -coverage on a file bla.py with the above content yields:
----------------------------------------------------------------------
bla.py
SCORE bla.py: 100% (1 of 1)

Possibly wrong (function name doesn't occur in doctests):
* ngens (self):

----------------------------------------------------------------------


Cheers,
Simon

[David Roe]

The problem is probably the space between "ngens" and the parenthesis.
David

[Simon King]

Hi David,

On Jun 23, 12:49 am, David Roe <r...@math.harvard.edu> wrote:
> The problem is probably the space between "ngens" and the parenthesis.
> David

I haven't read William's script, but that sounds like a good
explanation.

I thought about what my "building brick" for a "coverage" script can
and can't. As I said, it relies on Python introspection, hence, it
would
- work for def methods
- work for cpdef methods
- not work for cdef methods.

William's script
- works for def methods
- finds the doc strings for both cdef and cpdef methods, but
- complains about both cdef and cpdef methods: If it gets
cdef int fancy_method(self):
then it seems to believes that the method is called "int
fancy_method", not "fancy_method". At least this is what I got from
the warning messages. Hence, it can't find the method name in the doc
string.

It seems to me that the problem with the method names can be quite
easily addressed. So, I would prefer William's script, since it can
see cdef methods. When I find some time, I'll try to work on the
script.

Cheers,
Simon

[David Roe]

I have an old patch sitting around on trac that did this: #1795.  If you want to pick up from there, it fixes a lot of these problems.
David

[Ondrej Certik]

On Sun, Jun 21, 2009 at 6:20 PM, William Stein<wst...@gmail.com> wrote:
>
> On Mon, Jun 22, 2009 at 1:41 AM, Ondrej Certik<ond...@certik.cz> wrote:
>>
>> Hi,
>>
>> one of the rule for getting code into Sage is 100% doctesting --- what
>> does it mean exactly?
>> At least one doctest per function/method?
>
> Yes.
>
>>  Is there some tool to check
>> that?
>
> Yes.
>
>>  I think I remember there was some script for it, but I can't
>> find it now.
>
> sage -coverage file.py
>
> I wrote the first version of that script during a docday so I could
> figure out what to doctest.  It subsequently got traction.

I wrote a similar script for sympy here:

http://github.com/certik/sympy/blob/44725bb2720afdbb2997c4184f8d9919e151af86/bin/coverage_doctest.py

it's still being reviewed, but I copied lots of parts from
sage-coverage.py. Do you think you would please allow me to use my
script under the BSD license? If that is not possible, I'll rewrite
the parts, so that it's not a derived work.

Thanks,
Ondrej

[William Stein]

Yes, I officially give you written permission to license at least
anything I wrote in coverage_doctest.py under the BSD license. I'm
glad to be contributing to sympy :-).

-- William Stein