doctest coverage to 75%

0 views
Skip to first unread message

William Stein

unread,
Apr 9, 2009, 4:45:02 AM4/9/09
to sage-devel
Hi,

In order to get Sage's doctest coverage to 75%, which is one of the
main goals for Sage-4.0, it is necessary to doctest about 1900
functions in the Sage library. I just did a test (see
http://trac.sagemath.org/sage_trac/ticket/5724 and
http://trac.sagemath.org/sage_trac/ticket/5725) and it took me on
average about 2-3 minutes per doctest, including fixing a couple of
bugs I found. Estimating 3 minutes each, that is about 95 hours of
nonstop work.

It is very easy to find files that are full of code that isn't
doctested and *must* be doctested. It was trivial for me to find the
sequences and quotient_ring_element had very bad coverage. In both
cases, I found *several* small bugs in both files. Probably, any
random file with low coverage has several bugs in it. That's why
getting the doctest coverage in Sage up is critical to Sage becoming a
truly high quality system.

I hope those of you who feel qualified to write doctests, will help
out. It's nearly impossible for one person to do all 1900 of those
doctests in the next month. Writing doctests is not easy and it
absolutely requires experience experience with the Sage library, and
good understanding of the code you are testing.

Thanks,

William


--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

John Cremona

unread,
Apr 9, 2009, 4:49:30 AM4/9/09
to sage-...@googlegroups.com
Can we at the same time convert more files to ReST and hence get them
included in the Reference manual? Although that manual is long, there
is a whole lot of stuff not mentioned in it, which is bad PR.

What I mean in practice is, if someone is going through a source file
whose doctests are incomplete and also old-style, then they convert
each to new-style as they go, as well as (at the same time as) adding
the new doctests.

John

2009/4/9 William Stein <wst...@gmail.com>:

Robert Bradshaw

unread,
Apr 9, 2009, 5:19:28 AM4/9/09
to sage-...@googlegroups.com
On Apr 9, 2009, at 1:49 AM, John Cremona wrote:

> Can we at the same time convert more files to ReST and hence get them
> included in the Reference manual? Although that manual is long, there
> is a whole lot of stuff not mentioned in it, which is bad PR.
>
> What I mean in practice is, if someone is going through a source file
> whose doctests are incomplete and also old-style, then they convert
> each to new-style as they go, as well as (at the same time as) adding
> the new doctests.

That seems like a good idea to me.

> 2009/4/9 William Stein <wst...@gmail.com>:
>>
>> Hi,
>>
>> In order to get Sage's doctest coverage to 75%, which is one of the
>> main goals for Sage-4.0, it is necessary to doctest about 1900
>> functions in the Sage library. I just did a test (see
>> http://trac.sagemath.org/sage_trac/ticket/5724 and
>> http://trac.sagemath.org/sage_trac/ticket/5725) and it took me on
>> average about 2-3 minutes per doctest, including fixing a couple of
>> bugs I found. Estimating 3 minutes each, that is about 95 hours of
>> nonstop work.
>>
>> It is very easy to find files that are full of code that isn't
>> doctested and *must* be doctested. It was trivial for me to find the
>> sequences and quotient_ring_element had very bad coverage. In both
>> cases, I found *several* small bugs in both files. Probably, any
>> random file with low coverage has several bugs in it. That's why
>> getting the doctest coverage in Sage up is critical to Sage
>> becoming a
>> truly high quality system.
>>
>> I hope those of you who feel qualified to write doctests, will help
>> out. It's nearly impossible for one person to do all 1900 of those
>> doctests in the next month. Writing doctests is not easy and it
>> absolutely requires experience experience with the Sage library, and
>> good understanding of the code you are testing.

+1. I often end up writing a lot of doctests for surrounding code
when converting things to the new coercion model.

How about another Sage doc day in the coming month? I bet with enough
of us we could at least make a sizeable dent in the 1900 tests needed
in one big shot.

- Robert


David Roe

unread,
Apr 9, 2009, 5:38:14 AM4/9/09
to sage-...@googlegroups.com
On Thu, Apr 9, 2009 at 5:19 AM, Robert Bradshaw <robe...@math.washington.edu> wrote:

On Apr 9, 2009, at 1:49 AM, John Cremona wrote:

> Can we at the same time convert more files to ReST and hence get them
> included in the Reference manual?  Although that manual is long, there
> is a whole lot of stuff not mentioned in it, which is bad PR.
>
> What I mean in practice is, if someone is going through a source file
> whose doctests are incomplete and also old-style, then they convert
> each to new-style as they go, as well as (at the same time as) adding
> the new doctests.

That seems like a good idea to me.

I've seen the guide to ReST, but is there a document somewhere summarizing what usually needs to be done to Sage functions?  Is changing EXAMPLES: to EXAMPLES:: and adding a newline after it usually enough?

Sounds like a good idea to me.  I'll try to show up and work on p-adics.  :-)

How about this Saturday?
David

Jason Grout

unread,
Apr 9, 2009, 10:52:31 AM4/9/09
to sage-...@googlegroups.com
David Roe wrote:
> On Thu, Apr 9, 2009 at 5:19 AM, Robert Bradshaw <
> robe...@math.washington.edu> wrote:
>
>> On Apr 9, 2009, at 1:49 AM, John Cremona wrote:
>>
>>> Can we at the same time convert more files to ReST and hence get them
>>> included in the Reference manual? Although that manual is long, there
>>> is a whole lot of stuff not mentioned in it, which is bad PR.
>>>
>>> What I mean in practice is, if someone is going through a source file
>>> whose doctests are incomplete and also old-style, then they convert
>>> each to new-style as they go, as well as (at the same time as) adding
>>> the new doctests.
>> That seems like a good idea to me.
>
>
> I've seen the guide to ReST, but is there a document somewhere summarizing
> what usually needs to be done to Sage functions? Is changing EXAMPLES: to
> EXAMPLES:: and adding a newline after it usually enough?


I thought Mike Hansen had some tools to (largely) automate the
conversion. Were those merged into Sage, or are they available somewhere?

Jason

davidloeffler

unread,
Apr 9, 2009, 2:13:19 PM4/9/09
to sage-devel


On Apr 9, 9:45 am, William Stein <wst...@gmail.com> wrote:
>
> I hope those of you who feel qualified to write doctests, will help
> out.  It's nearly impossible for one person to do all 1900 of those
> doctests in the next month.  Writing doctests is not easy and it
> absolutely requires experience experience with the Sage library, and
> good understanding of the code you are testing.
>
> Thanks,
>

I've had a look at the sage/modular directory, for which doctest
coverage is 74.9% as of 3.4.1.rc1. OK, so this is perhaps one of the
best-doctested parts already, but it's also one of the few I know much
about. I've posted a patch at http://trac.sagemath.org/sage_trac/ticket/5727
which adds 28 new doctests (and fixes 2 small bugs) and adds another
module to the reference manual. More to follow (maybe).

David

William Stein

unread,
Apr 9, 2009, 8:53:02 PM4/9/09
to sage-...@googlegroups.com

Great. I've refereed this (positively). You fixed *numerous* bugs in
the code, imho, when writing those 28 new doctests -- great work!
Let's keep the doctest patches coming.

I'm mostly available on Saturday for a doctest day... so mark your calendars:

SATURDAY, APRIL 11 -- Doctest Day.

-- William

mabshoff

unread,
Apr 9, 2009, 9:32:47 PM4/9/09
to sage-devel


On Apr 9, 5:53 pm, William Stein <wst...@gmail.com> wrote:
> On Thu, Apr 9, 2009 at 11:13 AM, davidloeffler <dave.loeff...@gmail.com> wrote:

<SNIP>

> Great.  I've refereed this (positively).  You fixed *numerous* bugs in
> the code, imho, when writing those 28 new doctests -- great work!
> Let's keep the doctest patches coming.

Yep, note that this fix for David's patch is a common issue:

202 sage: cmp(G, 1)
203 -1
202 sage: cmp(G, 1) in [1,-1]
203 True

cmp($FOO,$BAR) returns -1,0 or 1 depending on the memory address, so
the version that William did is much cleaner since G != 1 in this
case. A lot of those cmp() tests were set to random, but it is bad
style to do that.

> I'm mostly available on Saturday for a doctest day... so mark your calendars:
>
> SATURDAY, APRIL 11 -- Doctest Day.

+1 - someone might want to make an official wiki page for DocDay X and
send out an official announcement.

>  -- William

Cheers,

Michael

William Stein

unread,
Apr 9, 2009, 9:35:15 PM4/9/09
to sage-...@googlegroups.com

Thanks! (It sounds to me like you just volunteered yourself.)

William

William Stein

unread,
Apr 9, 2009, 10:06:01 PM4/9/09
to sage-devel
Hi,

WHERE DO WE STAND?
Here's the Sage doctest coverage status:

We need 296 more function to get to 68% coverage.
We need 756 more function to get to 70% coverage.
We need 1905 more function to get to 75% coverage.

We will have a docday this Saturday, with the insanely ambitious goal
of getting to 70%. This is probably not humanely possible for a
single person to do in 8 hours, so I hope I'm not the only one (I
estimate it takes *at least* 3 minutes per doctest, which comes to
37.8 hours for 756 doctests).

WHAT SHOULD WE DOCTEST?
I'm going to list some ideas for code to "hit", which have bad
coverage and for which there will be a big payoff.

schemes/generic/morphism.py: 2% (1 of 40)
modular/modsym/manin_symbols.py: 2% (2 of 68)
interfaces/kash.py: 3% (1 of 30)
probability/random_variable.py: 3% (1 of 30)
rings/extended_integer_ring.py: 4% (6 of 128)
rings/pari_ring.py: 4% (1 of 22) # could this
be deleted???
structure/formal_sum.py: 4% (1 of 22)
interfaces/cocoa.py: 4% (1 of 23) # delete this,
it doesn't work yet
interfaces/reduce.py: 4% (1 of 23) # delete; it doesn't work
plot/plot3d/base.pyx: 5% (4 of 78)
plot/tachyon.py: 8% (5 of 58)
rings/morphism.pyx: 8% (3 of 35)
structure/generators.pyx: 11% (5 of 45)
categories/pushout.py: 12% (9 of 73)
rings/laurent_series_ring.py: 13% (3 of 23)
plot/plot3d/shapes.pyx: 16% (5 of 31)
gsl/probability_distribution.pyx: 20% (4 of 20) # has a nodoctest at
the top of file. very scary!
schemes/elliptic_curves/monsky_washnitzer.py: 23% (25 of 107) #
doctests here find bad things often, e.g., a bug in FLINT!
modular/hecke/ambient_module.py: 26% (9 of 34)
modular/modsym/ambient.py: 26% (26 of 97)

The above would be 874 doctests, which would solidly get us over 70%.

I've avoided code above that is being refactored, or rumored to be
refactored, or rewritten or whatever. This is all
stuff that just needs to get coveraged.

Alex Ghitza

unread,
Apr 9, 2009, 10:41:17 PM4/9/09
to sage-...@googlegroups.com
>
> We will have a docday this Saturday, with the insanely ambitious goal
> of getting to 70%.  This is probably not humanely possible for a
> single person to do in 8 hours, so I hope I'm not the only one (I
> estimate it takes *at least* 3 minutes per doctest, which comes to
> 37.8 hours for 756 doctests).

This Saturday translates into this Sunday down under, which won't work
for me. But I'll get some stuff done before then.

>
> schemes/generic/morphism.py: 2% (1 of 40)

I already have some code for this file, so I'll try to add to it.
Also, if someone wants to finish reviewing #5631, there's a handful of
doctests there.


Best,
Alex

--
Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne
-- Australia -- http://www.ms.unimelb.edu.au/~aghitza/

William Stein

unread,
Apr 10, 2009, 1:38:30 AM4/10/09
to sage-...@googlegroups.com
On Thu, Apr 9, 2009 at 7:41 PM, Alex Ghitza <agh...@gmail.com> wrote:
>
>>
>> We will have a docday this Saturday, with the insanely ambitious goal
>> of getting to 70%.  This is probably not humanely possible for a
>> single person to do in 8 hours, so I hope I'm not the only one (I
>> estimate it takes *at least* 3 minutes per doctest, which comes to
>> 37.8 hours for 756 doctests).
>

I just noticed that doctesting extended_integer_ring.py would make a
really big dent -- it has 128 functions in it of which only 4% are
doctested now, most functions are very easy to doctest, *and*
extended_rational_field.py, which should be very similar, has 98%
coverage. If somebody wants to easily make a big dent in the
coverage problem, volunteer now to doctest extended_integer_ring.py,
and do it asap:

rings/extended_integer_ring.py: 4% (6 of 128)

rings/extended_rational_field.py: 98% (73 of 74)

-- William

John Cremona

unread,
Apr 10, 2009, 6:09:14 AM4/10/09
to sage-...@googlegroups.com
For me the trouble with extended_*.py is that I have no idea what they
aactually are, so it is hard to get started....

John

2009/4/10 William Stein <wst...@gmail.com>:

David Roe

unread,
Apr 10, 2009, 8:03:38 AM4/10/09
to sage-...@googlegroups.com
I can work on that tomorrow, in addition to p-adics.
David

William Stein

unread,
Apr 10, 2009, 10:20:53 AM4/10/09
to sage-...@googlegroups.com
On Fri, Apr 10, 2009 at 3:09 AM, John Cremona <john.c...@gmail.com> wrote:
>
> For me the trouble with extended_*.py is that I have no idea what they
> aactually are, so it is hard to get started....

But aren't you just dying to find out what they are? :-)

William

John Cremona

unread,
Apr 10, 2009, 10:43:03 AM4/10/09
to sage-...@googlegroups.com
2009/4/10 William Stein <wst...@gmail.com>:
Well... I did not even know that these classes exited until I made
the patch at #5685 (still awaiting review) and had to adjust a couple
of functions in one of those files. I can guess: it somehow adds
+infinity and -infinity to ZZ in a way that makes as much sense as
possible.

But search_src("ExtendedInteger") only picks up these two files, which
suggests to me (maybe wrongly) that they are not actually used
anywhere!

John

>
> William
>
> >
>

William Stein

unread,
Apr 10, 2009, 10:53:02 AM4/10/09
to sage-...@googlegroups.com

They are used. ExtendedInteger is used by the extended_rational_field
file, and that is used in at least two places:

rings/rational_field.py: sage: E = ExtendedRationalField
rings/padics/valuation.py:QQe =
sage.rings.extended_rational_field.ExtendedRationalField

William

John Cremona

unread,
Apr 10, 2009, 11:40:06 AM4/10/09
to sage-...@googlegroups.com
Fair enough, they are used in the padics to hold valuations which can
be infinite. (Though I notice that ZZ(0).valuation(3) is plain old
<class 'sage.rings.infinity.PlusInfinity'>).

Now the padics section is also seriously under-documented. Funny
coincidence, that...

John


>
> William
>
> >
>

David Roe

unread,
Apr 10, 2009, 12:51:11 PM4/10/09
to sage-...@googlegroups.com

:-)  There's a patch in trac (#4637) adding documentation to p-adics, and I'm going to write a bunch of doctests tomorrow.

And yes, I originally wrote Extended Integers and Extended Rationals when I was writing lazy p-adics.  Since lazy p-adics aren't currently in good enough shape to be turned on, I don't think any part of the sage library uses Extended Integers or Extended Rationals.  Upon further reflection, I'm not sure I even need them for lazy p-adics.  I don't know if it's a good idea to just get rid of extended integers and rationals or not.
David

 


John


>
> William
>
> >
>



William Stein

unread,
Apr 10, 2009, 1:08:38 PM4/10/09
to sage-...@googlegroups.com

+1 to getting rid of them. I say we just delete both files. Nobody
knows what they are really, and they aren't needed, and they are
probably partly broken given the bad coverage (at least of integer).
I made this #5735:

http://trac.sagemath.org/sage_trac/ticket/5735

-- William

Robert Bradshaw

unread,
Apr 10, 2009, 2:23:32 PM4/10/09
to sage-...@googlegroups.com

I'll second this. If we need them someday, then we'll accept them
with full coverage. Actually, I think this could probably be better
done lazily and generically rather than re-defining every method that
exists in the integers/rationals.

- Robert

John Cremona

unread,
Apr 10, 2009, 3:16:26 PM4/10/09
to sage-...@googlegroups.com
2009/4/10 Robert Bradshaw <robe...@math.washington.edu>:
+1. Note that at #2515 Burcin already proposed this, pointing out
that they are redundant.

Deleting these two files will increase the overall coverage rather quickly ;)

John

>
> - Robert
>
>
> >
>

mabshoff

unread,
Apr 10, 2009, 5:10:42 PM4/10/09
to sage-devel
FYI: I have started a wiki page at

http://wiki.sagemath.org/doc4

If you are working on any coverage at the moment and/or have any
relevant tickets please add them there so we avoid duplication of
work.

Re p-adics: It would be nice if the doctesting issues at #5499 could
be resolved so that p-adic followup tickets, i.e. #4637 could be
resolved.

Cheers,

Michael

William Stein

unread,
Apr 10, 2009, 7:02:41 PM4/10/09
to sage-...@googlegroups.com
On Fri, Apr 10, 2009 at 2:10 PM, mabshoff <mabs...@googlemail.com> wrote:
>
> FYI: I have started a wiki page at
>
>   http://wiki.sagemath.org/doc4

We will start at 9am tomorrow morning. Latecomers and earlycomers
welcome too. Everyone welcome.

Login to sage.math.washington.edu with ssh
Run irssi

/server irc.freenode.net
/join #sage-devel

-- William

>
> If you are working on any coverage at the moment and/or have any
> relevant tickets please add them there so we avoid duplication of
> work.
>
> Re p-adics: It would be nice if the doctesting issues at #5499 could
> be resolved so that p-adic followup tickets, i.e. #4637 could be
> resolved.
>
> Cheers,
>
> Michael
> >
>



Nicolas M. Thiery

unread,
Apr 14, 2009, 1:56:24 AM4/14/09
to sage-...@googlegroups.com

Please also exclude categories/pushout.py for that reason.

Cheers,

Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

Reply all
Reply to author
Forward
0 new messages