Can we develop a plan to get 100% doctest coverage?
159 views
Skip to first unread message
David Kirkby
Apr 29, 2012, 10:35:34 AM4/29/12
to sage-devel
The fact there are functions in Sage untested is worrying, and I think
its fair to say most people would like to see 100% doctest coverage,
but IMHO, we need a plan for this to happen, and the plan inforced,
otherwise this will not happen soon. So I'm suggesting we collect
ideas for a plan.
Two things that do NOT appear to work, at least in isolation are:
A) Saying coverage should be x % in release Y, then hoping it is.
B) Saying coverage should be x % by the date DD/MM/YY, then hoping it is.
Both of these have been tried, and niether are working. The graph here
http://thales.math.uqam.ca/~labbes/blogue/2011/03/evolution-of-the-overall-doctest-coverage-of-sage/
seems to indicate that progress on this has slowed over the last year
or so, where progress was about 4.7% per year. (I'm not actually sure
I agree with his line of best fit, which would suggest to me the
situation has slowed even more.)
A few ideas that might work, are below. I would add, I'm not
suggesting they are all very good (in particular 1 and 4 are a bit
excessive), but I mention them anyway.
1) Having a release, where the only aim is to increase doctest
coverage. No patches are accepted, unless they increase coverage.
2) Finding sections of code which are not tested, and depreciating
them, unless someone writes the doctests, in which case the
depreciation can be removed.
3) Having a release, where anyone can submit a patch which gets
reviewed, but the release manager does not merge it unless the author
of the patch can provide another patch which increase doctest
coverage.
4) Finding individuals that have written code that is not tested, and
not merging any more patches from them unless they first add tests to
the ALL code they have already written.
5) Finding individuals that have written code that is not tested, and
not merging any more patches from them unless they first add tests to
a number (say 5) tests for all the code they have written which is
untested.
6) Not accepting any patches, unless a equal number of lines of code
are submitted which add doctests.
7) Paying individuals to just write tests.
I believe if we want to achieve 100% coverage, we need to do more than
A and B above, as they are proved not to work.
Can anyone suggest anything else, or have any comments.
Dave
its fair to say most people would like to see 100% doctest coverage,
but IMHO, we need a plan for this to happen, and the plan inforced,
otherwise this will not happen soon. So I'm suggesting we collect
ideas for a plan.
Two things that do NOT appear to work, at least in isolation are:
A) Saying coverage should be x % in release Y, then hoping it is.
B) Saying coverage should be x % by the date DD/MM/YY, then hoping it is.
Both of these have been tried, and niether are working. The graph here
http://thales.math.uqam.ca/~labbes/blogue/2011/03/evolution-of-the-overall-doctest-coverage-of-sage/
seems to indicate that progress on this has slowed over the last year
or so, where progress was about 4.7% per year. (I'm not actually sure
I agree with his line of best fit, which would suggest to me the
situation has slowed even more.)
A few ideas that might work, are below. I would add, I'm not
suggesting they are all very good (in particular 1 and 4 are a bit
excessive), but I mention them anyway.
1) Having a release, where the only aim is to increase doctest
coverage. No patches are accepted, unless they increase coverage.
2) Finding sections of code which are not tested, and depreciating
them, unless someone writes the doctests, in which case the
depreciation can be removed.
3) Having a release, where anyone can submit a patch which gets
reviewed, but the release manager does not merge it unless the author
of the patch can provide another patch which increase doctest
coverage.
4) Finding individuals that have written code that is not tested, and
not merging any more patches from them unless they first add tests to
the ALL code they have already written.
5) Finding individuals that have written code that is not tested, and
not merging any more patches from them unless they first add tests to
a number (say 5) tests for all the code they have written which is
untested.
6) Not accepting any patches, unless a equal number of lines of code
are submitted which add doctests.
7) Paying individuals to just write tests.
I believe if we want to achieve 100% coverage, we need to do more than
A and B above, as they are proved not to work.
Can anyone suggest anything else, or have any comments.
Dave
David Kirkby
Apr 29, 2012, 10:59:30 AM4/29/12
to sage-devel
On 29 April 2012 15:35, David Kirkby <david....@onetel.net> wrote:
> A few ideas that might work, are below. I would add, I'm not
> suggesting they are all very good (in particular 1 and 4 are a bit
> excessive), but I mention them anyway.
Sorry, it was 2 and 4 which I feel are going too far.
> A few ideas that might work, are below. I would add, I'm not
> suggesting they are all very good (in particular 1 and 4 are a bit
> excessive), but I mention them anyway.
This is going ta bit too far, but I mention it for completeness.
> 2) Finding sections of code which are not tested, and depreciating
> them, unless someone writes the doctests, in which case the
> depreciation can be removed.
> 4) Finding individuals that have written code that is not tested, and
> not merging any more patches from them unless they first add tests to
> the ALL code they have already written.
discussion.
Dave
William Stein
Apr 29, 2012, 11:10:10 AM4/29/12
to sage-...@googlegroups.com
On Sunday, April 29, 2012, David Kirkby wrote:
On 29 April 2012 15:35, David Kirkby <david....@onetel.net> wrote:
> A few ideas that might work, are below. I would add, I'm not
> suggesting they are all very good (in particular 1 and 4 are a bit
> excessive), but I mention them anyway.
Sorry, it was 2 and 4 which I feel are going too far.
This is going ta bit too far, but I mention it for completeness.
> 2) Finding sections of code which are not tested, and depreciating
> them, unless someone writes the doctests, in which case the
> depreciation can be removed.
Same with this.
> 4) Finding individuals that have written code that is not tested, and
> not merging any more patches from them unless they first add tests to
> the ALL code they have already written
I would be banned for life!
But I feel the other 5 have some merrit, and all are at least worth of
discussion.
Dave
--
To post to this group, send an email to sage-...@googlegroups.com
To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
David Kirkby
Apr 29, 2012, 12:01:43 PM4/29/12
to sage-...@googlegroups.com
On 29 April 2012 16:10, William Stein <wst...@gmail.com> wrote:
>
>
> On Sunday, April 29, 2012, David Kirkby wrote:
>> > 4) Finding individuals that have written code that is not tested, and
>> > not merging any more patches from them unless they first add tests to
>> > the ALL code they have already written
>
>
>
> I would be banned for life!
Perhaps you would agree to the number 5 I suggested earlier.
>
>
> On Sunday, April 29, 2012, David Kirkby wrote:
>> > 4) Finding individuals that have written code that is not tested, and
>> > not merging any more patches from them unless they first add tests to
>> > the ALL code they have already written
>
>
>
> I would be banned for life!
5) Finding individuals that have written code that is not tested, and
not merging any more patches from them unless they first add tests to
a number (say 5) tests for all the code they have written which is
untested.
I feel #5 is more likely to lead to a 100% doctested Sage than setting
untested.
arbitrary percentages by arbitrary dates or arbitrary version numbers.
Dave
William Stein
Apr 29, 2012, 1:00:17 PM4/29/12
to sage-...@googlegroups.com
On Sunday, April 29, 2012, David Kirkby wrote:
On 29 April 2012 16:10, William Stein <wst...@gmail.com> wrote:
>
>
> On Sunday, April 29, 2012, David Kirkby wrote:
>> > 4) Finding individuals that have written code that is not tested, and
>> > not merging any more patches from them unless they first add tests to
>> > the ALL code they have already written
>
>
>
> I would be banned for life!
Perhaps you would agree to the number 5 I suggested earlier.
5) Finding individuals that have written code that is not tested, and
not merging any more patches from them unless they first add tests to
a number (say 5) tests for all the code they have written which is
untested.
I am opposed to anything that restricts an author from including good new up to snuff code in Sage based on any properties ir behaviour of said author.
I feel #5 is more likely to lead to a 100% doctested Sage than setting
arbitrary percentages by arbitrary dates or arbitrary version numbers.
Dave
--
To post to this group, send an email to sage-...@googlegroups.com
To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Johan Grönqvist
Apr 29, 2012, 1:05:34 PM4/29/12
to sage-...@googlegroups.com
2012-04-29 16:35, David Kirkby skrev:
> The fact there are functions in Sage untested is worrying, and I think
> its fair to say most people would like to see 100% doctest coverage,
> but IMHO, we need a plan for this to happen, and the plan inforced,
> otherwise this will not happen soon. So I'm suggesting we collect
> ideas for a plan.
>
As a humble user, I would like to mention something that could increase
> The fact there are functions in Sage untested is worrying, and I think
> its fair to say most people would like to see 100% doctest coverage,
> but IMHO, we need a plan for this to happen, and the plan inforced,
> otherwise this will not happen soon. So I'm suggesting we collect
> ideas for a plan.
>
the chance of me contributing something. Perhaps it would just increase
the change from 0.5% to 5%, but if many users think like me, that might
still be worthwhile.
My assumption is that it is relatively easy to write a doctest.
It may very well be a bad idea, as it may be a bad user experience for
people expecting a professional-looking and polished math-suite. If so,
feel free to ignore it.
I use a sage 5 beta, and if I would run some code and get a message like
the following:
"This function you just used lacks doctests. Contributing doctests would
be a simple way for you to help improving the quality of sage. If you
have an hour [I do not know what is reasonable for a first doctest
submission] to spare, please have a look at the page [link here] and
consider writing a test for this function."
I expect this might be a performance problem, as a lot of functions may
have to be tested for the presence of doctests on every invocation.
Again, feel free to tell me why it is a bad idea, and then ignore it.
(My only, very small, contribution to sage, was prompted by a confusing
(to me) part of the sage documentation, and I do believe that I would
need to be pointed to simple problems in parts of sage that I use, and
lack of doctests might be such a simple thing.)
Regards
Johan
Nils Bruin
Apr 29, 2012, 2:24:19 PM4/29/12
to sage-devel
On Apr 29, 7:35 am, David Kirkby <david.kir...@onetel.net> wrote:
> The fact there are functions in Sage untested is worrying,
Indeed, it would be nice if all code in Sage would be covered by
> The fact there are functions in Sage untested is worrying,
tests. Having doctests for very many functions probably correlates
with that, so it probably helps to aim for it. However, doctests don't
*perfectly* correlate with covering all code. As an example (from
ecl.pyx):
def __repr__(self):
r"""
...
"""
return "<ECL: "+str(self)+">"
This routine does have a doctest (and counts in the general tally),
but as you see, there is very little to test about this particular
routine and in fact the correct functioning of __repr__ will be
implicitly tested by many of the other doctests in that file.
On the other hand,
cdef void remove_node(cl_object node):
cdef cl_object next, prev
next=cl_cadr(node)
prev=cl_cddr(node)
if next != Cnil:
cl_rplacd(cl_cdr(next),prev)
if prev != Cnil:
cl_rplaca(cl_cdr(prev),next)
does not have a doctest (because it cannot be tested directly from
sage) and does not count for the tally (because it's a cdef). It has a
much worse potential for messing up sage's state, but the doctest
count does not measure whether it's tested. I happen to know it is
tested elsewhere, but the greatest worry -- silent memory corruption
or memory leaks -- is not explicitly covered by those tests.
Don't get hung up on the doctest count. It's a mildly relevant
statistic.
William Stein
Apr 29, 2012, 3:14:37 PM4/29/12
to sage-...@googlegroups.com
Simon King
Apr 29, 2012, 3:58:50 PM4/29/12
to sage-...@googlegroups.com
Hi Johan,
On 2012-04-29, Johan Grönqvist <johan.g...@gmail.com> wrote:
> My assumption is that it is relatively easy to write a doctest.
>
> It may very well be a bad idea, as it may be a bad user experience for
> people expecting a professional-looking and polished math-suite. If so,
> feel free to ignore it.
>
> I use a sage 5 beta, and if I would run some code and get a message like
> the following:
>
> "This function you just used lacks doctests. Contributing doctests would
> be a simple way for you to help improving the quality of sage. If you
> have an hour [I do not know what is reasonable for a first doctest
> submission] to spare, please have a look at the page [link here] and
> consider writing a test for this function."
Writing doc tests is certainly a good way to start contributing to Sage!
> I expect this might be a performance problem, as a lot of functions may
> have to be tested for the presence of doctests on every invocation.
Technically, it would probably be possible to make Python print a
message whenever a function is called that is not tested. But that would
mean a HUGE performance regression, and it would be rather annoying:
Just think of a function that calls an undocumented function in a loop.
However, a variant of your idea might be feasible: When reading the
documentation of an object or when inspecting the sources, such as
sage: ZZ?
or
sage: ZZ??
some functions from sage.misc.sageinspect are called that pre-process
the documentation. Hence, it would be possible to *additionally* make
them test whether there is a doc test - and add a message to the docs
if this is not the case.
Example: Suppose you have a function
def foobar(x,y):
"""
This function does useful stuff.
"""
return x+y
and then do
sage: foobar?
it would be doable that one reads:
Type: function
Base Class: <type 'function'>
String Form: <function foobar at 0x4eb9f50>
Namespace: Interactive
Loaded File: ...
Source File: ...
Definition: foobar(x, y)
Docstring:
Please contribute to Sage by writing a doctest
for this function!
This function does useful stuff.
Here, there is no regression (reading documentation is not
time-critical), and it is clearly doable.
What do people think? Would that kind of message encourage people to do
their first contribution, or would that scare people off?
Best regards,
Simon
On 2012-04-29, Johan Grönqvist <johan.g...@gmail.com> wrote:
> My assumption is that it is relatively easy to write a doctest.
>
> It may very well be a bad idea, as it may be a bad user experience for
> people expecting a professional-looking and polished math-suite. If so,
> feel free to ignore it.
>
> I use a sage 5 beta, and if I would run some code and get a message like
> the following:
>
> "This function you just used lacks doctests. Contributing doctests would
> be a simple way for you to help improving the quality of sage. If you
> have an hour [I do not know what is reasonable for a first doctest
> submission] to spare, please have a look at the page [link here] and
> consider writing a test for this function."
> I expect this might be a performance problem, as a lot of functions may
> have to be tested for the presence of doctests on every invocation.
message whenever a function is called that is not tested. But that would
mean a HUGE performance regression, and it would be rather annoying:
Just think of a function that calls an undocumented function in a loop.
However, a variant of your idea might be feasible: When reading the
documentation of an object or when inspecting the sources, such as
sage: ZZ?
or
sage: ZZ??
some functions from sage.misc.sageinspect are called that pre-process
the documentation. Hence, it would be possible to *additionally* make
them test whether there is a doc test - and add a message to the docs
if this is not the case.
Example: Suppose you have a function
def foobar(x,y):
"""
This function does useful stuff.
"""
return x+y
and then do
sage: foobar?
it would be doable that one reads:
Type: function
Base Class: <type 'function'>
String Form: <function foobar at 0x4eb9f50>
Namespace: Interactive
Loaded File: ...
Source File: ...
Definition: foobar(x, y)
Docstring:
Please contribute to Sage by writing a doctest
for this function!
This function does useful stuff.
Here, there is no regression (reading documentation is not
time-critical), and it is clearly doable.
What do people think? Would that kind of message encourage people to do
their first contribution, or would that scare people off?
Best regards,
Simon
David Kirkby
Apr 29, 2012, 6:34:29 PM4/29/12
to sage-...@googlegroups.com
On 29 April 2012 20:58, Simon King <simon...@uni-jena.de> wrote:
> sage: foobar?
> it would be doable that one reads:
> Type: function
> Base Class: <type 'function'>
> String Form: <function foobar at 0x4eb9f50>
> Namespace: Interactive
> Loaded File: ...
> Source File: ...
> Definition: foobar(x, y)
> Docstring:
> Please contribute to Sage by writing a doctest
> for this function!
>
> This function does useful stuff.
>
> Here, there is no regression (reading documentation is not
> time-critical), and it is clearly doable.
> What do people think? Would that kind of message encourage people to do
> their first contribution, or would that scare people off?
I think it might get people to contribute. It needs a link to a page
> sage: foobar?
> it would be doable that one reads:
> Type: function
> Base Class: <type 'function'>
> String Form: <function foobar at 0x4eb9f50>
> Namespace: Interactive
> Loaded File: ...
> Source File: ...
> Definition: foobar(x, y)
> Docstring:
> Please contribute to Sage by writing a doctest
> for this function!
>
> This function does useful stuff.
>
> Here, there is no regression (reading documentation is not
> time-critical), and it is clearly doable.
> What do people think? Would that kind of message encourage people to do
> their first contribution, or would that scare people off?
describing in detail the process of writing a doctest for Sage, and
how to get that doctest into Sage.
Perhaps it woud scare a few people off, but overall I would have
thought the result positive.
> Best regards,
> Simon
Currently there seems to be no plan in place to get Sage doctested.
William has rejected the idea of getting people to write 5 doctests
for previous code they have written, before a patch of theirs gets
merged. So the suggestion here might go some way to solving the
problem, though I personally think it needs a more drastic approach.
But this is certainly a step in the right direction.
Dave
David Kirkby
Apr 29, 2012, 6:36:58 PM4/29/12
to sage-...@googlegroups.com
On 29 April 2012 18:00, William Stein <wst...@gmail.com> wrote:
>
>
> On Sunday, April 29, 2012, David Kirkby wrote:
>>
>> Perhaps you would agree to the number 5 I suggested earlier.
>>
>> 5) Finding individuals that have written code that is not tested, and
>> not merging any more patches from them unless they first add tests to
>> a number (say 5) tests for all the code they have written which is
>> untested.
>>
>
> I am opposed to anything that restricts an author from including good new up
> to snuff code in Sage based on any properties ir behaviour of said author.
Would you be opposed to insisting someone write one doctest for
>
>
> On Sunday, April 29, 2012, David Kirkby wrote:
>>
>> Perhaps you would agree to the number 5 I suggested earlier.
>>
>> 5) Finding individuals that have written code that is not tested, and
>> not merging any more patches from them unless they first add tests to
>> a number (say 5) tests for all the code they have written which is
>> untested.
>>
>
> I am opposed to anything that restricts an author from including good new up
> to snuff code in Sage based on any properties ir behaviour of said author.
previous code they have written, for every new patch they want merged?
I originally said 5, but would you find 1 acceptable?
Dave
William Stein
Apr 29, 2012, 6:49:37 PM4/29/12
to sage-...@googlegroups.com
On Sunday, April 29, 2012, David Kirkby wrote:
On 29 April 2012 18:00, William Stein <wst...@gmail.com> wrote:
>
>
> On Sunday, April 29, 2012, David Kirkby wrote:
>>
>> Perhaps you would agree to the number 5 I suggested earlier.
>>
>> 5) Finding individuals that have written code that is not tested, and
>> not merging any more patches from them unless they first add tests to
>> a number (say 5) tests for all the code they have written which is
>> untested.
>>
>
> I am opposed to anything that restricts an author from including good new up
> to snuff code in Sage based on any properties ir behaviour of said author.
Would you be opposed to insisting someone write one doctest for
previous code they have written, for every new patch they want merged?
I am opposed. Moreover all code that has gone into sage for the last about 4 years has already had 100% coverage (or a major error was made when refereeing); since there have been many (most?) new developers since then, this proposal would have little impact.
I originally said 5, but would you find 1 acceptable?
Dave
--
To post to this group, send an email to sage-...@googlegroups.com
To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Benjamin Jones
Apr 30, 2012, 12:32:59 AM4/30/12
to sage-...@googlegroups.com
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
Option 1) seems like a good idea to me. The others might also work,
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
but would surely require a lot of extra work on the part the release
manager or whoever ends up having to enforce / implement the
restrictions. That extra work would be better funneled into increasing
doctest coverage in my opinion.
We might consider combining option 1 and plan A, as you put it, but
set a realistic target %.
--
Benjamin Jones
David Kirkby
Apr 30, 2012, 1:24:11 AM4/30/12
to sage-...@googlegroups.com
> Option 1) seems like a good idea to me. The others might also work,
> but would surely require a lot of extra work on the part the release
> manager or whoever ends up having to enforce / implement the
> restrictions. That extra work would be better funneled into increasing
> doctest coverage in my opinion.
>
> We might consider combining option 1 and plan A, as you put it, but
> set a realistic target %.
>
> --
> Benjamin Jones
Thank you Benjamin. Both ideas of yours seem good to me, but I don't
> but would surely require a lot of extra work on the part the release
> manager or whoever ends up having to enforce / implement the
> restrictions. That extra work would be better funneled into increasing
> doctest coverage in my opinion.
>
> We might consider combining option 1 and plan A, as you put it, but
> set a realistic target %.
>
> --
> Benjamin Jones
think they will work. William has said
==========
"I am opposed to anything that restricts an author from including good
new up to snuff code in Sage based on any properties ir behaviour of
said author."
==========
new up to snuff code in Sage based on any properties ir behaviour of
said author."
I can't see a way forward in this case. I guess we will asymptotically
approach 100%, if all new code is tested!
Personally, I would have thought it a "selling point" if we could say
"Every function in Sage is tested on numerous platforms on every
release", but alas I don't think that will happen.
I give up.
Dave
Simon King
Apr 30, 2012, 11:02:45 AM4/30/12
to sage-devel
Hi!
On 29 Apr., 21:58, Simon King <simon.k...@uni-jena.de> wrote:
> However, a variant of your idea might be feasible: When reading the
> documentation of an object or when inspecting the sources, such as
> sage: ZZ?
> or
> sage: ZZ??
> some functions from sage.misc.sageinspect are called that pre-process
> the documentation. Hence, it would be possible to *additionally* make
> them test whether there is a doc test - and add a message to the docs
> if this is not the case.
The problem is that apparently the notebook and the command line
interface of Sage use different ways for getting the docs: There seems
to be
SAGE_ROOT/devel/sagenb/sagenb/misc/sageinspect.py and
SAGE_ROOT/devel/sagenb/sagenb/misc/support.py
for the notebook, and
SAGE_ROOT/devel/sage/sage/misc/sageinspect.py and
SAGE_ROOT/devel/sage/sage/misc/sagedoc.py
on the other hand.
What an awkward and difficult to maintain duplication of code!
Cheers,
Simon
On 29 Apr., 21:58, Simon King <simon.k...@uni-jena.de> wrote:
> However, a variant of your idea might be feasible: When reading the
> documentation of an object or when inspecting the sources, such as
> sage: ZZ?
> or
> sage: ZZ??
> some functions from sage.misc.sageinspect are called that pre-process
> the documentation. Hence, it would be possible to *additionally* make
> them test whether there is a doc test - and add a message to the docs
> if this is not the case.
interface of Sage use different ways for getting the docs: There seems
to be
SAGE_ROOT/devel/sagenb/sagenb/misc/sageinspect.py and
SAGE_ROOT/devel/sagenb/sagenb/misc/support.py
for the notebook, and
SAGE_ROOT/devel/sage/sage/misc/sageinspect.py and
SAGE_ROOT/devel/sage/sage/misc/sagedoc.py
on the other hand.
What an awkward and difficult to maintain duplication of code!
Cheers,
Simon
Simon King
Apr 30, 2012, 11:47:22 AM4/30/12
to sage-devel
FWIW: I created trac ticket #12891 (http://trac.sagemath.org/sage_trac/
ticket/12891), if someone likes the idea to invite users to contribute
when reading the docs of an undocumented function or method.
Cheers,
Simon
ticket/12891), if someone likes the idea to invite users to contribute
when reading the docs of an undocumented function or method.
Cheers,
Simon
Dr. David Kirkby
Apr 30, 2012, 4:44:38 PM4/30/12
to sage-...@googlegroups.com
mathematician, I might be able to write doctests for some of them. I think
William, since he started the project, has probably contributed a fair number of
undocumented functions. Since they were present at the start of Sage, I assume
that mathematically they are not so advanced as some of the later material.
Dave
William Stein
Apr 30, 2012, 4:50:47 PM4/30/12
to sage-...@googlegroups.com
On Mon, Apr 30, 2012 at 1:44 PM, Dr. David Kirkby
<david....@onetel.net> wrote:
> On 04/30/12 04:47 PM, Simon King wrote:
>>
>> FWIW: I created trac ticket #12891 (http://trac.sagemath.org/sage_trac/
>> ticket/12891), if someone likes the idea to invite users to contribute
>> when reading the docs of an undocumented function or method.
>>
>> Cheers,
>> Simon
>>
>
> Is there an easy way to find a list of undocumented functions?
sage -coverage
<david....@onetel.net> wrote:
> On 04/30/12 04:47 PM, Simon King wrote:
>>
>> FWIW: I created trac ticket #12891 (http://trac.sagemath.org/sage_trac/
>> ticket/12891), if someone likes the idea to invite users to contribute
>> when reading the docs of an undocumented function or method.
>>
>> Cheers,
>> Simon
>>
>
> Is there an easy way to find a list of undocumented functions?
For example,
cd SAGE_ROOT/devel/sage/sage/coding
sage -coverage .
Outputs:
ag_code.py: 100% (1 of 1)
binary_code.pyx: 91% (41 of 45)
code_bounds.py: 100% (17 of 17)
code_constructions.py: 96% (25 of 26)
decoder.py: 100% (3 of 3)
guava.py: 100% (3 of 3)
linear_code.py: 82% (52 of 63)
sd_codes.py: 20% (1 of 5)
source_coding/huffman.py: 100% (9 of 9)
Overall weighted coverage score: 88.1%
Total number of functions: 172
We need 3 more function to get to 90% coverage.
We need 11 more function to get to 95% coverage.
We need 18 more function to get to 99% coverage.
---
You can then do the following to see what is wrong with a given file:
sage -coverage sd_codes.py
----------------------------------------------------------------------
sd_codes.py
SCORE sd_codes.py: 20% (1 of 5)
Missing documentation:
* MS(n):
* matA(n):
* matId(n):
* MS2(n):
> Whilst I'm
> not a mathematician, I might be able to write doctests for some of them. I
> think William, since he started the project, has probably contributed a fair
> number of undocumented functions. Since they were present at the start of
> Sage, I assume that mathematically they are not so advanced as some of the
> later material.
>
> Dave
>
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to
> sage-devel+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to
> sage-devel+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
rjf
Apr 30, 2012, 8:04:14 PM4/30/12
to sage-...@googlegroups.com
If you said "We want to assure that each command (or each software "unit") has at least 1 test" and
asked for enforcement proposals, maybe this makes a little bit of sense.
"100% coverage" is not what I'd call it.
Having people who wrote the code be the only person who tests it is not a great idea.
Neither is having people who know nothing much at all about the code write a test.
Does UW have a software engineering course??
RJF
William Stein
Apr 30, 2012, 8:51:12 PM4/30/12
to sage-...@googlegroups.com
On Monday, April 30, 2012, rjf wrote:
If you said "We want to assure that each command (or each software "unit") has at least 1 test" and
asked for enforcement proposals, maybe this makes a little bit of sense.
"100% coverage" is not what I'd call it.
What would you call it?
Having people who wrote the code be the only person who tests it is not a great idea.
Neither is having people who know nothing much at all about the code write a test.
Does UW have a software engineering course??
Nope.
RJF
--
To post to this group, send an email to sage-...@googlegroups.com
To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
David Kirkby
Apr 30, 2012, 9:04:01 PM4/30/12
to sage-...@googlegroups.com
On 1 May 2012 01:04, rjf <fat...@gmail.com> wrote:
>
>
>
> If you said "We want to assure that each command (or each software "unit")
> has at least 1 test" and
> asked for enforcement proposals, maybe this makes a little bit of sense.
> "100% coverage" is not what I'd call it.
It's not 100% branch coverage, or any of the other metrics which can
>
>
>
> If you said "We want to assure that each command (or each software "unit")
> has at least 1 test" and
> asked for enforcement proposals, maybe this makes a little bit of sense.
> "100% coverage" is not what I'd call it.
be used. There are several definitions of "coverage", but then I guess
you know that.
> Having people who wrote the code be the only person who tests it is not a
> great idea.
what is happening with 100's of functions.
> Neither is having people who know nothing much at all about the code write a
> test.
> Does UW have a software engineering course??
software engineering, and even suggested he buy a copy each for the
main developers. I have personally bought a couple of books on the
subject.
http://www.amazon.co.uk/Software-Engineering-International-Ian-Sommerville/dp/0137053460/ref=sr_1_1?s=books&ie=UTF8&qid=1335833610&sr=1-1
http://www.amazon.co.uk/Software-Engineering-A-Practitioners-Approach/dp/0071267824/ref=sr_1_3?s=books&ie=UTF8&qid=1335833610&sr=1-3
I've spent a lot of time porting Sage to Solaris, but I'm reluctant to
use the program professionally, due to what I personally perceive as a
lack of testing. So I still use Mathematica, and will probably
continue to do so.
In theory, being open-source, I can check the code of Sage, but in
practice it's not practical, so the open-source nature is not much
benefit to me, apart from the cost.
> RJF
Dave
R. Andrew Ohana
Apr 30, 2012, 9:10:36 PM4/30/12
to sage-...@googlegroups.com
On Mon, Apr 30, 2012 at 17:51, William Stein <wst...@gmail.com> wrote:
>
>
> On Monday, April 30, 2012, rjf wrote:
>>
>>
>>
>>
>> If you said "We want to assure that each command (or each software "unit")
>> has at least 1 test" and
>> asked for enforcement proposals, maybe this makes a little bit of sense.
>> "100% coverage" is not what I'd call it.
>
>
> What would you call it?
>
>>
>>
>> Having people who wrote the code be the only person who tests it is not a
>> great idea.
>> Neither is having people who know nothing much at all about the code write
>> a test.
>>
>> Does UW have a software engineering course??
>
>
> Nope.
Yes it does:
>
>
> On Monday, April 30, 2012, rjf wrote:
>>
>>
>>
>>
>> If you said "We want to assure that each command (or each software "unit")
>> has at least 1 test" and
>> asked for enforcement proposals, maybe this makes a little bit of sense.
>> "100% coverage" is not what I'd call it.
>
>
> What would you call it?
>
>>
>>
>> Having people who wrote the code be the only person who tests it is not a
>> great idea.
>> Neither is having people who know nothing much at all about the code write
>> a test.
>>
>> Does UW have a software engineering course??
>
>
> Nope.
http://www.cs.washington.edu/education/courses/cse403/
http://www.cs.washington.edu/education/courses/cse503/
>
>>
>>
>> RJF
>>
>>
>>
>> --
>> To post to this group, send an email to sage-...@googlegroups.com
>> To unsubscribe from this group, send an email to
>> sage-devel+...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/sage-devel
>> URL: http://www.sagemath.org
>
>
>
> --
> William Stein
> Professor of Mathematics
> University of Washington
> http://wstein.org
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to
> sage-devel+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
--
David Kirkby
Apr 30, 2012, 9:12:00 PM4/30/12
to sage-...@googlegroups.com
On 1 May 2012 01:51, William Stein <wst...@gmail.com> wrote:
>
>
> On Monday, April 30, 2012, rjf wrote:
>>
>>
>>
>>
>> If you said "We want to assure that each command (or each software "unit")
>> has at least 1 test" and
>> asked for enforcement proposals, maybe this makes a little bit of sense.
>> "100% coverage" is not what I'd call it.
>
>
> What would you call it?
It is one defintion, but there are many others too, which deman more,
>
>
> On Monday, April 30, 2012, rjf wrote:
>>
>>
>>
>>
>> If you said "We want to assure that each command (or each software "unit")
>> has at least 1 test" and
>> asked for enforcement proposals, maybe this makes a little bit of sense.
>> "100% coverage" is not what I'd call it.
>
>
> What would you call it?
such as that every branch is covered, or every line of code gets
executed. That does not happen with a doc test - at least in general -
it might happen for the odd test.
>> Does UW have a software engineering course??
>
>
> Nope.
Engineering Group"
http://www.cs.washington.edu/research/plse
Dave
William Stein
Apr 30, 2012, 9:26:17 PM4/30/12
to sage-...@googlegroups.com
On Mon, Apr 30, 2012 at 6:10 PM, R. Andrew Ohana <andrew...@gmail.com> wrote:
> On Mon, Apr 30, 2012 at 17:51, William Stein <wst...@gmail.com> wrote:
>>
>>
>> On Monday, April 30, 2012, rjf wrote:
>>>
>>>
>>>
>>>
>>> If you said "We want to assure that each command (or each software "unit")
>>> has at least 1 test" and
>>> asked for enforcement proposals, maybe this makes a little bit of sense.
>>> "100% coverage" is not what I'd call it.
>>
>>
>> What would you call it?
>>
>>>
>>>
>>> Having people who wrote the code be the only person who tests it is not a
>>> great idea.
>>> Neither is having people who know nothing much at all about the code write
>>> a test.
>>>
>>> Does UW have a software engineering course??
>>
>>
>> Nope.
>
> Yes it does:
> http://www.cs.washington.edu/education/courses/cse403/
> http://www.cs.washington.edu/education/courses/cse503/
I was being facetious since his question was so absurd. Sorry you missed that.
> On Mon, Apr 30, 2012 at 17:51, William Stein <wst...@gmail.com> wrote:
>>
>>
>> On Monday, April 30, 2012, rjf wrote:
>>>
>>>
>>>
>>>
>>> If you said "We want to assure that each command (or each software "unit")
>>> has at least 1 test" and
>>> asked for enforcement proposals, maybe this makes a little bit of sense.
>>> "100% coverage" is not what I'd call it.
>>
>>
>> What would you call it?
>>
>>>
>>>
>>> Having people who wrote the code be the only person who tests it is not a
>>> great idea.
>>> Neither is having people who know nothing much at all about the code write
>>> a test.
>>>
>>> Does UW have a software engineering course??
>>
>>
>> Nope.
>
> Yes it does:
> http://www.cs.washington.edu/education/courses/cse403/
> http://www.cs.washington.edu/education/courses/cse503/
William
Keshav Kini
May 1, 2012, 7:05:57 AM5/1/12
to sage-...@googlegroups.com
David Kirkby <david....@onetel.net> writes:
> I don't know, but I have in the past suggested William read a book on
> software engineering, and even suggested he buy a copy each for the
> main developers. I have personally bought a couple of books on the
> subject.
>
> http://www.amazon.co.uk/Software-Engineering-International-Ian-Sommerville/dp/0137053460/ref=sr_1_1?s=books&ie=UTF8&qid=1335833610&sr=1-1
> http://www.amazon.co.uk/Software-Engineering-A-Practitioners-Approach/dp/0071267824/ref=sr_1_3?s=books&ie=UTF8&qid=1335833610&sr=1-3
So would you say that you recommend these books?
> I don't know, but I have in the past suggested William read a book on
> software engineering, and even suggested he buy a copy each for the
> main developers. I have personally bought a couple of books on the
> subject.
>
> http://www.amazon.co.uk/Software-Engineering-International-Ian-Sommerville/dp/0137053460/ref=sr_1_1?s=books&ie=UTF8&qid=1335833610&sr=1-1
> http://www.amazon.co.uk/Software-Engineering-A-Practitioners-Approach/dp/0071267824/ref=sr_1_3?s=books&ie=UTF8&qid=1335833610&sr=1-3
-Keshav
----
Join us in #sagemath on irc.freenode.net !
William Stein
May 1, 2012, 9:44:19 AM5/1/12
to sage-...@googlegroups.com
On Monday, April 30, 2012, rjf wrote:
If you said "We want to assure that each command (or each software "unit") has at least 1 test" and
asked for enforcement proposals, maybe this makes a little bit of sense.
"100% coverage" is not what I'd call it.
What would you call it?
Having people who wrote the code be the only person who tests it is not a great idea.
Neither is having people who know nothing much at all about the code write a test.
Does UW have a software engineering course??
Nope.
Sébastien Labbé
May 1, 2012, 3:49:42 PM5/1/12
to sage-devel
> Both of these have been tried, and niether are working. The graph here
>
> http://thales.math.uqam.ca/~labbes/blogue/2011/03/evolution-of-the-ov...
>
>
> seems to indicate that progress on this has slowed over the last year
> or so, where progress was about 4.7% per year. (I'm not actually sure
> I agree with his line of best fit, which would suggest to me the
> situation has slowed even more.)
I updated my graph (and removed extrapolations...) :
> seems to indicate that progress on this has slowed over the last year
> or so, where progress was about 4.7% per year. (I'm not actually sure
> I agree with his line of best fit, which would suggest to me the
> situation has slowed even more.)
http://www.thales.math.uqam.ca/~labbes/blogue/2012/05/evolution-of-the-overall-doctest-coverage-of-sage/
In Sage 4.7 : 23769 out of 27833 were doctested. Undoctested : 4064.
In Sage 4.8 : 24601 out of 28573 were doctested. Undoctested : 3972.
During this period, the number of doctested functions increased by
832.
But, the number of undoctested functions decreased only by 92.
Sébasten
Sébastien Labbé
May 1, 2012, 3:51:32 PM5/1/12
to sage-devel
> But, the number of undoctested functions decreased only by 92.
Also, I believe we should stop looking at the coverage percentage
between releases put rather at the number of undoctested functions.
SL
Dr. David Kirkby
May 1, 2012, 4:32:19 PM5/1/12
to sage-...@googlegroups.com
On 05/ 1/12 08:51 PM, S�bastien Labb� wrote:
>> But, the number of undoctested functions decreased only by 92.
Well, I guess that's better than nothing, but given there are still around 4000
>> But, the number of undoctested functions decreased only by 92.
which are not tested, it hardly makes a dent in the problem.
> Also, I believe we should stop looking at the coverage percentage
> between releases put rather at the number of undoctested functions.
>
> SL
>
Otherwise, as I stated earlier, the doctest coverage will asymptotically
approach 100%, even if no further doctests are added to older code. (This
assumes all new functions have at least one test - which I believe is happening.)
The second graph at
http://www.thales.math.uqam.ca/~labbes/blogue/2012/05/evolution-of-the-overall-doctest-coverage-of-sage/
makes interesting reading. It appears that the number of untested functions
dropped from 12000 to 5000 in about 18 months, but in the following 3 years, it
only dropped a further 500 or so.
It seems there was a period (late 2007 to early 2009) when an effort was made to
reduce the number of untested functions. But that impetus has been lost.
I must admit, I was not aware there were several thousand untested functions.
Dave
Benjamin Jones
May 1, 2012, 5:33:34 PM5/1/12
to sage-...@googlegroups.com
Here [1] is a patch I just wrote to increase the doctest coverage of
one file mentioned here (linear_code.py) to 100%. Please review! I'm
going to work on the other files in that directory as I have time.
[1] http://trac.sagemath.org/sage_trac/ticket/12893
--
Benjamin Jones
Nicolas M. Thiery
May 1, 2012, 6:15:16 PM5/1/12
to sage-...@googlegroups.com
On Tue, May 01, 2012 at 09:32:19PM +0100, Dr. David Kirkby wrote:
> Otherwise, as I stated earlier, the doctest coverage will
> asymptotically approach 100%, even if no further doctests are added
> to older code. (This assumes all new functions have at least one
> test - which I believe is happening.)
I definitely would love to achieve 100% test coverage (and better
> Otherwise, as I stated earlier, the doctest coverage will
> asymptotically approach 100%, even if no further doctests are added
> to older code. (This assumes all new functions have at least one
> test - which I believe is happening.)
branch coverage, not just doctest coverage).
Now, I personally would put the emphasis on encouraging the
*refactoring* of the early Sage code (of course with appropriate
testing), rather than just doctesting it. There is for example not
much point adding a doctest on a function that is likely to be
discarded in the refactoring.
I don't have good ideas on how to promote refactoring or doctesting,
beside rewarding (at least by a congratulation) any volunteer doing
so.
Cheers,
Nicolas
PS: by the above I mean no criticizing of early Sage code: it had to
be written quickly for the project to get started, and many tools to
write it nicely were just not available.
--
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/
William Stein
May 1, 2012, 6:42:38 PM5/1/12
to sage-...@googlegroups.com
On Tue, May 1, 2012 at 3:15 PM, Nicolas M. Thiery
<Nicolas...@u-psud.fr> wrote:
> On Tue, May 01, 2012 at 09:32:19PM +0100, Dr. David Kirkby wrote:
>> Otherwise, as I stated earlier, the doctest coverage will
>> asymptotically approach 100%, even if no further doctests are added
>> to older code. (This assumes all new functions have at least one
>> test - which I believe is happening.)
>
> I definitely would love to achieve 100% test coverage (and better
> branch coverage, not just doctest coverage).
And Nicolas, I applaud you for ensuring 100% coverage of the combinat/ codebase!
<Nicolas...@u-psud.fr> wrote:
> On Tue, May 01, 2012 at 09:32:19PM +0100, Dr. David Kirkby wrote:
>> Otherwise, as I stated earlier, the doctest coverage will
>> asymptotically approach 100%, even if no further doctests are added
>> to older code. (This assumes all new functions have at least one
>> test - which I believe is happening.)
>
> I definitely would love to achieve 100% test coverage (and better
> branch coverage, not just doctest coverage).
-- William
>
> Now, I personally would put the emphasis on encouraging the
> *refactoring* of the early Sage code (of course with appropriate
> testing), rather than just doctesting it. There is for example not
> much point adding a doctest on a function that is likely to be
> discarded in the refactoring.
>
> I don't have good ideas on how to promote refactoring or doctesting,
> beside rewarding (at least by a congratulation) any volunteer doing
> so.
>
> Cheers,
> Nicolas
>
> PS: by the above I mean no criticizing of early Sage code: it had to
> be written quickly for the project to get started, and many tools to
> write it nicely were just not available.
>
> --
> http://Nicolas.Thiery.name/
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
Dr. David Kirkby
May 1, 2012, 8:26:53 PM5/1/12
to sage-...@googlegroups.com
On 05/ 1/12 10:33 PM, Benjamin Jones wrote:
> Here [1] is a patch I just wrote to increase the doctest coverage of
> one file mentioned here (linear_code.py) to 100%. Please review! I'm
> going to work on the other files in that directory as I have time.
>
> [1] http://trac.sagemath.org/sage_trac/ticket/12893
>
> --
> Benjamin Jones
>
Great. I'm not able to review it, but hopefully someone else can.
> Here [1] is a patch I just wrote to increase the doctest coverage of
> one file mentioned here (linear_code.py) to 100%. Please review! I'm
> going to work on the other files in that directory as I have time.
>
> [1] http://trac.sagemath.org/sage_trac/ticket/12893
>
> --
> Benjamin Jones
>
Dave
John Cremona
May 2, 2012, 4:43:10 AM5/2/12
to sage-...@googlegroups.com
This one example (#12893) is a good illustration of how non-trivial a
task it is to add doctests, and also how important it is to do so:
not ony was a (small) bug found in one the the newly tested functions,
but another bug was revealed somewhere else in Sage. So adding
doctests is extremely likely to end up in identifying and (hopefully)
fixing bugs, which can be hard and time-consuming, and certainly
involves some knowledge of what the functions are really doing.
That said, there is also a use for the testing which (I think) David K
did a while back which was to fire random input at random functions to
see what happens, with somewhat embarrassing results for some function
I wrote which have not all been fixed yet. (e.g. mwrank('0 0 0 0')
hangs forever because -- as everyone is supposed to know -- mwrank
wants 5 integers input, and it is waiting for a 5th, but will never
get it. In this example I did write a patch which I thought solved
the problem 19 months ago, but it was not approved and nothing has
been done on that since....)
John
task it is to add doctests, and also how important it is to do so:
not ony was a (small) bug found in one the the newly tested functions,
but another bug was revealed somewhere else in Sage. So adding
doctests is extremely likely to end up in identifying and (hopefully)
fixing bugs, which can be hard and time-consuming, and certainly
involves some knowledge of what the functions are really doing.
That said, there is also a use for the testing which (I think) David K
did a while back which was to fire random input at random functions to
see what happens, with somewhat embarrassing results for some function
I wrote which have not all been fixed yet. (e.g. mwrank('0 0 0 0')
hangs forever because -- as everyone is supposed to know -- mwrank
wants 5 integers input, and it is waiting for a 5th, but will never
get it. In this example I did write a patch which I thought solved
the problem 19 months ago, but it was not approved and nothing has
been done on that since....)
John
David Kirkby
May 2, 2012, 5:36:43 AM5/2/12
to sage-...@googlegroups.com
On 2 May 2012 09:43, John Cremona <john.c...@gmail.com> wrote:
> This one example (#12893) is a good illustration of how non-trivial a
> task it is to add doctests, and also how important it is to do so:
> not ony was a (small) bug found in one the the newly tested functions,
> but another bug was revealed somewhere else in Sage. So adding
> doctests is extremely likely to end up in identifying and (hopefully)
> fixing bugs, which can be hard and time-consuming,
Yes, I agree.
> This one example (#12893) is a good illustration of how non-trivial a
> task it is to add doctests, and also how important it is to do so:
> not ony was a (small) bug found in one the the newly tested functions,
> but another bug was revealed somewhere else in Sage. So adding
> doctests is extremely likely to end up in identifying and (hopefully)
> fixing bugs, which can be hard and time-consuming,
I wish I could convince William a bit more of the importance of this.
> and certainly
> involves some knowledge of what the functions are really doing.
though I'm not totally mathematically retarded. Number theory used to
be a hobby of mine, but not any more.
> That said, there is also a use for the testing which (I think) David K
> did a while back which was to fire random input at random functions to
> see what happens, with somewhat embarrassing results for some function
> I wrote which have not all been fixed yet. (e.g. mwrank('0 0 0 0')
> hangs forever because -- as everyone is supposed to know -- mwrank
> wants 5 integers input, and it is waiting for a 5th, but will never
> get it.
19 months ago I found:
sage: seed(1,2)
will crash Sage, on every platform I checked (Linux, OS X and OpenSolaris).
http://trac.sagemath.org/sage_trac/ticket/10113
I've just checked the latest beta (only on Linux) and again it crashes
Sage. I guess it will crash on Solaris and OS X too, though I don't
have time to check.
It's possible that indicates an underlying problem, that is not
specific to calling "seed" incorrectly, but I just happened to find it
using "seed".
If anyone else has some spare CPU cycles, they could try writing code
to crash Sage. It would be best if they did not use mine, but
developed their own, as I suspect each person would approach it a
different way, and be more likely to find bugs.
> In this example I did write a patch which I thought solved
> the problem 19 months ago, but it was not approved and nothing has
> been done on that since....)
> John
fully solve the problem, but at least partially solved it. Can you
give me the number, and I'll take a look again. I guess if it
partially solves a problem, but not fully solves it, the patch is
better than no patch. I don't have time to look now - I am at work.
Dave
John Cremona
May 2, 2012, 6:10:48 AM5/2/12
to sage-...@googlegroups.com
On 2 May 2012 10:36, David Kirkby <david....@onetel.net> wrote:
> On 2 May 2012 09:43, John Cremona <john.c...@gmail.com> wrote:
>> This one example (#12893) is a good illustration of how non-trivial a
>> task it is to add doctests, and also how important it is to do so:
>> not ony was a (small) bug found in one the the newly tested functions,
>> but another bug was revealed somewhere else in Sage. So adding
>> doctests is extremely likely to end up in identifying and (hopefully)
>> fixing bugs, which can be hard and time-consuming,
>
> Yes, I agree.
>
> I wish I could convince William a bit more of the importance of this.
I'm sure that William agrees entirely with what I said.
> On 2 May 2012 09:43, John Cremona <john.c...@gmail.com> wrote:
>> This one example (#12893) is a good illustration of how non-trivial a
>> task it is to add doctests, and also how important it is to do so:
>> not ony was a (small) bug found in one the the newly tested functions,
>> but another bug was revealed somewhere else in Sage. So adding
>> doctests is extremely likely to end up in identifying and (hopefully)
>> fixing bugs, which can be hard and time-consuming,
>
> Yes, I agree.
>
> I wish I could convince William a bit more of the importance of this.
If the best solution is an upstream change to mwrank's input parsing,
then I am willing to do what is needed. We are going to upgrade the
version of eclib (which includes mwrank) in sage in the next release
anyway, as in ticket #10993. Currently, using the default way to
input integers into C++, all whitespace including newlines is ignored,
so whenever the input is incomplete mwrank just waits for the rest,
and this never happens since Sage only ever sends one line of input.
John
Simon King
May 2, 2012, 6:33:15 AM5/2/12
to sage-devel
Hi David,
On 2 Mai, 11:36, David Kirkby <david.kir...@onetel.net> wrote:
> sage: seed(1,2)
>
> will crash Sage, on every platform I checked (Linux, OS X and OpenSolaris).
>
> http://trac.sagemath.org/sage_trac/ticket/10113
I uploaded a patch that solves the problem and needs review (and adds
seed(1,2) as doctest, of course...)
Cheers,
Simon
On 2 Mai, 11:36, David Kirkby <david.kir...@onetel.net> wrote:
> sage: seed(1,2)
>
> will crash Sage, on every platform I checked (Linux, OS X and OpenSolaris).
>
> http://trac.sagemath.org/sage_trac/ticket/10113
seed(1,2) as doctest, of course...)
Cheers,
Simon
mmarco
May 2, 2012, 10:41:43 AM5/2/12
to sage-devel
About doctsting, when we talk about 100% coverage, do we mean really
100% of the functions?
I mean: i am writing classes for braid groups and finitely presented
groups. This implies writing the corresponding procedures for _mul_
_pow_ _inverse_ _call_ and so on. This procedures are vastly tested in
all the other doctests of the class. There are also some auxiliary
functions which are writen only to be used by other functions (which
are apropiatelly tested). Do i have to write documentation strings,
with the corresponding doctest for all these functions? Even for
trivial ones such as _repr_ or _latex_?
100% of the functions?
I mean: i am writing classes for braid groups and finitely presented
groups. This implies writing the corresponding procedures for _mul_
_pow_ _inverse_ _call_ and so on. This procedures are vastly tested in
all the other doctests of the class. There are also some auxiliary
functions which are writen only to be used by other functions (which
are apropiatelly tested). Do i have to write documentation strings,
with the corresponding doctest for all these functions? Even for
trivial ones such as _repr_ or _latex_?
Simon King
May 2, 2012, 11:27:58 AM5/2/12
to sage-devel
Hi Marco,
On 2 Mai, 16:41, mmarco <mma...@unizar.es> wrote:
> Do i have to write documentation strings,
> with the corresponding doctest for all these functions? Even for
> trivial ones such as _repr_ or _latex_?
Yes. In these cases, the method definition plus doc string would
likely look like
def _repr_(self):
"""
TESTS::
sage: B = <some command to create a braid>
sage: B # indirect doctest
<the representation of B>
"""
Note the "indirect doctest", which is important, since otherwise the
coverage script will complain about the fact that the name of the to-
be-tested method ("_repr_") does not occur in the test.
Cheers,
Simon
On 2 Mai, 16:41, mmarco <mma...@unizar.es> wrote:
> Do i have to write documentation strings,
> with the corresponding doctest for all these functions? Even for
> trivial ones such as _repr_ or _latex_?
likely look like
def _repr_(self):
"""
TESTS::
sage: B = <some command to create a braid>
sage: B # indirect doctest
<the representation of B>
"""
Note the "indirect doctest", which is important, since otherwise the
coverage script will complain about the fact that the name of the to-
be-tested method ("_repr_") does not occur in the test.
Cheers,
Simon
kcrisman
May 2, 2012, 1:00:21 PM5/2/12
to sage-devel
can really be as trivial as that - just repeat some other doctest
where it is used.
Michael Orlitzky
May 2, 2012, 5:24:54 PM5/2/12
to sage-...@googlegroups.com
On 05/02/2012 01:00 PM, kcrisman wrote:
>> """
>>
>> Note the "indirect doctest", which is important, since otherwise the
>> coverage script will complain about the fact that the name of the to-
>> be-tested method ("_repr_") does not occur in the test.
>>
>
> Yup, and same for things like _pow_ as you say. On the plus side, it
> can really be as trivial as that - just repeat some other doctest
> where it is used.
>
You can usually call the underscore methods directly. An obvious test
>> """
>>
>> Note the "indirect doctest", which is important, since otherwise the
>> coverage script will complain about the fact that the name of the to-
>> be-tested method ("_repr_") does not occur in the test.
>>
>
> Yup, and same for things like _pow_ as you say. On the plus side, it
> can really be as trivial as that - just repeat some other doctest
> where it is used.
>
for them is that they actually get used:
sage: import sys
sage: a = ZZ(randint(-sys.maxint, sys.maxint))
sage: b = ZZ(randint(-sys.maxint, sys.maxint))
sage: a.__mul__(b) == a*b
True
Simon King
May 2, 2012, 5:38:42 PM5/2/12
to sage-...@googlegroups.com
Hi Michael,
On 2012-05-02, Michael Orlitzky <mic...@orlitzky.com> wrote:
> You can usually call the underscore methods directly.
I thought (i.e.: I am sure that I was repeatedly told) that calling
magical methods in a doctest is strongly discouraged. The reason is
that one purpose of doctests is educational. As a reviewer, I would
not hesitate to ask for changing such an overly explicit test, when
the doctest fails to show how the method is really supposed to be
used. Of course, `_repr_` and friends is not supposed to be called
directly - hence the indirect test.
Cheers,
Simon
On 2012-05-02, Michael Orlitzky <mic...@orlitzky.com> wrote:
> You can usually call the underscore methods directly.
magical methods in a doctest is strongly discouraged. The reason is
that one purpose of doctests is educational. As a reviewer, I would
not hesitate to ask for changing such an overly explicit test, when
the doctest fails to show how the method is really supposed to be
used. Of course, `_repr_` and friends is not supposed to be called
directly - hence the indirect test.
Cheers,
Simon
Michael Orlitzky
May 2, 2012, 5:49:43 PM5/2/12
to sage-...@googlegroups.com
they should be tested /somewhere/, even if not in the docstring.
William Stein
May 2, 2012, 6:55:12 PM5/2/12
to sage-...@googlegroups.com
make indirect doctests when there is no easy way to make a good direct
one.
>
> Cheers,
> Simon
>
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
John H Palmieri
May 2, 2012, 7:15:59 PM5/2/12
to sage-...@googlegroups.com
On Wednesday, May 2, 2012 2:38:42 PM UTC-7, Simon King wrote:
Hi Michael,
On 2012-05-02, Michael Orlitzky wrote:
> You can usually call the underscore methods directly.
I thought (i.e.: I am sure that I was repeatedly told) that calling
magical methods in a doctest is strongly discouraged. The reason is
that one purpose of doctests is educational. As a reviewer, I would
not hesitate to ask for changing such an overly explicit test, when
the doctest fails to show how the method is really supposed to be
used. Of course, `_repr_` and friends is not supposed to be called
directly - hence the indirect test.
I've never heard of such a rule. I notice that the Developer's Guide needs to be updated: I can't find mention of "indirect doctest" anywhere there, so there is obviously no mention of how or when it should be used. Indeed, I can't find "indirect doctest" in any documentation except for the reference manual.
Also, I believe that the Sage coverage script doesn't require an indication of '# indirect doctest' for doctests for underscore methods; it is assumed (I guess) that in those cases, you may very well doctest those indirectly. For example, the file sage/homology/delta_complex.py has a method '__cmp__' in which the doctests don't call it explicitly, there is no '# indirect doctest' anywhere, and yet the file has 100% coverage. I found several other such examples browsing through the source code and searching for "def _".
Anyway, as far as whether you should directly call the method, I think arguments can be made on both sides. An explicit call to the method is convincing, and since these methods aren't included in the reference manual (at least not right now), the educational aspect is not as great. However, for people browsing the code, using the methods (indirectly) in the intended way can be helpful. For indirect doctests, either marked explicitly or not, it's up to the author and the referee to make sure that the doctest actually calls the method being tested, obviously. Other than that, I personally think it's best not to have a policy one way or the other.
--
John
John H Palmieri
May 2, 2012, 7:17:58 PM5/2/12
to sage-...@googlegroups.com
On Wednesday, May 2, 2012 4:15:59 PM UTC-7, John H Palmieri wrote:
Also, I believe that the Sage coverage script doesn't require an indication of '# indirect doctest' for doctests for underscore methods; it is assumed (I guess) that in those cases, you may very well doctest those indirectly.
Wait, this seems to only be true for methods starting with two underscores...
--
John
Simon King
May 3, 2012, 1:06:18 AM5/3/12
to sage-...@googlegroups.com
Hi William,
On 2012-05-02, William Stein <wst...@gmail.com> wrote:
> On Wed, May 2, 2012 at 2:38 PM, Simon King <simon...@uni-jena.de> wrote:
>> I thought (i.e.: I am sure that I was repeatedly told) that calling
>> magical methods in a doctest is strongly discouraged.
>
(and I think not just once) to better provide an indirect test, like
"Only ever make a direct doctest for a magical method when there is no
easy way to make a good indirect test."
Cheers,
Simon
On 2012-05-02, William Stein <wst...@gmail.com> wrote:
> On Wed, May 2, 2012 at 2:38 PM, Simon King <simon...@uni-jena.de> wrote:
>> I thought (i.e.: I am sure that I was repeatedly told) that calling
>> magical methods in a doctest is strongly discouraged.
>
> I *always* call _repr_ directly in the doctest of repr. I only ever
> make indirect doctests when there is no easy way to make a good direct
> one.
Really? I don't recall who was referee, but I do recall that I was asked
> make indirect doctests when there is no easy way to make a good direct
> one.
(and I think not just once) to better provide an indirect test, like
"Only ever make a direct doctest for a magical method when there is no
easy way to make a good indirect test."
Cheers,
Simon
William Stein
May 3, 2012, 1:13:41 AM5/3/12
to sage-...@googlegroups.com
Really. I completely disagree with the above quote.
Robert Bradshaw
May 3, 2012, 2:24:25 AM5/3/12
to sage-...@googlegroups.com
On Sun, Apr 29, 2012 at 3:34 PM, David Kirkby <david....@onetel.net> wrote:
> On 29 April 2012 20:58, Simon King <simon...@uni-jena.de> wrote:
>
>> sage: foobar?
>> it would be doable that one reads:
>> Type: function
>> Base Class: <type 'function'>
>> String Form: <function foobar at 0x4eb9f50>
>> Namespace: Interactive
>> Loaded File: ...
>> Source File: ...
>> Definition: foobar(x, y)
>> Docstring:
>> Please contribute to Sage by writing a doctest
>> for this function!
>>
>> This function does useful stuff.
>>
>> Here, there is no regression (reading documentation is not
>> time-critical), and it is clearly doable.
>
>> What do people think? Would that kind of message encourage people to do
>> their first contribution, or would that scare people off?
>
> I think it might get people to contribute. It needs a link to a page
> describing in detail the process of writing a doctest for Sage, and
> how to get that doctest into Sage.
>
> Perhaps it woud scare a few people off, but overall I would have
> thought the result positive.
+1, I think that's a great idea.
What may also help is making it trivial to create doctests, e.g.
sage: add_doctest(some_method, "Description.")
sage: some_method(4)
2
sage: some_method(5)
5
sage: some_method(32)
5
sage: some_method(3)
7
sage: finish_doctest()
Please upload /tmp/doctest-some_method.patch to trac.
- Robert
> On 29 April 2012 20:58, Simon King <simon...@uni-jena.de> wrote:
>
>> sage: foobar?
>> it would be doable that one reads:
>> Type: function
>> Base Class: <type 'function'>
>> String Form: <function foobar at 0x4eb9f50>
>> Namespace: Interactive
>> Loaded File: ...
>> Source File: ...
>> Definition: foobar(x, y)
>> Docstring:
>> Please contribute to Sage by writing a doctest
>> for this function!
>>
>> This function does useful stuff.
>>
>> Here, there is no regression (reading documentation is not
>> time-critical), and it is clearly doable.
>
>> What do people think? Would that kind of message encourage people to do
>> their first contribution, or would that scare people off?
>
> I think it might get people to contribute. It needs a link to a page
> describing in detail the process of writing a doctest for Sage, and
> how to get that doctest into Sage.
>
> Perhaps it woud scare a few people off, but overall I would have
> thought the result positive.
+1, I think that's a great idea.
What may also help is making it trivial to create doctests, e.g.
sage: add_doctest(some_method, "Description.")
sage: some_method(4)
2
sage: some_method(5)
5
sage: some_method(32)
5
sage: some_method(3)
7
sage: finish_doctest()
Please upload /tmp/doctest-some_method.patch to trac.
- Robert
Robert Bradshaw
May 3, 2012, 2:30:07 AM5/3/12
to sage-...@googlegroups.com
at least a new variation.
Simon King
May 3, 2012, 2:45:06 AM5/3/12
to sage-...@googlegroups.com
Hi Robert,
On 2012-05-03, Robert Bradshaw <robe...@math.washington.edu> wrote:
>>> What do people think? Would that kind of message encourage people to do
>>> their first contribution, or would that scare people off?
>>
>> I think it might get people to contribute. It needs a link to a page
>> describing in detail the process of writing a doctest for Sage, and
>> how to get that doctest into Sage.
>>
>> Perhaps it woud scare a few people off, but overall I would have
>> thought the result positive.
>
> +1, I think that's a great idea.
OK. I think I can do the part for the command line version of Sage, but
I am afraid someone else would need to do the notebook part: I hardly
have experience with using the notebook, even less with developing the
notebook, and AFAIK the new notebook is developed not on the sage trac
but on github, which I know virtually nothing about.
One question: Shall the "encouraging message" appear in the reference
manual as well, or only when the user interactively reads the documentation
of an object?
I think suggestions should go to the ticket (#12891).
> What may also help is making it trivial to create doctests, e.g.
>
> sage: add_doctest(some_method, "Description.")
> sage: some_method(4)
> 2
> sage: some_method(5)
> 5
> sage: some_method(32)
> 5
> sage: some_method(3)
> 7
> sage: finish_doctest()
> Please upload /tmp/doctest-some_method.patch to trac.
I like that.
Best regards,
Simon
On 2012-05-03, Robert Bradshaw <robe...@math.washington.edu> wrote:
>>> What do people think? Would that kind of message encourage people to do
>>> their first contribution, or would that scare people off?
>>
>> I think it might get people to contribute. It needs a link to a page
>> describing in detail the process of writing a doctest for Sage, and
>> how to get that doctest into Sage.
>>
>> Perhaps it woud scare a few people off, but overall I would have
>> thought the result positive.
>
> +1, I think that's a great idea.
I am afraid someone else would need to do the notebook part: I hardly
have experience with using the notebook, even less with developing the
notebook, and AFAIK the new notebook is developed not on the sage trac
but on github, which I know virtually nothing about.
One question: Shall the "encouraging message" appear in the reference
manual as well, or only when the user interactively reads the documentation
of an object?
I think suggestions should go to the ticket (#12891).
> What may also help is making it trivial to create doctests, e.g.
>
> sage: add_doctest(some_method, "Description.")
> sage: some_method(4)
> 2
> sage: some_method(5)
> 5
> sage: some_method(32)
> 5
> sage: some_method(3)
> 7
> sage: finish_doctest()
> Please upload /tmp/doctest-some_method.patch to trac.
Best regards,
Simon
Robert Bradshaw
May 3, 2012, 2:46:58 AM5/3/12
to sage-...@googlegroups.com
On Sun, Apr 29, 2012 at 7:35 AM, David Kirkby <david....@onetel.net> wrote:
> The fact there are functions in Sage untested is worrying, and I think
> its fair to say most people would like to see 100% doctest coverage,
> but IMHO, we need a plan for this to happen, and the plan inforced,
> otherwise this will not happen soon.
To take things down a completely different path, I might argue that
part of the problem is that there is diminishing return to doctests as
we approach the 100% mark. In particular, I think it would be worth
starting to look at other forms of coverage, e.g. line (or even
branch) coverage--I feel much more confident in a method that has high
branch coverage and no doctest than the other way around (and a lot of
the painful-to-doctest methods are the internal, underscore methods
that are tested indirectly elsewhere). I would also like to see more
random fuzz testing (like you've done to see if you can crash things
with random input) and correctness testing (e.g. do the same
computation (or verify deep theorems and conjectures) with random
inputs using different algorithms (this could perhaps even be
semi-automated for any method that has an algorithm parameter).
Thoughts?
> So I'm suggesting we collect ideas for a plan.
>
> Two things that do NOT appear to work, at least in isolation are:
>
> A) Saying coverage should be x % in release Y, then hoping it is.
> B) Saying coverage should be x % by the date DD/MM/YY, then hoping it is.
> suggesting they are all very good (in particular 1 and 4 are a bit
> excessive), but I mention them anyway.
>
> 1) Having a release, where the only aim is to increase doctest
> coverage. No patches are accepted, unless they increase coverage.
> 2) Finding sections of code which are not tested, and depreciating
> them, unless someone writes the doctests, in which case the
> depreciation can be removed.
> 3) Having a release, where anyone can submit a patch which gets
> reviewed, but the release manager does not merge it unless the author
> of the patch can provide another patch which increase doctest
> coverage.
> 4) Finding individuals that have written code that is not tested, and
> not merging any more patches from them unless they first add tests to
> the ALL code they have already written.
> 5) Finding individuals that have written code that is not tested, and
> not merging any more patches from them unless they first add tests to
> a number (say 5) tests for all the code they have written which is
> untested.
> 6) Not accepting any patches, unless a equal number of lines of code
> are submitted which add doctests.
> 7) Paying individuals to just write tests.
>
> I believe if we want to achieve 100% coverage, we need to do more than
> A and B above, as they are proved not to work.
>
> Can anyone suggest anything else, or have any comments.
7') Some kind of gamification. This could be as simple as a monthly
leader-board for whoever contributed the most doctests, the patchbot
giving preference to patches increasing coverage, or whatever. Carrots
are more enjoyable than sticks.
7'') Personal bounties of some kind. E.g. I will referee any ticket of
your choosing if you contribute 100 new, unrelated doctests.
8) Make it easier (and unobtrusive) to contribute doctests.
- Robert
> The fact there are functions in Sage untested is worrying, and I think
> its fair to say most people would like to see 100% doctest coverage,
> but IMHO, we need a plan for this to happen, and the plan inforced,
> otherwise this will not happen soon.
To take things down a completely different path, I might argue that
part of the problem is that there is diminishing return to doctests as
we approach the 100% mark. In particular, I think it would be worth
starting to look at other forms of coverage, e.g. line (or even
branch) coverage--I feel much more confident in a method that has high
branch coverage and no doctest than the other way around (and a lot of
the painful-to-doctest methods are the internal, underscore methods
that are tested indirectly elsewhere). I would also like to see more
random fuzz testing (like you've done to see if you can crash things
with random input) and correctness testing (e.g. do the same
computation (or verify deep theorems and conjectures) with random
inputs using different algorithms (this could perhaps even be
semi-automated for any method that has an algorithm parameter).
Thoughts?
> So I'm suggesting we collect ideas for a plan.
>
> Two things that do NOT appear to work, at least in isolation are:
>
> A) Saying coverage should be x % in release Y, then hoping it is.
> B) Saying coverage should be x % by the date DD/MM/YY, then hoping it is.
>
> Both of these have been tried, and niether are working. The graph here
>
> http://thales.math.uqam.ca/~labbes/blogue/2011/03/evolution-of-the-overall-doctest-coverage-of-sage/
> Both of these have been tried, and niether are working. The graph here
>
>
> seems to indicate that progress on this has slowed over the last year
> or so, where progress was about 4.7% per year. (I'm not actually sure
> I agree with his line of best fit, which would suggest to me the
> situation has slowed even more.)
>
>
> A few ideas that might work, are below. I would add, I'm not
> seems to indicate that progress on this has slowed over the last year
> or so, where progress was about 4.7% per year. (I'm not actually sure
> I agree with his line of best fit, which would suggest to me the
> situation has slowed even more.)
>
>
> suggesting they are all very good (in particular 1 and 4 are a bit
> excessive), but I mention them anyway.
>
> 1) Having a release, where the only aim is to increase doctest
> coverage. No patches are accepted, unless they increase coverage.
> 2) Finding sections of code which are not tested, and depreciating
> them, unless someone writes the doctests, in which case the
> depreciation can be removed.
> 3) Having a release, where anyone can submit a patch which gets
> reviewed, but the release manager does not merge it unless the author
> of the patch can provide another patch which increase doctest
> coverage.
> 4) Finding individuals that have written code that is not tested, and
> not merging any more patches from them unless they first add tests to
> the ALL code they have already written.
> 5) Finding individuals that have written code that is not tested, and
> not merging any more patches from them unless they first add tests to
> a number (say 5) tests for all the code they have written which is
> untested.
> 6) Not accepting any patches, unless a equal number of lines of code
> are submitted which add doctests.
> 7) Paying individuals to just write tests.
>
> I believe if we want to achieve 100% coverage, we need to do more than
> A and B above, as they are proved not to work.
>
> Can anyone suggest anything else, or have any comments.
7') Some kind of gamification. This could be as simple as a monthly
leader-board for whoever contributed the most doctests, the patchbot
giving preference to patches increasing coverage, or whatever. Carrots
are more enjoyable than sticks.
7'') Personal bounties of some kind. E.g. I will referee any ticket of
your choosing if you contribute 100 new, unrelated doctests.
8) Make it easier (and unobtrusive) to contribute doctests.
- Robert
kcrisman
May 3, 2012, 9:34:34 AM5/3/12
to sage-devel
On May 3, 2:24 am, Robert Bradshaw <rober...@math.washington.edu>
wrote:
> On Sun, Apr 29, 2012 at 3:34 PM, David Kirkby <david.kir...@onetel.net> wrote:
Though I suspect you could only easily do the above for things with no
docstring...
Simon King
May 3, 2012, 9:47:18 AM5/3/12
to sage-...@googlegroups.com
Hi!
On 2012-05-03, kcrisman <kcri...@gmail.com> wrote:
>> sage: finish_doctest()
>> Please upload /tmp/doctest-some_method.patch to trac.
>
> Zap! Yes! Just like the fabled "making a patch on the web" idea.
> Though I suspect you could only easily do the above for things with no
> docstring...
It could also append a new "TESTS::" or "EXAMPLES::" section to an
existing doc string.
On 2012-05-03, kcrisman <kcri...@gmail.com> wrote:
>> sage: finish_doctest()
>> Please upload /tmp/doctest-some_method.patch to trac.
>
> Zap! Yes! Just like the fabled "making a patch on the web" idea.
> Though I suspect you could only easily do the above for things with no
> docstring...
existing doc string.
kcrisman
May 3, 2012, 9:54:12 AM5/3/12
to sage-devel
On May 3, 9:47 am, Simon King <simon.k...@uni-jena.de> wrote:
> Hi!
to be reviewed, so this is not a big deal. Maybe Robert will write
and advertise this new function :)
rjf
May 3, 2012, 11:50:26 AM5/3/12
to sage-...@googlegroups.com
It seems to me that "doctests" are supposed to serve two purposes.
Since I have not looked at them (well, maybe one or two over the years?),
my comments may be naive or irrelevant, but here goes.
documentation has to be written by the programmer who writes the code,
preferably before the code is written, but certainly before the code is
inserted into a system. It should include examples of how the code is
thought to be useful, written as complete examples if possible. It should
include, in a separate section, the "edge" cases that are either the most
extreme but "still working" examples, and maybe the just one-more than
that which fail to work, but which an unwary user might think might work.
then there are the tests which are almost inevitably inadequate. Many
programs (certainly most in Maxima) do not exhaustively test their inputs
for nonsense (e.g. non-terminating computations, ill-specified tasks)
because these conditions cannot be detected 100%, and sometimes it
is better to just let stuff go through and hope that some program further down
the line can make sense of it. For example, one could check for strict
adherence to expectations, like "argument must be a polynomial in x" but
would you notice that acos(n*cos(x)) is a polynomial?
But some checking is probably worthwhile, like checking for the expected
number of arguments. Though even there, someone might have ANOTHER
function in mind...
Anyway, a list of things that the programmer has thought of that will not
work and might be useful for someone to fix up later could be helpful.
A list of things that you might think the program will do but it won't
(features?) e.g. "This program does not work for complex numbers".
Of course in some cases writing the framework to do checking of all kinds of buggy input
can be daunting, and may even be expensive at runtime.
In the case of Sage, where important functionality is simply imported from
external packages after the fact, and the programmer may be long gone,
your options are limited.
Having a naive person come by after the fact and write tests can even be
counterproductive --- if it gives you a false sense of confidence that
something works when the tests are quite meaningless. As an example,
I recall someone testing/comparing Macsyma's polynomial factoring program
against some other program, and drawing conclusions about the speed of
the Berlekamp algorithm.
In fact most of the tests were composed of "well-known" factoring special cases,
most of which were specifically tested for in Macsyma, and for which the
algorithm was never invoked.
Also, if you insist on writing doctests for maxima functionality accessed through
Sage, you should probably document what maxima does. Then you can
separately document what Sage does to prepare input and process output
from maxima.
Robert Bradshaw
May 4, 2012, 5:53:17 PM5/4/12
to sage-...@googlegroups.com
On Thu, May 3, 2012 at 8:50 AM, rjf <fat...@gmail.com> wrote:
>
>
> It seems to me that "doctests" are supposed to serve two purposes.
> Since I have not looked at them (well, maybe one or two over the years?),
> my comments may be naive or irrelevant, but here goes.
>
> documentation has to be written by the programmer who writes the code,
> preferably before the code is written, but certainly before the code is
> inserted into a system. It should include examples of how the code is
> thought to be useful, written as complete examples if possible. It should
> include, in a separate section, the "edge" cases that are either the most
> extreme but "still working" examples, and maybe the just one-more than
> that which fail to work, but which an unwary user might think might work.
Yes, exactly.
>
>
> It seems to me that "doctests" are supposed to serve two purposes.
> Since I have not looked at them (well, maybe one or two over the years?),
> my comments may be naive or irrelevant, but here goes.
>
> documentation has to be written by the programmer who writes the code,
> preferably before the code is written, but certainly before the code is
> inserted into a system. It should include examples of how the code is
> thought to be useful, written as complete examples if possible. It should
> include, in a separate section, the "edge" cases that are either the most
> extreme but "still working" examples, and maybe the just one-more than
> that which fail to work, but which an unwary user might think might work.
not be completely naive, at least often enough that this is not a big
issue. Author-written tests are valuable (and have been a requirement
for most of Sage's history) but tests written by a non-author are
quite valuable as well (verifying that the function conforms to its
specifications as provided by its definition/documentation, and
hitting corner cases the original author may not have thought of.)
> As an example,
> I recall someone testing/comparing Macsyma's polynomial factoring program
> against some other program, and drawing conclusions about the speed of
> the Berlekamp algorithm.
>
> In fact most of the tests were composed of "well-known" factoring special
> cases,
> most of which were specifically tested for in Macsyma, and for which the
> algorithm was never invoked.
>
> Also, if you insist on writing doctests for maxima functionality accessed
> through
> Sage, you should probably document what maxima does. Then you can
> separately document what Sage does to prepare input and process output
> from maxima.
"just trust" it. Our tests will then be that we are using maxima
correctly, and that it and our interface to it, behave as expected. Of
course due to limited resources we are in a less than ideal case on
both sides of the fence (and some packages have little to no tests, in
which case Sage's tests themselves have proved a valuable regression
suite).
I do feel though at this point simply writing more doctests has less
value than investigating other forms of coverage and testing, but it's
still a worthwhile task (as is adding better/more doctests to those
functions that already have time).
- Robert
Reply all
Reply to author
Forward
0 new messages