Doc testing regression

[Simon King]

Dear all,

at http://groups.google.com/group/sage-support/browse_thread/thread/534306c87e7fac7a
I was asking about the apparently changed behaviour of "sage -t".
Georg suggested to move the discussion to sage-devel, so, here it
is...

On 20 Jun., 22:10, gsw <GeorgSWe...@googlemail.com> wrote:
> On 20 Jun., 08:15, Simon King <simon.k...@uni-jena.de> wrote:
> > Apparently the difference lies in Sage and not in my tests. I just
> > tried again the exact setting in which testing my extension modules
> > used to work -- now it fails, since "sage -t mtx.pyx" tries to compile
> > mtx.pyx for whatever reason.
> ...
> If I understand it correctly, the doctesting mechanism was changed, to
> be part of (or to make use of) the "load/attach" mechanism of Sage ---
> so if you can successfully "load" foobar.pyx into Sage, you also
> should be able to doctest it, with current Sage versions. But for most
> stuff which is greater than one file, the current behaviour is worse
> than it was before and is a regression IMHO.

I agree. Source code is one thing, an importable module is a
completely different thing.

Moreover: *Why* would one like to use load/attach for doc testing? I
mean, imaging a big chunk of code (in my case some 10,000 lines of
code); why should one compile it just for doing doc tests? In
particular, why *re-*compile it if the compilation of the code has
been done before?

> I could make doctesting my code (extension modules) work again in two
> hours or so mainly by inserting some of these infamous "#clib ..."
> "#cinclude ..." poor-man's pragmas I heartily dislike.

This does not look like a convenient solution.

> > Couldn't "sage -t" just take any text file, search for "sage:" prompts
> > etc, and verify the output?

... as it used to do, IIRC !

> I didn't check it, but if you move all your doctests out of a *.pyx
> file into another file --- say a *.py file with the necessary
> "import"s --- then doctesting this new file should work fine.

Sure, but it's clumsy.

> > Another idea.
>
> > Let "knight" be a (python) package or module. Is there a function
> > (say, recursive_doc_test) in Sage that does the doc tests for "knight"
> > and, recursively, for its contents (functions; classes; methods of
> > these classes; other modules, if knight is a package; ...) and returns
> > the results of the test as a string? I mean
> > sage: import knight
> > sage: recursive_doc_test(knight)
> > 'The following items had faiilures:
> > In knight.Ni.Shrubbery, l. 12:
> > expected:
> > "herring"
> > got:
> > nothing
> > ...'
>
> > The line number would refer to the 12th line in the doc string of the
> > class knight.Ni.Shrubbery, say.

I did something like that in sort of a quick hack. Also rather
clumsy:
- Recursively determine everything that is in some package/module/
class/type and retrieve the respective doc strings, so that in the end
you have a dictionary of doc strings indexed by the fully qualified
names of the things (classes, instances, functions, methods,...) they
belong to.
- For each single doc string, create a temporary file F.py. This file
contains the doc string and no code, so that it can certainly be
attached/loaded, and thus doc testing works for F.py.
- Get the results of the doc test via os.popen('sage -t F.py').read()

Advantage: I get the results of the doc tests sorted by the name of
the method/function/..., hence, if something goes wrong I can easily
locate the error.

Disadvantage: It takes a long time -- much longer than testing a
single file, since "sage -t" is called over and over again, and:

> > The disadvantage would be that "recursive_doc_test" probably couldn't
> > test cdef-methods.
>
> If inside Sage something "polished" existed as you describe it, most
> probably one *could* doctest cdef-methods with it. At least it should
> be written this way IMHO.

How could it be?

Cheers,
Simon

[gsw]

On 21 Jun., 08:28, Simon King <simon.k...@uni-jena.de> wrote:
> Dear all,
>
> athttp://groups.google.com/group/sage-support/browse_thread/thread/5343...

The "origonal" doc test module Sage builds upon already does exactly
that, IIRC. Remember, if you get doctest failures "...in
example_blah..." then this means that the doctest framework had had
built and tested one "example" file for each doc string. Also note
that the sequence this is done is *not* the sequence the docstrings
occur in the source file, but rather the examples are gone through _1,
_10, _11, .... _19, _2, _20, _21, _3, _4, _5, .. (if there are less
than 30) which already caused funny-to-debug failures. William has
introduced an option IIRC with which you can make this sequence random
(pseudo-random with a seed, so one might reproduce these sequences),
but this is not the main point here.

The main point is that the "low-level" doc test framework already does
exactly what you did with your "sort of quick hack approach", so one
"only" needs to re-enable this behaviour in such a way that it works
again for 10000-lines external modules as before, but also still works
for these new examples which have caused the changes.

In a ten-minute search I only found the following thread :
http://groups.google.com/group/sage-devel/browse_thread/thread/8da41d49ffe6c6d6#

where e.g. Jason Grout posted: "+1 on making sage -t my_file.py work
with minimal hassle. " but I didn't get closer to who actually changed
the doctesting code and when.
I haven't done a search on trac yet.

>  - Get the results of the doc test via os.popen('sage -t F.py').read()
>
> Advantage: I get the results of the doc  tests sorted by the name of
> the method/function/..., hence, if something goes wrong I can easily
> locate the error.
>
> Disadvantage: It takes a long time -- much longer than testing a
> single file, since "sage -t" is called over and over again, and:
>
> > > The disadvantage would be that "recursive_doc_test" probably couldn't
> > > test cdef-methods.
>
> > If inside Sage something "polished" existed as you describe it, most
> > probably one *could* doctest cdef-methods with it. At least it should
> > be written this way IMHO.
>
> How could it be?
>

Maybe I am a bit overconfident here, but what you did with a bunch of
external "F.py" I suppose one could do with a buch of external
"F.pyx", too.

Cheers,
gsw

> Cheers,
>    Simon

[William Stein]

2009/6/21 gsw <Georg...@googlemail.com>:

I did at the Sage Days 12 -- bug fix. I did this to fix several
reported bugs, which making that change did. I guess one person's bug
is another person's feature.

Anyway, you guys (or whoever) should just make trac tickets and fix stuff.

William

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[gsw]

On 21 Jun., 15:37, William Stein <wst...@gmail.com> wrote:
> 2009/6/21 gsw <GeorgSWe...@googlemail.com>:

> >http://groups.google.com/group/sage-devel/browse_thread/thread/8da41d...

>
> > where e.g. Jason Grout posted: "+1 on making sage -t my_file.py work
> > with minimal hassle. " but I didn't get closer to who actually changed
> > the doctesting code and when.
> > I haven't done a search on trac yet.
>
> I did at the Sage Days 12  -- bug fix.   I did this to fix several
> reported bugs, which making that change did.  I guess one person's bug
> is another person's feature.
>
> Anyway, you guys (or whoever) should just make trac tickets and fix stuff.
>

You're right of course. I just currently don't know which trac ticket
to open first ...

Cheers,
gsw