Testing corner cases

[Jori Mäntysalo]

I just play a little with Python. I think others might also benefit on
this when testing corner cases like 0x0 or 1x1 matrices, 1-element groups
and so on.

G = Graph()
known_kaboom = ['is_prime', 'layout_graphviz']
errors = []
for f in [y for y in dir(G) if y[0] != '_']:
if f in known_kaboom:
continue
try:
attrcall(f)(G)
except TypeError as e:
if "takes exactly" in e.message:
continue
except Exception as e:
print(f, e)
print
errors.append(f)
continue
errors

(This found that for example is_triangle_free() and is_cayley() fails.
known_kaboom is needed as for example is_prime() don't just throw an
exception, it stops the whole workseet.)

--
Jori Mäntysalo

[Jakob Kroeker]

@Jori Mäntysalo

So you have a mind of a tester? That's good to  know. 
In general I think it would be good to have a random testing framework  together with a testbot in sage.


I'm working on that in Singular and Macaulay2  and already catched more than 100 bugs
 Most of them were fixed (some by me). Unfortunately I make only slow progress, because I moved to industry a year ago.

Maybe you are also interested in the slides from my presentation about bug hunting and bug preventing on
https://wiki.sagemath.org/days66

Jakob

[Jori Mäntysalo]

On Mon, 31 Oct 2016, Jakob Kroeker wrote:

> So you have a mind of a tester? That's good to  know. 

Well, my "trivialcase-tester" is just those few lines of code. But as you
can see from https://trac.sagemath.org/ticket/21741 , it found quite many
error from graphs. In matrices I did not found that many errors, but I
only tested integer matrices.

Should have a list of corner cases? I don't know if that would be easy to
do or not, something like modulename._cornercases() or so.

> In general I think it would be good to have a random testing framework 
> together with a testbot in sage.

Yes, could be useful.

> Maybe you are also interested in the slides from my presentation about bug hunting and bug preventing
> on
> https://wiki.sagemath.org/days66

Good slides.

I would also add a crosstesting several functions with known identity. For
example

len(list(Posets(4))) == len([g for g in digraphs(4) if g.is_directed_acyclic() and g.is_transitively_reduced()])

(I.e. a poset can be seen as just a special case for directed graph.)

or

Posets.RandomPoset(10, 0.1).order_ideals_lattice().is_modular()

(Every order ideal lattice is a distributive lattice and every
distributive lattice is modular.)

etc.

--
Jori Mäntysalo

[Jakob Kroeker]

> I would also add a crosstesting several functions with known identity.

Good idea.

>Well, my "trivialcase-tester" is just those few lines of code. But as you
> can see from https://trac.sagemath.org/ticket/21741 , it found quite many
> error from graphs.

That is not surprising for me. Maybe it is surprising for non-testers ;-)

My code to catch bugs in Singular for groebner basis in Polynomial rings over integers was something like:

randomIdeal = generateRandomIdeal();
groebnerBase = groebner(randomIdeal );
groebnerBaseOnceAgain = groebner(groebnerBase);
checkResultEquivalence (groebnerBase, groebnerBaseOnceAgain);

But, If you catched issues for non-corner cases, the challenge is often to find a minimal/simple failing example,
which is often necessary for successful debugging.

>Should have a list of corner cases? I don't know if that would be easy to
>do or not, something like modulename._cornercases() or so.

I would expect that you will need corner cases per function and per data type (graph, Ideal, prime number...)

>In matrices I did not found that many errors, but I
>only tested integer matrices.

I recommend that you continue test sage library functions which you use in your own research projects,
or functions where you already found bugs or expect that they are improperly coded.

But probably you should not do that, you may get unpleasantly surprised...

[Jori Mäntysalo]

On Sun, 6 Nov 2016, Jakob Kroeker wrote:

> But, If you catched issues for non-corner cases, the challenge is often
> to find a minimal/simple failing example, which is often necessary for
> successful debugging.

True, but I don't see any easy solution to this. As an example, about two
years ago I found bugs in Singular 3, but no failing polynomial of less
than four variables and the polynomials where at least few lines long when
written out.

>> Should have a list of corner cases? I don't know if that would be easy
>> to do or not, something like modulename._cornercases() or so.
>
> I would expect that you will need corner cases per function and per data
> type (graph, Ideal, prime number...)

Yes, that would be good. But for example Sage should never halt on simple
f(0), not for any built-in f. This could be tested automatically.

> I recommend that you continue test sage library functions which you use
> in your own research projects, or functions where you already found bugs
> or expect that they are improperly coded.

> But probably you should not do that, you may get unpleasantly surprised...

:=)

Of course this kind of testing found only bugs that are not dangerous. You
won't get a wrong result from stack trace.

--
Jori Mäntysalo