Randomized testing gives many errors

[Jori Mäntysalo]

One example only:

./sage -t --randorder 0 src/sage/modular/modform_hecketriangle/hecke_triangle_group_element.py

is OK, but same with --randorder 1 gives one error, and with --randorder
21739 gives three errors. This not at all the only example.

I think that patchbot maintainers should use different randorder
parameters. But of course first we should try to correct those codes
giving wrong output and/or doctests expecting some list to always be in
some specific order etc.

(I ran tests on Linux e5410 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13
00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux, but I suppose that every
64-bit Linux will give same errors.)

--
Jori Mäntysalo

[Volker Braun]

At the same time there is some value to testing seed-dependent ordering; Sure, it is an implementation detail, but it does help catch errors if you change something in the code. And sometimes you must make a choice, e.g. when iterating over an infinite set. So before you go around and delete all doctests that depend on the seed, how about a "# seed0" doctest marker that disables tests when an alternative seed is given?

[Jori Mäntysalo]

On Sun, 24 Jul 2016, Volker Braun wrote:

> At the same time there is some value to testing seed-dependent ordering;
> Sure, it is an implementation detail, but it does help catch errors if you
> change something in the code. And sometimes you must make a choice, e.g.
> when iterating over an infinite set. So before you go around and delete all
> doctests that depend on the seed, how about a "# seed0" doctest marker that
> disables tests when an alternative seed is given?

Might be a good idea. But I suggest that first we collect some view of the
failures. I opened #21080 with keyword "random_test_failure" as an example
and I hope that others will follow that for some time.

At least on #21054 there was (or "is", not reviewed yet) a real error.

--
Jori Mäntysalo