Randomize order of gotest tests and benchmarks

[Albert Strasheim]

Hello devs

Would you guys be receptive to a patch that adds a -randomize=false/
true flag to gotest that would randomize the order in which tests and/
or benchmarks are run?

We've found a few places where this could have shaken out a few bugs
in our code, especially when tests are doing setup/teardown.

Regards

Albert

[rh]

> Would you guys be receptive to a patch that adds a -randomize=false/
> true flag to gotest that would randomize the order in which tests and/
> or benchmarks are run?

Instead of taking a boolean, what if it took a number that seeded the
random order? This way, failures due to random test order could be
reproduced consistently.

[Albert Strasheim]

Hello

Good idea.

I think there are two separate settings here:

1. Run tests in fixed/random order.

2. Specify/don't specify random seed.

When you have a build server that builds periodically, you want random
order without a fixed seed so that you can discover problems over
time.

Once you have detected a problem on the build server, you want to be
able to get the seed from the build log so that you can immediately
reproduce the problem on your workstation by specifying the seed.

Regards

Albert

[Rob 'Commander' Pike]

This is an easy change to make but seems odd to me. If a randomized
test finds a bug, it means your tests aren't good enough. You
shouldn't celebrate randomness so much as figure out why you needed
those conditions to trigger the bug and recreate them in a
deterministic test.

I'm not thrilled about this idea, but I'm also not dead set against
it. I do worry about flagging up gotest, making it a complex mess,
because it's easier to do that than to write good tests. Gotest's
main virtue, which is a Very Big one, is its utter simplicity. It's a
testing tool; let's no turn it into a testing development environment.

-rob

[Albert Strasheim]

Hello

On Dec 8, 5:09 pm, "Rob 'Commander' Pike" <r...@google.com> wrote:
> This is an easy change to make but seems odd to me.  If a randomized
> test finds a bug, it means your tests aren't good enough.  You
> shouldn't celebrate randomness so much as figure out why you needed
> those conditions to trigger the bug and recreate them in a
> deterministic test.

I guess there's a bit of a chicken and egg situation here: you're not
going to know that you need to write a new, better, deterministic test
until your randomized test points out that you have a problem.

> I'm not thrilled about this idea, but I'm also not dead set against
> it.  I do worry about flagging up gotest, making it a complex mess,
> because it's easier to do that than to write good tests.  Gotest's
> main virtue, which is a Very Big one, is its utter simplicity. It's a
> testing tool; let's no turn it into a testing development environment.

I hear what you're saying.

The situation we're grappling with:

We're doing a bunch of library-type development. This kind of code is
amenable to unit testing using gotest and since there probably isn't
global state (except maybe some files written to disk by some tests),
you really wouldn't expect a randomized test order to break anything.

On top of these libraries we are writing a bunch of components, which
is where goroutines and channels come into play. To some extent each
component can still be unit tested, but you also want to write tests
where a small set of components interact with each other, passing
messages and whatnot.

Maybe this kind of testing is outside the purview of gotest, and we
should be looking to create the testing development environment you
talk about.

I guess one reason one might be tempted to put more stuff into gotest
is that "make test" is just such a convenient way to run tests.

Any thoughts appreciated.

Regards

Albert

[Russ Cox]

Randomizing the order in which tests run is a
very very weak amount of randomness.
If you really want multiple possible executions
out of rand, why not seed the generator yourself?

Russ

[Gustavo Niemeyer]

Hi Albert,

> I guess there's a bit of a chicken and egg situation here: you're not
> going to know that you need to write a new, better, deterministic test
> until your randomized test points out that you have a problem.

Ideally the test shouldn't have to *break* for the person writing the
test to understand the difference between a deterministic and a
non-deterministic test. Test frameworks generally try to do the
precise opposite: running tests the same way so that breakages are
more easily reproducible. An interesting case is QuickCheck for
instance, from Haskell, which inherently depends on "random" data to
function, but really uses a fixed seed.

That said, nothing against a flag, of course.

>> because it's easier to do that than to write good tests.  Gotest's
>> main virtue, which is a Very Big one, is its utter simplicity. It's a
>> testing tool; let's no turn it into a testing development environment.

I half agree. I see the motivation and benefit on your end, but I
also miss some additional features when writing libraries. That's
where gocheck[1] came from.

[1] http://labix.org/gocheck

> We're doing a bunch of library-type development. This kind of code is
> amenable to unit testing using gotest and since there probably isn't
> global state (except maybe some files written to disk by some tests),
> you really wouldn't expect a randomized test order to break anything.
>
> On top of these libraries we are writing a bunch of components, which
> is where goroutines and channels come into play. To some extent each
> component can still be unit tested, but you also want to write tests
> where a small set of components interact with each other, passing
> messages and whatnot.

It's a bit late here, so maybe it'll come up naturally in the morning,
but I don't really get the motivation for the randomized tests in the
above description.

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter

[rh]

A more general approach that doesn't complicate gotest too badly might
be to optionally consult an outside program for order of execution.

(apologies for the ugly prototype)
# somewhere after $tests gets set
PICK="${PICK:-cat}"
export PICK
tests=$(echo $tests | tr ' ' '\n' | $PICK | tr '\n' ' ' )

This would allow for random test order, without making gotest know the
details.