[proposal] ExUnit: add `--rerun-count` CLI option to run selected tests "count" times or until failure

[lee eggebroten]

Problem: 

for heisnbugs and race conditions it is useful to rerun a set of tests a number of times.

Currently you must alter the suite(s) to wrap the the test(s) in `Enum`s which is obvioiusly tedious

Naive Implementation Proposal:

Add a `--rerun-count` command line option that would default to 1 but wrap the test run with `1..<rerun-count> |> Enum( current_test_runner_parent_function )`

[shanes...@gmail.com]

I feel this is better addressed through simple shell automation, which has probably a dozen different ways to solve for this. Blessing the idea of "just try again" as an actual ExUnit feature implicitly encourages community/developer tolerance of this kind of circumstance when all practical efforts should be made to eliminate it from the code first, wherever the problem actually originates. If those efforts fail, then fall back on surrounding it with retry logic on a case-by-case basis. I would personally discourage use of such a feature if it existed.

Shane Sveller

[Austin Ziegler]

I agree that this is fairly easy to do in shell script, but that results in a clean VM every time. This is intended to find that rare case where an error only shows up with repeated executions on the VM, not as a “run this repeatedly until it works”. It’s more of a “run this until it breaks, up to N times”.

I’m not sure that this is substantially better than modifying the test suite in this case, but it’s not a bad idea.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/3581cdf4-4d23-41d1-afaa-151995e450f8n%40googlegroups.com.

--

Austin Ziegler • halos...@gmail.com • aus...@halostatue.ca
http://www.halostatue.ca/ • http://twitter.com/halostatue

[Wiebe-Marten Wijnja]

I agree with Shane.

If you want to do this in your test suite (without a clean VM), I'd recommend creating an Elixir library that abstracts the `1..rerun_count |> Enum.each(current_test_function)` functionality that you require.
I think that would be a better way to allow people that really want it to use this technique,
while not implicitly encouraging people to resort to this logic in the way it would do when it were added as a built-in ExUnit feature.

~Marten

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQvmO_sO-QuqbxAbaOZc_Y8PVx7t64PFWU40YSk4ttordg%40mail.gmail.com.

[lee eggebroten]

It's Austin's case that's I'm trying to address: 

- It's often time-intensive uncovering where a race condition is occurring.  Not to "prove" that it IS fixed 'cause that's just dumb., 

The counter cases feel very ... elitist and controlling

- "I think you might abuse the tool so I'm not going to let you have it".

- Will someone else's "misuse" damage you in some way?

-lee

[Devon Estes]

Making tools that folks can easily misuse does indeed damage the greater community, in that it can harm the perception of that tool (or In this case, language), which in turn hinders its adoption.

There are already existing examples of this sort of choice in ExUnit where restrictions are put on APIs to help guide users towards decisions that are more sustainable in the long run, like not allowing more than one level of nesting with the ‘describe/2’ macro.

Basically, if something isn’t going to help users of the tool in the long run, it shouldn’t be made easier to do. Yes, it is time intensive _now_ to find and fix things difficult situations like race conditions, but by ignoring them you’re both going to make it harder to find and fix later on, as well as decreasing the quality of your tests until they are properly fixed.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/00dff9ce-fa83-4d20-8b98-d2ef1cb66ae5n%40googlegroups.com.

--

_________________
Devon Estes
+49 176 2356 4717
www.devonestes.com

[Austin Ziegler]

I don’t really have a strong opinion on this feature (it feels like it would be easy to do on a per-suite or per-test basis by modifying the test code, if tedious), but I think that a lot of the counter-cases are built on a fundamental misunderstanding of lee’s use case.

Basically, if something isn’t going to help users of the tool in the long run, it shouldn’t be made easier to do. Yes, it is time intensive _now_ to find and fix things difficult situations like race conditions, but by ignoring them you’re both going to make it harder to find and fix later on, as well as decreasing the quality of your tests until they are properly fixed.

The use case presented here is specifically to permit the repeated execution of a test, test suite, or all suites N number of times to see if a failure due to a race condition that does not express itself in a single run (which every separate invocation of `mix test` is) can be produced by a repeated run. This could be done with a command-line parameter or some other mechanism (although other mechanisms would need to wrap either a `test` or `describe` block explicitly, and would have no way of repeating all suites in the event of some sort of race condition between multiple async tests across the suite.

That is, lee’s looking for assistance from ExUnit (or more specifically the `mix test` runner of ExUnit) to figure out what is causing a race condition that is appearing occasionally in their tests or perhaps more frequently in production.

It’s not a bad request, because the shell script repeater approach provides a clean VM every time and the in-line code repeater approach doesn’t permit cross-suite checking in the event that it really is a weird interaction issue. Repeating all tests N times in the same VM can only be done by modifying the runner (and provides the benefit of working for test-and-suite repeats when a suite file or test line are provided); repeating suite or tests can be done by wrapping the `describe` or `test` blocks in loops.

-a

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgrLRT7_njz3673jqPM7a_nLQO4UPFesJenfq3S_zL-Mw%40mail.gmail.com.

[Devon Estes]

You don’t need to modify the runner to re-run tests on the same VM. You can have a short mix task that looks something like this:

def rerun() do

  for _ <- 0..3 do

    Mix.Task.run(:test)

    Code.unrequire_files(Code.required_files())

  end

end

However, I also can’t think of any race condition that couldn’t be reliably reproduced by running the tests on a different VM instance with the same seed but would be reliably reproduced by running the tests in the same VM, and setting the seed is already available behavior.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuj8g8Eb8eReKQS9gTKa8d%2BC8HLC%2B0H9k%3DzGBJxqZDGTA%40mail.gmail.com.

[Austin Ziegler]

Modifying your code a little even makes it work with other `mix test` arguments.

def run(args) do

  for _ <- 0..3 do

    Mix.Task.run(:test, args)

    Code.unrequire_files(Code.required_files())

  end

end

That seems like the simplest solution that doesn’t require changes to the standard test runner task.

-a

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgOoYTj8Y9kBapEo-ZcxAt0w3DbbVj0ZXzCs9WPGO9x_w%40mail.gmail.com.