[Proposal] Add support for --repeat to mix test.

[Aziz Köksal]

Hi!

From the Github issue I posted yesterday:

Mix doesn't support a --repeat option, which makes repeating the previously executed tests difficult or inconvenient.

I'm missing such an option because:

• in an editor it's pretty simple to bind a shortcut to mix test --repeat which aids in TDD
• in the CLI it's practical to have mix repeat the tests if all succeeded after a mix test --failed
• likewise after a mix test --stale or any other similar command

To give various examples:

• $ mix test test/some/particular/file_test.exs:12
  $ mix test --repeat
  # Mix would repeat the test in file_test.exs that was at line 12.
  # It would be great if Mix remembers the actual name of the test and
  # doesn't blindly use the line number which may have shifted after editing
  # and cause an entirely different test to be executed.
  
• $ mix test
  # 5  tests failed.
  $ mix test --failed
  # Mix retries the 5 failing tests. 3 out of 5 tests succeed.
  $ mix test --repeat
  # Mix would simply repeat the 5 tests, no matter their status.
  
• $ mix test --seed 1234
  $ mix test --repeat
  # Mix would repeat all the tests but also remember to use the given seed.
  # If --seed is never defined then a different seed is used with each --repeat.
  
• $ mix test apps/my_sub_app/test
  $ mix test --repeat
  # Mix would repeat the tests for the given umbrella application.
  $ mix test --only external
  $ mix test --repeat
  # Mix would repeat only the tests tagged by "external".
• $ mix test --stale
  $ mix test --repeat
  # Mix would repeat all the tests if this was the first run with --stale.
• $ mix test --stale
  # Edit code ...
  $ mix test --stale
  # Mix runs only tests affected by the code changes.
  $ mix test --repeat
  # Mix would repeat only the tests that had been marked as stale.
  
• $ mix test ...
  $ mix test --repeat
  $ mix test --repeat
  # Running with --repeat multiple times would naturally repeat the same tests again.
  

The basic premise is always: tests X, Y, Z have been run, therefore X, Y, Z will be run again with mix test --repeat.That makes the most sense to me. Maybe there are corner cases, like for example, when you run mix test inside an umbrella app and then you run mix test --repeat outside in the umbrella root. It probably should repeat the tests in that initial umbrella app, but what if the other apps have no previously executed tests? Not a difficult thing to figure out but I'm sure it would happen.

Thanks in advance for your time and considering my proposal!

[Mario Olivio Flores]

I think `mix test --listen-on-stdin` might meet your use case.  From the docs:

    --listen-on-stdin - runs tests, and then listens on stdin. It will re-run tests once a newline is received. See the "File system watchers" section below

I setup an alias for this: alias mtl="PORT=4006 mix test --listen-on-stdin"  (I specify the port so I can still run individual tests in my text editor)

   > mtl test/some/particular/file_test.exs:12 --seed 1234

   > [hit enter to repeat]

I have a second alias for mtl0 to pass in a seed of zero to help catch flaky tests. 

[Aziz Köksal]

Hi! Using `--listen-on-stdin` will certainly repeat the tests affected by each code change, but a `--repeat` flag would cover a different functionality. Sometimes it's prohibitively expensive to execute all affected tests upon each code change. Business projects are usually not that well architected when it comes to this and engineers don't get time to work on efficient, clean and fast test executions. Even if the project isn't a mess it may be useful to have the `--repeat` flag in order to focus on one test only, because maybe all the others tests are testing something that takes up to a minute or even more (downloading/generating files for example).

Thanks for the suggestion but it's not entirely what I'm looking for. I've been meaning to implement this feature but whenever I have a little time I can't find my way into the Mix codebase... :shrug: