Proposal: Sort ExUnit output by line number and file path

[Filip Haglund]

I'd like ExUnit to sort its output by source code line number and path. This simplifies fixing multiple test cases in a row, when that's what you're doing, and it shouldn't slow down other development. If it would noticeably slow down parallell testing, only sorting output when `--max-cases 1` or when some other flag is set would also be an option. 

Right now, I ran `mix test` and got failing test cases on lines `99, 118, 59, 133` in the same file. The order is not even deterministic right now, as the next run gave `59, 118, 133, 99` with no code changes.

[José Valim]

The tests run in random order to help you catch bugs as tests should not be order dependent. You can set the seed to 0 if you want to keep the original order (at the cost of not having the random order). Other than that, I wouldn't like to sort output because sorting means we need to collect the ouput before printing and this means we would no longer be able to print failures as they come.

TL;DR - we choose to have random tests and immediate failure reports, hence we can't sort the failure output.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Wed, Jul 6, 2016 at 7:07 PM, Filip Haglund <fille....@gmail.com> wrote:

I'd like ExUnit to sort its output by source code line number and path. This simplifies fixing multiple test cases in a row, when that's what you're doing, and it shouldn't slow down other development. If it would noticeably slow down parallell testing, only sorting output when `--max-cases 1` or when some other flag is set would also be an option. 

Right now, I ran `mix test` and got failing test cases on lines `99, 118, 59, 133` in the same file. The order is not even deterministic right now, as the next run gave `59, 118, 133, 99` with no code changes.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1ac8f79e-f587-4cdb-8f8b-6c0c3c109877%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Filip Haglund]

That's a good default, but what about a flag then? Guessing a seed that makes all output sorted is neither feasible nor desirable, and if the tests finish within a few seconds, waiting for the result isn't too bad. The execution order can still be random and concurrent.

[José Valim]

> Guessing a seed that makes all output sorted is neither feasible nor desirable

--seed 0 does not sort the tests, which means failures should come in the same order as the one defined in your test module. Why is that not enough?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Filip Haglund]

Because when there are errors in different files, you're very likely to have to switch back and forth between files. Also, test cases close to each other are more likely to be related to the same set of files, reducing the number of files opened/closed and the scrolling around within a file. 

Sorry, I did not test `--seed 0` until now. I assumed it to still be random, but in a deterministic ordering due to using the same seed each time. It seems to actually be sorted by (path, line), as I wanted, when using `--seed 0`, with 39 failing test cases in multiple times.

This will definitely work for me. Thanks for pointing this out!