SBT Plug-In Producing Tool-Friendly Tabular Reports

[Felix Leipold]

Hi All,

As I spent a lot of time finding flaky tests and time wasters in a huge set of acceptance tests. To help me with that task I came up with a new test reporter that produces tabular result files rather than the old junit xml.

Here is an example:

2015-02-23T00:22:37 FAILURE      0.0 ExampleSpec errors_should_be_reported My error
2015-02-23T00:22:37 SUCCESS    2.001 ExampleSpec test_should_take_approximately_2_seconds
2015-02-23T00:22:37 SUCCESS    0.502 ExampleSpec test_should_take_approximately_0.5_seconds
2015-02-23T00:22:37 IGNORED      0.0 AnotherSpec this_should_be_ignored

…

Finding the top three time wasters is now a matter of:

cat target/test-results-latest.txt \
    | sort --numeric --reverse --key=3 \
    | head -n 3

It also allows to conveniently analyse across multiple runs. More information is available here:
https://github.com/programmiersportgruppe/sbt-tabular-test-reporter

To get started add this to your project/plugins.sbt:

addSbtPlugin("org.programmiersportgruppe.sbt" %% "tabulartestreporter" % "1.4.0")

I am keen on your feedback/ pull requests.

Best regards and sorry for the shameless plug,

Felix

[Fernando Racca]

That sounds useful, thanks!

Fernando

[Marco Massenzio]

Thanks for sharing.

How do you invoke it?

I've looked in the README too and it doesn't say what sbt command to use.

I tried sbt test, but it doesn't generate the file either.

Also, does it work with sub-projects?

Thanks!

[Marco Massenzio]

Never mind, found it: as I suspected, with sub-projects, it only generates the file in one of them, not the top-level.

(this is a recurring pattern, there is a long thread in the sbt-coveralls plugin github too)

This is really useful, thanks!

A couple of nits/suggestions (hey, you asked for it :):

1. instead of space-separated (which is awkward, although admittedly easier to eyeball) it would be better to have it comma-separated;
2. instead of replacing spaces (and, I'm guessing, lowercasing it?) in the message with underscores, have it double-quote-delimited;
3. and, yeah, supporting sub-projects would be awesome too

Thanks!

[Felix Leipold]

Hi Marco,

Thanks for your feedback - I did indeed ask for it and I appreciate it :).

1. instead of space-separated (which is awkward, although admittedly easier to eyeball) it would be better to have it comma-separated;

I chose the space separated format, because it is the default for the unix utilities awk, sort, uniq and join. What kind of tools are you trying to use?
I intend to make it configurable (I originally used tabs, which plays nicely with copy & paste into Excel). I created have created a ticket in case you want to comment.

1. instead of replacing spaces (and, I'm guessing, lowercasing it?) in the message with underscores, have it double-quote-delimited;

I think double quote delimiting seems also a bit awkward to me, because then double quotes need to be escaped. Perhaps I am just too lazy to think about it. The question is again, what tool do you want to use it with? I will consider this issue as part of the field separator ticket, because I agree that mangling the test names is ugly.
 

1. and, yeah, supporting sub-projects would be awesome too

I thought I'd stick with the sbt convention with separate target folders. Which is a bit annoying.
How would you like this to work?

What I currently do in my sub-project build to get all the output is this:

find . -name test-results-latest.txt | xargs cat

Best,

Felix