Is it possible to only show the errors when executing tests from sbt

[Niklas Lochschmidt]

Hi

So, is it possible to disable all the positive output from scalatest and just
show the errors when executing tests using sbt?

We have a multiproject-build using sbt 0.13.5 and scalatest 2.2.0 with a couple
hundred tests. When we run the tests and one of them fails we have to scroll
through the whole output to search for the failing test. What I usually end up
doing is just running testQuick to see the failing test. However we still have
to scroll through all the log output if tests fail on our integration server.

I found a post from two years ago https://groups.google.com/d/topic/scalatest-users/ESotz6OTSTQ/discussion
around the idea of the dot-reporter which is basically what I would like to
have, but I guess the idea was discarded again?

Some pointers into the right direction would be appreciated.

Cheers
Niklas

[Bill Venners]

Hi Niklas,

You can drop non-critical events from the output, and I'll explain how
below, but it sounds more like you may want to turn on "reminders".
The reminders feature will repeat failed and/or canceled tests at the
end of the "string" reports: standard out, standard error, and file
reports. Search for "reminders" on this page:

http://doc.scalatest.org/2.2.1/index.html#org.scalatest.tools.Runner$@configuringReporters

The relevant params are:

I - show reminder of failed and canceled tests without stack traces
T - show reminder of failed and canceled tests with short stack traces
G - show reminder of failed and canceled tests with full stack traces
K - exclude TestCanceled events from reminder

You can control the stack trace size of reminders independently from
that in the regular report. So in your sbt file you'd add something
like "-oGK" to get reminders with full stack traces and no reminders
of canceled tests (just failed tests), something like:

testOptions in Test += Tests.Argument("-oGK")

If you still want to turn off events, look at the beginning of the
configuring reporters section, and you'll find this list:

N - drop TestStarting events
C - drop TestSucceeded events
X - drop TestIgnored events
E - drop TestPending events
H - drop SuiteStarting events
L - drop SuiteCompleted events
O - drop InfoProvided events
P - drop ScopeOpened events
Q - drop ScopeClosed events
R - drop ScopePending events
M - drop MarkupProvided events

So if you wanted to drop TestSucceeded events, you'd say something
like "-oC" in your sbt build file.

Bill

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
> ---
> You received this message because you are subscribed to the Google Groups
> "scalatest-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalatest-use...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Bill Venners
Artima, Inc.
http://www.artima.com

[Niklas Lochschmidt]

Hi Bill

Thanks for pointing out the reminder feature, it is very useful!

Regarding the use of "-oC" to drop TestSucceeded events, it doesn't seem to be
working. I created an example project over at Github with a very simple sbt file:

scalaVersion := "2.11.2"

testOptions in Test := Seq(Tests.Argument("-oC"))

libraryDependencies := Seq(
  "org.scala-lang" % "scala-compiler" % scalaVersion.value,
  "org.scalatest" %% "scalatest" % "2.2.1")

When I let the tests run, the successul tests are still printed as you can see in the
Travis Log.
To test whether the arguments are applied, I exchanged "-oC" with "-oI" which
reprints all the failed tests at the end of the build. So in general it seems to be
working however not for dropping events.

Cheers
Niklas

[Niklas Lochschmidt]

Hi

So, it seems like I had been to quick in saying that the reminder feature is useful

in this situation. I didn't see the issue because in my simple first test. I only had 

one project and in my second test with multiple projects the tests ran so fast that

fast that both reminder outputs where printed at the very and of the test. This is 

not the case in my real project.

To demonstrate this, I let the tests of two submodules in my example project 

(https://github.com/nlochschmidt/scalatest-sbt-output) run in sequence, then the

reminder output of the first module that gets tested is printed as soon as it is 

finished. Then all the test output of the second module is printed and only the 

failed tests of this second module are printed at the very end of the log.

See https://travis-ci.org/nlochschmidt/scalatest-sbt-output/builds/32703529

Having seen that, is there really no way to disable output of the successful tests

in sbt? Otherwise would it be possible to combine the reminder outputs from 

multiple sub projects and print them at the very end of the project-root's test run?

Cheers

Niklas

-----------------------------------------------

Software Developer at vaamo.de

Am Donnerstag, 14. August 2014 17:33:49 UTC+2 schrieb Niklas Lochschmidt:

Hi Bill

Thanks for pointing out the reminder feature, it is very useful!

Regarding the use of "-oC" to drop TestSucceeded events, it doesn't seem to be
working. I created an example project over at Github with a very simple sbt file:

scalaVersion := "2.11.2"

testOptions in Test := Seq(Tests.Argument("-oC"))

libraryDependencies := Seq(
  "org.scala-lang" % "scala-compiler" % scalaVersion.value,
  "org.scalatest" %% "scalatest" % "2.2.1")

When I let the tests run, the successul tests are still printed as you can see in the
Travis Log.
To test whether the arguments are applied, I exchanged "-oC" with "-oI" which
reprints all the failed tests at the end of the build. So in general it seems to be
working however not for dropping events.

Cheers
Niklas

Am Donnerstag, 14. August 2014 16:49:05 UTC+2 schrieb Bill Venners:

Hi Niklas, 

You can drop non-critical events from the output, and I'll explain how 

below, but it sounds more like you may want to turn on "reminders". 

The reminders feature will repeat failed and/or canceled tests at the 

> Hi 

> 

> So, is it possible to disable all the positive output from scalatest and 

> just 

> show the errors when executing tests using sbt? 

> 

> We have a multiproject-build using sbt 0.13.5 and scalatest 2.2.0 with a 

> couple 

> hundred tests. When we run the tests and one of them fails we have to scroll 

> through the whole output to search for the failing test. What I usually end 

> up 

> doing is just running testQuick to see the failing test. However we still 

> have 

> to scroll through all the log output if tests fail on our integration 

> server. 

> 

> I found a post from two years ago 

> https://groups.google.com/d/topic/scalatest-users/ESotz6OTSTQ/discussion 

> around the idea of the dot-reporter which is basically what I would like to 

> have, but I guess the idea was discarded again? 

> 

> Some pointers into the right direction would be appreciated. 

> 

> Cheers 

> Niklas 

> 

> -- 

[Chua Chee Seng]

Hi, 

-oC 'should' work, but unfortunately it didn't due to a bug in the SBT integration, I submitted a pull request to fix it: 

https://github.com/scalatest/scalatest/pull/368

Sorry for the inconvenience caused, hopefully it could be included in a release soon.

Chee Seng

[Niklas Lochschmidt]

Excellent! Thank you.
I hope the fix makes it into the next release.

Kind regards
Niklas
-------------------------------------
Developer at vaamo.de

> You received this message because you are subscribed to a topic in the Google Groups "scalatest-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/scalatest-users/Ht-5Z4JXjHQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to scalatest-use...@googlegroups.com.

[Bill Venners]

Hi Niklas,

We'll make a 2.2.2 release with this in it. By the way one other idea
you might try in the meantime is to use a file reporter that excludes
the events you don't want to see. Then you can look in the file for
the summary.

Bill

On Mon, Aug 18, 2014 at 12:43 AM, Niklas Lochschmidt

[Chua Chee Seng]

Hi Niklas, 

Just to update that version 2.2.2 has been published, would you mind giving it a try?

Thanks.

Chee Seng

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
> ---
> You received this message because you are subscribed to a topic in the Google Groups "scalatest-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/scalatest-users/Ht-5Z4JXjHQ/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to scalatest-users+unsubscribe@googlegroups.com.

[Niklas Lochschmidt]

Hi Chee Seng

Thank you for getting back to me.

I changed my little example to only print failed tests using the option "-oNCXHELPOQRM" and
first of all noticed that InfoProvided events are still printed. In addition the output now
doesn't read very nicely because I get the test names for failed tests but not the scopes. If I
enable output of the scopes then of course I get all scopes not just the ones with failed tests.

Going back to Bill's recommendation about using the reminder-feature I added the 'I' to the
output configuration to enable reminders. This prints the reminders with all the relevant test
however the failed tests and info events are still printed and because it's a multi-project build
they are printed in front and behind reminders for other projects.
(see https://travis-ci.org/nlochschmidt/scalatest-sbt-output/builds/33078047)

So I think ideally I would like to really disable all output of the reporter other then the reminders.
I might just take a look into the scalatest source on the weekend to see if I can come up with a
solution.

Cheers
Niklas
-------------------------------
Developer at vaamo.de

> To post to this group, send email to scalate...@googlegroups.com

> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
> ---

> You received this message because you are subscribed to a topic in the Google Groups "scalatest-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/scalatest-users/Ht-5Z4JXjHQ/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to scalatest-use...@googlegroups.com.

[Bill Venners]

Hi Niklas,

There's also the -oU (for Unformatted or Ugly) mode, which will print
the whole test name when the test fails. You might try adding that to
get the entire nest name (and its Suite) when a test fails. I didn't
provide a way to turn off all output, because I didn't want people to
be able to accidentally hide information for failures. We'll check
into the InfoProvided events you say you're seeing, because those
should be filtered out.

The problem you're encountering is that the reminders feature is
designed to show reminders at the end of a run, but sbt models
multi-project builds as multiple runs. Did you consider adding a File
reporter for each project so you could see one report per project? We
have also had this challenge with the HTML reporter, as people wanted
one ueber-HTML report for their multi-project build, but ScalaTest
needs to write a summary page at the end of the run. Well, because sbt
gives ScalaTest multiple runs, you need separate HTML reports.

Anyway, it is worth trying to figure out how to achieve it, but it is
really more a question of improving ScalaTest/sbt integration than it
is something ScalaTest can do on its own.

Bill

On Wed, Aug 20, 2014 at 8:18 AM, Niklas Lochschmidt

[Niklas Lochschmidt]

Hi Bill

As strange as this might sound, the Ugly mode is exactly what I was
missing.

The events that are still printed look like this:
> Alert Provided - SetSpec: A mutable Set should allow an element to be
> added: alerts are also sent immediately
> Note Provided - SetSpec: Feature: An element can be added to an empty
> mutable Set Scenario: When an element is added to an empty mutable
> Set: notes are sent immediately
https://travis-ci.org/nlochschmidt/scalatest-sbt-output/builds/33097277

I didn't look into the FileReporter because afaik it would have had the
same limitations as the normal reporter and on the CI server I can't
really look at the individual files. That means at the end of the test
I would have to cat them to the build output which then again wouldn't
bring me any benefit over using the std out reporter in the first place.

Anyways I am pretty satisfied with "-oUNCXHELPOQRM" :-) as it allows me
to quick and accurately determine what's wrong with the build.

Cheers
Niklas
-------------------------------
Developer at vaamo.de

[Bill Venners]

Hi Niklas,

On Wed, Aug 20, 2014 at 11:33 AM, Niklas Lochschmidt
<nlochs...@gmail.com> wrote:
> Hi Bill
>
> As strange as this might sound, the Ugly mode is exactly what I was
> missing.
>
> The events that are still printed look like this:
>>
>> Alert Provided - SetSpec: A mutable Set should allow an element to be
>> added: alerts are also sent immediately
>> Note Provided - SetSpec: Feature: An element can be added to an empty
>> mutable Set Scenario: When an element is added to an empty mutable Set:
>> notes are sent immediately
>
> https://travis-ci.org/nlochschmidt/scalatest-sbt-output/builds/33097277
>
> I didn't look into the FileReporter because afaik it would have had the
> same limitations as the normal reporter and on the CI server I can't
> really look at the individual files. That means at the end of the test
> I would have to cat them to the build output which then again wouldn't
> bring me any benefit over using the std out reporter in the first place.
>
> Anyways I am pretty satisfied with "-oUNCXHELPOQRM" :-) as it allows me
> to quick and accurately determine what's wrong with the build.
>

Yes, you say you're happy, but there's a little cry for HELP embedded
in your configuration! We'll try to come up with something better for
multi-project builds in sbt, but as I said before, it is not obvious
how to do it. We'll work with the sbt folks on it.

Bill

[Niklas Lochschmidt]

Hi Bill

Wow you actually caught that! ;-)

Many thanks to you and Chee Seng for the great help you provide.

Cheers
Niklas
-------------------------------
Developer at vaamo.de