New ScalaTest 1.0-SNAPSHOT

[Bill Venners]

Hi All,

I just released a fresh ScalaTest 1.0 snapshot, both for 2.7.5 and the
latest nightly 2.8 build. Info on how to access it is here:

http://www.artima.com/scalatest/#snapshot

Some people have recently posted about trying to get ScalaTest 1.0 to
build under 2.8. It requires a few hand changes each time I do it,
because there is some code that simply won't compile under both 2.7
and 2.8. Each time I do this make a new branch, and from now on I will
check in the changes I make to get it to build under 2.8. If you want
to work with it, you can get tonight's build here:

svn checkout https://scalatest.dev.java.net/svn/scalatest/branches/app/trunk-for-scala-2.8f
--username guest

I compiled it with the latest 2.8 nightly build (downloaded today):

scala-2.8.0.r18678-b20090910020815

The only tasks remaining are more testing of the
org.scalatest.concurrent package, a bit more work in the
org.scalatest.mock package, and documentation. With luck the next
snapshot will be a release candidate. If you have time please give it
a test drive and post feedback to scalatest-users.

Thanks.

Bill
----
Bill Venners
Artima, Inc.
http://www.artima.com

[Cay Horstmann]

Thanks very much! It works like a charm for me. My students will be
delighted--I originally had them go back to 2.7.6, and that version's
Eclipse plugin seems to have more problems than the latest one.

ScalaTest has worked out very nicely for me for grading homework :-) I
really enjoyed the fact that I was able to choose the testing style
that was best for my situation.

Just one question--how do I suppress the exception trace when an
assertion fails? I mean, if I get a report that foo isn't actually
bar, that's good enough for me. I don't care to see the life story of
the test runner.

Thanks,

Cay

On Sep 20, 7:21 pm, Bill Venners <b...@artima.com> wrote:
> Hi All,
>
> I just released a fresh ScalaTest 1.0 snapshot, both for 2.7.5 and the

> latest nightly 2.8 build...If you have time please give it

[Joakim Ohlrogge]

Tried it form eclipse and buildr. Works like charm (buildr reports that the tests fail but that seems to be a problem with failure detection in compination with using BDD-style (extends Spec)).

Thanks for staying up to date (eclipse only has the latest nightly build available so new eclipse users will probably appreciate it :))

--
-----------------------------------------------------
Joakim Ohlrogge
Agical AB
Västerlånggatan 79, 2 tr
111 29 Stockholm, SWEDEN

Mobile: +46-708-754004
Blog: johlrogge.wordpress.com
E-mail: joakim....@agical.se

[Bill Venners]

Hi Cay,

On Sun, Sep 20, 2009 at 9:33 PM, Cay Horstmann <c...@horstmann.com> wrote:
>
> Thanks very much! It works like a charm for me. My students will be
> delighted--I originally had them go back to 2.7.6, and that version's
> Eclipse plugin seems to have more problems than the latest one.
>
> ScalaTest has worked out very nicely for me for grading homework :-) I
> really enjoyed the fact that I was able to choose the testing style
> that was best for my situation.
>
> Just one question--how do I suppress the exception trace when an
> assertion fails? I mean, if I get a report that foo isn't actually
> bar, that's good enough for me. I don't care to see the life story of
> the test runner.
>

Which reporter or reporters are you using? The only ones in which the
stack trace can really get in your face is the standard out, standard
err, or file reporters.

The story of stack traces is that I wanted to try and save people from
needing to scan through them to find the line of code for failed
tests, because that slows people down. When an assertion or matcher
expression fails, it does so with a TestFailedException, which mixes
in trait StackDepth. A StackDepth exception contains an integer stack
depth which is the index in the stack trace array of the line of code
of the failed assertion. There are other StackDepthExceptions as well,
JUnitTestFailedError for when you're using JUnit,
DuplicateTestNameException, etc. The reporters highlight the file name
and line number of the offending line of test code so you don't have
to scan, when the exception is a StackDepth. If it isn't a StackDepth,
such as if the code under test through some random exception, then the
reporters can only give you the stack trace.

However, I also show at least a truncated stack trace in the case of
StackDepths, because what can happen is you may factor out duplicate
code into a method. The method has an assertion that may fail, and if
you don't have any stack trace, you won't know which call site failed.
For example I have this helper method:

def ensureTestFailedEventReceived(suite: Suite, testName: String) {
val reporter = new EventRecordingReporter
suite.run(None, reporter, new Stopper {}, Filter(), Map(), None,
new Tracker)
val testFailedEvent =
reporter.eventsReceived.find(_.isInstanceOf[TestFailed])
assert(testFailedEvent.isDefined)
assert(testFailedEvent.get.asInstanceOf[TestFailed].testName === testName)
}

If the second assertion fails, the stack depth will point to that
assertion. I'll also get the name of the test that failed, but I'll
still want to know more if I call this helper method multiple times
from the same test, like this:

test("something") {
// ...
ensureTestFailedEventReceived(spec1, "should blow up")
// ...
ensureTestFailedEventReceived(spec2, "should blow up")
}

Which one of these was the one that failed? spec1 or spec2? So that's
why I show around 7 lines of stack trace in the standard out, standard
err, and file reporters for StackDepth exceptions. I cut off the top,
which is above the stack depth, show the next seven lines of stack
trace, then truncate the rest. I'm figuring you'd have to do a lot of
factoring out into helper methods for those 7 lines of code to not be
enough.

If it isn't a StackDepth exception, you get the whole stack trace, and
if someone always want full stack traces, you can configure that by
adding a config param F to the reporter arg, like a -oF on the command
line will get you a standard out reporter that always shows full stack
traces for everything.

By the way, what RSpec, a Ruby BDD tool, does is put a footnote next
to a failure message, and then show the stack trace at the bottom.
When doing BDD like tests, a stack trace can make the output a bit
ugly. But I think when you have a failing test you really want to know
the info as quickly as possible, so I didn't want you to have to scan
for footnotes. By truncating it most of the time, my hope was that you
get quick access to the info without completely cluttering up your
nice spec-like output.

Bill

> Thanks,
>
> Cay
>
> On Sep 20, 7:21 pm, Bill Venners <b...@artima.com> wrote:
>> Hi All,
>>
>> I just released a fresh ScalaTest 1.0 snapshot, both for 2.7.5 and the
>> latest nightly 2.8 build...If you have time please give it
>> a test drive and post feedback to scalatest-users.
>
> >
>

--

[Bill Venners]

Hi Joakim,

A saw that you tweeted that with a patch to buildr you got this all
working under 2.8. Can you elaborate on what the patch was? I'd like
to understand what was needed.

Thanks.

Bill

--

[Joakim Ohlrogge]

Absolutely. Basically I did three things:

1. The obvious thing, change the version to be downloaded by buildr to 1.0-for-scala-2.8.0-SNAPSHOT

2. Change the maven group from org.scala-tools.testing to org.scalatest

These two changes were pretty straight forward though the group was not parameterized in buildr while I could probably get away with changing the scalatest-version by setting some variable in my project (but I didn't look into that, i just hacked the scala/tests.rb file directly.

Now for the strangest thing I did: The buildr integration looked at scalatests console output to determine if any tests failed. Since the output looks a bit different when extending Spec I had to patch the condition that checked if the run had competed.

3. I changed the regexp: /Run completed\./ to  /(Run completed\.)|(All tests passed\.)/

I meant to ask you what would be the best way to do this since the buildr integration I looked into (which is Daniel Spiewak's git-branch, unofficial and probably work in progress) seems a bit hacky. Buildr uses the ant-task under the hood, maybe I can get the same report output to file for all the different runners and it would be a more reliable strategy to parse those?

Also the "completed" regexp seems to be there for this reason:

# Parse for failures, errors, etc.

# This is a bit of a pain right now because ScalaTest doesn't flush its

# output synchronously before the Ant test finishes so we have to loop 

# and wait for an indication that the test run was completed. 

So a flush at the end of a run (if 1.0 does not already do this) would maybe speed things up a bit as well as relying only on the failure regexp:

/(TEST FAILED -)|(RUN STOPPED)|(RUN ABORTED)/

Which seems to work for the Spec style as well.

I meant to commit my changes to my git-branch so any suggestions for improved strategies for failure detection etc would be most welcome.

You can have a look at my clone here:

http://github.com/johlrogge/buildr

At the moment I have not pushed any changes back so it is identical to Daniel's clone.

Hope this helps

/J

[Bill Venners]

Hi Joakim,

There's actually a method in org.scalatest.tools.Runner intended for
this use case. Runner has a main method and a run method. The run
method takes the same args as main, but returns a Boolean. Here's its
doc comment:

Runs a suite of tests, with optional GUI. See the main documentation
for this singleton object for the details. The difference between this
method and main is simply that this method will block until the run
has completed, aborted, or been stopped, and return true if all tests
executed and passed. In other words, if any test fails, or if any
suite aborts, or if the run aborts or is stopped, this method will
return false. This value is used, for example, by the ScalaTest ant
task to determine whether to continue the build if haltOnFailure is
set to true.

Does it sound like buildr could call this method directly?

Bill

[Joakim Ohlrogge]

Thanks for the tip, I would have to look into that I'm not that comfortable with the inner workings of buildr yet (or as cool people say: I'm a buildr n00b :)). At the moment buildr uses the ant-task which from what I understand is how things generally are done in buildr but it may be straight forward. Or is there a flag for the ant-task that gives the same behavior?

/J

[Bill Venners]

Hi Joakim,

Yes, there's a haltonfailure flag in the ant task. That may be what
they'd rather use in buildr:

<scalatest fork="true" runpath="${build.tests.jar}" parallel="true"
haltonfailure="true">

Bill

On Tue, Sep 22, 2009 at 12:27 AM, Joakim Ohlrogge