Timout testing

[Fabio Cechinel Veronez]

Hello,

I have a scenario where I want to test if a call really returned
before X seconds, throwing an error if it takes more than that. If i
were using junit i would use Timout rule [1].

Is org.scalatest.concurrent.Conductor the right way to go when using scalatest?

[1] http://kentbeck.github.com/junit/javadoc/latest/org/junit/rules/Timeout.html
--
Fabio Cechinel Veronez

[Bill Venners]

Hi Fabio,

Conductor isn't really meant for that, no. What you need is a timeout
construct, which I didn't want to add to ScalaTest until someone asked
for it from whom I could get requirements. In the meantime, you'll
need to roll your own. I don't have time today to look into it, but
I'd suggest you look at java.util.concurrent, which probably has
something that will let you run a bit of code with a timeout. Anyone
else have some suggestions?

BIll

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
>

--
Bill Venners
Artima, Inc.
http://www.artima.com

[Sean Griffin]

Fabio,
Well, an easy way to get this behavior is to simply define a trait
with a timeoutAfter method or something similar that accepts your test
code as a function value and sends it off to a Future. Something like
so:

trait Timed {
def timeoutAfter(timeout: Long)(codeToTest: => Unit): Unit = {
val executor = Executors.newSingleThreadExecutor
val future = executor.submit(new Runnable {
def run = codeToTest
})

try {
future.get(timeout, TimeUnit.MILLISECONDS)
}
finally {
executor.shutdown()
}
}
}

and then in your test:

class Test extends Suite with Timed {
def testTooLong() {
timeoutAfter(1000) {
Thread.sleep(3000)
}
}
}

This has the advantage that you can have some tests with a timeout
mixed in the same suite with other tests that don't have a timeout
restriction, but it's also more cumbersome if you actually do want all
tests in the suite to abide by the same timeout. So, I tried playing
around with a different approach that actually integrates with the
Suite trait.

trait Timed extends AbstractSuite {
this: Suite =>

protected val timeout: Long
protected val timeoutUnit = TimeUnit.MILLISECONDS

abstract protected override def runTest(testName: String, reporter:
Reporter, stopper: Stopper, configMap: Map[String, Any], tracker:
Tracker): Unit = {
val executor = Executors.newSingleThreadExecutor
val future = executor.submit(new Runnable {
def run = TimedTests.super.runTest(testName, reporter, stopper,
configMap, tracker)
})

try {
future.get(timeout, timeoutUnit)
}
catch {
case te: TimeoutException => fail("Test exceeded defined timeout
period.")
}
finally {
executor.shutdown()
}
}
}

class Test extends Suite with Timed {
override val timeout = 1000L
def testTooLong() {
Thread.sleep(3000)
}
}

However, this only comes close. The problem is that the test will
report a failure due to exceeding the timeout period, but it will then
go ahead and report a success anyway since the code under test itself
did not error and the Suite's runTest implementation will report a
TestSucceeded event in this case. I'm not sure how to solve the
problem--maybe Bill or someone else will have an idea--but this might
give you something to play around with.

-Sean

On Jun 17, 3:12 pm, Bill Venners <b...@artima.com> wrote:
> Hi Fabio,
>
> Conductor isn't really meant for that, no. What you need is a timeout
> construct, which I didn't want to add to ScalaTest until someone asked
> for it from whom I could get requirements. In the meantime, you'll
> need to roll your own. I don't have time today to look into it, but
> I'd suggest you look at java.util.concurrent, which probably has
> something that will let you run a bit of code with a timeout. Anyone
> else have some suggestions?
>
> BIll
>
> On Fri, Jun 17, 2011 at 12:57 PM, Fabio Cechinel Veronez
>
>
>
>
>
>
>
>
>
> <fabio.vero...@gmail.com> wrote:
> > Hello,
>
> > I have a scenario where I want to test if a call really returned
> > before X seconds, throwing an error if it takes more than that. If i
> > were using junit i would use Timout rule [1].
>
> > Is org.scalatest.concurrent.Conductor the right way to go when using scalatest?
>

> > [1]http://kentbeck.github.com/junit/javadoc/latest/org/junit/rules/Timeo...

[Bill Venners]

Hi Sean,

Thanks for your suggestion for Fabio. As to your question about how
best to run all tests in the same suite with a timeout, I'd probably
suggest overriding withFixture rather than runTests. I'll come back to
that, but first let me run by you and Fabio what I'd been imagining
might make sense to add to ScalaTest. I wanted to wait until someone
asked so I could have a real use case.

For ScalaTest I had been imagining a Timeout or Timeouts trait that
has a withTimeout method, that would take an implicit config parameter
that determines the timeout, but it could be overridden. So the client
code would look like:

withTimeout {
// test code you want to execute within the default timeout
}

withTimeout (1000) {
// Test code you'd want to execute within a 1000 millisecond timeout
}

You could also override for all uses of withTimeout in scope by
overriding the implicit

implicit override val timeoutConfig = new TimeoutConfig(100) // define
a new default

withTimeout {
// now this has to complete within 100 milliseconds
}

As Sean suggested, if you want to just do this in a few tests you can
just wrap those places with a withTimeout call. If you want all tests
to run with a timeout, I'd put it in withFixture. This is described,
for example, here (scroll down a bit until you see Overriding
withFixture(NoArgTest)):

http://www.scalatest.org/scaladoc-1.6.1/org/scalatest/Suite.html#sharedFixtures

override def withFixture(test: NoArgTest) {
withTimeout {
test()
}
}

If it times out, it will be reported as a test failure, which I
suspect is what most people would want.

One question I have is should withTimeout actually return a value that
results from the code passed to it. Seems like it would make it more
generally useful, and people who don't care would just ignore the
result.

How does that sound for your use case, Fabio? In fact, if you could
elaborate a bit more on exactly what you're trying to do that you want
to make sure happens within a time period that would be useful to
know.

Thanks.

Bill

[Luigi]

Hi all. I'd like to +1 for timeouts. I use Scalatest with SBT from within IntelliJ, and if the method I'm testing goes into an infinite recursion or is taking too long, the only option seems to be to kill the SBT session (SBT could do with a facility to stop testing without killing it, but just timing out the problem test would be better).

I'd also like to be able to time each test and show the run time next to the name of the test in the output - perhaps this could be part of the same solution?

My use-case is this: I've written a parser that takes problem statements from TopCoder competitions and spits out fully formed test scripts (example below). TopCoder doesn't support Scala so I have to run the tests locally, and it would be good to check that methods are taking less than the specified 2 seconds. Also if I've coded different implementations it would be nice to see which is faster.

There's a lot more that could be done if performance testing facilities were made available (timing not just one run but 10 million, and seeing how the JVM reacts, for example) and I think this would be great for improving performance and highlighting inefficient code. It would also differentiate Scalatest from its peers. But just a quick and simple timeout / timer facility would be really useful IMO.

package topcoder.test

import org.scalatest.FunSuite
import org.scalatest.matchers.ShouldMatchers
import topcoder.TCHS026_2._

class TCHS026_2_TEST extends FunSuite with ShouldMatchers {
  test("Case 0") { getLength("123") should equal (3) }
  test("Case 1") { getLength("10342(76)") should equal (8) }
  test("Case 2") { getLength("33(562(71(9)))") should equal (19) }
  test("Case 3") { getLength("0(0)") should equal (0) }
  test("Case 4") { getLength("1(1(1(1(1(1(1(0(1234567890))))))))") should equal (0) }
  test("Case 5") { getLength("1()66(5)") should equal (7) }

[Bill Venners]

Hi Luigi,

Ack, I wrote this this morning and thought I'd sent it out. Found it
just now sitting here. Sorry for the delay.

On the second subject, you can get durations printed out with the
following in (old) sbt:

Seq(TestArgument(TestFrameworks.ScalaTest, "-oD"))

http://www.scalatest.org/scaladoc-1.6.1/#org.scalatest.tools.ScalaTestFramework

Sorry I suspect that may be for sbt 0.9.x rather than 0.10.x. Can
someone post what it is for sbt 0.10.x?

On the first subject, timeouts, can you let me know what you'd like to
see happen when the timeout occurs? My guess is you'd want to see a
test failure. That means you're using timeouts to ensure a test
doesn't take longer than a specified max.

Another thing it could do (in the next version of ScalaTest, where
I'll be adding the timeouts) is "cancel" the test, which would mean
that this time it took too long, and I don't consider that a failure,
but I just want to note it this time. Next time hopefully the network
will be faster and the test will finish within the timeout.

Thanks.

Bill

[Luigi]

Hi Bill, thanks for the info about sbt. I now have the timing working, which is great! For 0.10 the build.sbt line is:

testOptions in Test += Tests.Argument ("-oD")

Or it can be called from the command line by adding "--" followed by the options, e.g.

test-only topcoder.test.TCHS028_2_TEST -- -oD

Regarding the timeout behaviour, for me it's a failure, but it's important to be able to distinguish between a test that failed because the return value was incorrect. The contents of the info message would be sufficient e.g. a normal failure looks like this:

[info] - Case 4 *** FAILED ***
[info]   20911 did not equal 117 (SRM519_2_TEST.scala:24)

and a timeout failure could look like this:

[info] - Case 5 *** FAILED ***
[info]   Test did not complete within 2000 ms

For my use-case I would want a timeout to fail the suite overall. I can't think of a situation where I'd want a test to be "cancelled" and not failed if it were taking too long. If it's testing a stable production code-base with the possibility of a slow network I'd have thought the solution would be to set the timeout higher, and if on some occasion it's taking too long, that's probably one of the things you were trying to test for. Other people may have a use for an alternative failure mode as you mention, but unless it's specifically requested, it might be simpler to stick with a simple pass / fail.

[Bill Venners]

Hi All,

I have uploaded a new 1.8-SNAPSHOT with the timeouts functionality. It
is built with Scala 2.9.0 and deployed for 2.9.0, 2.9.0-1, and 2.9.1.
These are on oss.sonatype.org now, to which scala-tools.org is
redirecting. You can see them here:

https://oss.sonatype.org/content/repositories/snapshots/org/scalatest/

There are two main traits concerned with timeouts: Timeouts and
TimeLimitedTests.

Timeouts provides a failAfter method (in 2.0 there will also be a
cancelAfter added) that let's you specify a time limit for a bit of
code, like this:

failAfter(100) {
Thread.sleep(200)
}

The duration values are milliseconds. I added a TimeSugar trait that
you can use if you want to specify units:

import org.scalatest.TimeSugar._
failAfter(100 millis) {
Thread.sleep(200 millis)
}

If you're using a library that already already offers such sugar, you
can use it instead, so long as there's an implicit conversion from
that libraries result to a Long number of milliseconds.

The code is executed in the main test thread, the thread that invokes
failAfter, so you don't need to worry about synchronization. A second
thread is fired off inside a timer to keep track of the timeout, and
to attempt to interrupt the main thread if the timeout expires. An
interruption strategy is passed as an implicit parameter. More info is
here:

http://www.artima.com/docs-scalatest-1.8-14.Feb.2012/#org.scalatest.concurrent.Timeouts

The other main timeout trait is TimeLimitedTests. This trait overrides
withFixture and wraps each test in a failAfter call using a timeLimit
val that you must specify as a member. It passes a
defaultTestInterruptor that you can override. This trait lets you
specify a time limit for all tests in a suite. More info is here:

http://www.artima.com/docs-scalatest-1.8-14.Feb.2012/#org.scalatest.concurrent.TimeLimitedTests

Please give it a try and post feedback and suggestions to
scalatest-users. I plan to release all these new
concurrency/assynchrony tools in 1.8, which I plan to start RCing
around the end of February. So the more people who can try them out in
the meantime the better.

Thanks.

Bill

On Fri, Jun 17, 2011 at 12:57 PM, Fabio Cechinel Veronez
<fabio....@gmail.com> wrote: