why does TestFailedException extend RuntimeException (a pain point for a common test use case)

[ano...@ariasystems.com]

it is a common coding practice in scala (as in Java) to have code that throws unchecked exceptions (ie: RuntimeException) when coding assumptions are broken. Most test libraries realize this and try to accommodate this use case by implementing their assertion / failure detection as Errors (usually as AssertionErrors) and not as RuntimeExceptions. The java / scala standard is that assertion failures are Errors and not Exceptions. This is because code should not catch Errors as Errors by their definition live outside the responsibility of the program to manage. 

Please don't try to dispute this. It's a fundamental plank that the Java / Scala community is based upon. 

Unfortunately ScalaTest does not implement this strategy and that unfortunately creates a large pain point for those who use scalatest to implement a negative testing (testing to ensure exceptions are thrown). It seems that the ScalaTest design didn't envision that developers would commonly be writing code that throws RuntimeExceptions and would need to develop a negative test to test that scenario.

The common way to write this sort of negative test is:

 

    scenario("negative test that  should throw a RuntimeException") {

      try {

        doSomething("anInvalidParameter")

        fail("exception not thrown")

      } catch {

        case e: RuntimeException => //passes

      }

    }

the problem is that the scalatest fail() method throws a TestFailedException (which is an extension of a RuntimeException and not a AssumptionError). The ramifications of this is that this commonly written test pattern will always pass. This is because either the RuntimeException is thrown by the method being tested or a TestFailedException is thrown by the test and in both cases they are RuntimeExceptions. This is doubly bad because many developers who are writing tests implementing this sort of pattern will not realize that the test isn't effectively working since it will pass by default. They'd have to purposely test it against code that they know should not throw an error (like I did) to know there is an issue here.

The way to work around the scalatest class heirarchy flaw is to write the test like this:

    scenario("negative test that should throw a RuntimeException") {

      try {

        doSomething("anInvalidParameter")

        fail("exception not thrown")

      } catch {

        case e: TestFailedException => throw e

        case e: RuntimeException => //passes

      }

    }

the problem here is that most developers won't realize the need to do this since it is a common fundamental plank in the java / scala community that test failures are Errors and not to be handled as Exception since test failures do not fall in the responsibility of an application to catch or handle. They truly need to live as Errors so that unexpected consequences like the scenario mentioned above do not happen!

[Bill Venners]

Hi,

ScalaTest offers assertions and matcher expressions to test for expected exceptions, for example:

assertThrows[StringIndexOutOfBoundsException] {

  "hi".charAt(3)

}

You can use intercept instead of assertThrows if you want to inspect the exception further:

val ex = intercept[StringIndexOutOfBoundsException] {

  "hi".charAt(3)

}

assert(ex.getMessage == "String index out of range: 3")

I don't recall seeing people test for RuntimeException. Usually people test for a more specific exception, like IllegalArgumentException.

The matcher syntax looks like:

a [StringIndexOutOfBoundsException] should be thrownBy "hi".charAt(3)

and if you want to perform subsequent assertions:

val ex = the [StringIndexOutOfBoundsException] thrownBy "hi".charAt(3)

ex should have message "String index out of range: 3"

Hopefully this clarifies ScalaTest's approach on these matters.

Bill

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalatest-users@googlegroups.com
To unsubscribe from this group, send email to
scalatest-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bill Venners
Artima, Inc.
http://www.artima.com

[ano...@ariasystems.com]

thats good to know. 

still though, I just wanted to let you know that the common way that developers write exception tests would silently always pass and that is probably a bad thing (since most developers come into using this framework with past experiences with other frameworks).

It would seem like scalatest should be written where this common usage wouldn't "stealth" pass.

The steathyness is the problem since most developers would see the test always pass and not realize that the test isn't actually doing anything

also not that RuntimeExceptions (or any type of exception) is a valid way to handle test failures which by definition should be some subclass of Error (most likely AssertionError)

http://stackoverflow.com/questions/912334/differences-between-exception-and-error

To post to this group, send email to scalate...@googlegroups.com

To unsubscribe from this group, send email to

scalatest-use...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.