Scalatests do not fail gracefully when stopped with the JUnitRunner

[J Knurek]

I'm using the JUnitRunner to run my tests. http://www.scalatest.org/user_guide/using_junit_runner

This is because our full collection of tests is a mix of java & scala, and we use the maven-surefire-plugin to execute the tests.

I've implemented the functionality in the RunListener to stop the tests from continuing when there is a failure using pleaseStop()

http://junit.sourceforge.net/javadoc/org/junit/runner/notification/RunNotifier.html#pleaseStop()

What this does is that it causes the all the future tests to throw a StoppedByUserException. 

This works perfect in Java I can catch that exception, mark the test as ignored, and as a bonus, stop any future tests from running. When a test fails in Scala however, all the future tests are marked as failed, and there is some other reporter repeating the exception. What appears to be the problem is that something deeper (and this is what I can't figure out) is catching the exception and not allowing org.scalatest.junit.JUnitRunner to throw the exception to the executor.

Here is what I'm seeing in the console:

Reporter completed abruptly with an exception after receiving event: TestStarting(Ordinal(0, 5), InventoryTest, sc.core.tests.integration.InventoryTest, Some(sc.core.tests.integration.InventoryTest), A product's inventory should be incremented, should be incremented, Some(MotionToSuppress), Some(LineInFile(32,InventoryTest.scala)), Some(sc.core.tests.integration.InventoryTest), None, ScalaTest-running-InventoryTest, 1398971791790).

org.junit.runner.notification.StoppedByUserException

  at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:110)

  at org.scalatest.junit.RunNotifierReporter.apply(RunNotifierReporter.scala:61)

  at org.scalatest.WrapperCatchReporter.doApply(CatchReporter.scala:70)

  at org.scalatest.CatchReporter$class.apply(CatchReporter.scala:36)

  at org.scalatest.WrapperCatchReporter.apply(CatchReporter.scala:63)

  at org.scalatest.Suite$.reportTestStarting(Suite.scala:2095)

  at org.scalatest.SuperEngine.runTestImpl(Engine.scala:267)

  at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1645)

  at sc.core.tests.TestBase.org$scalatest$BeforeAndAfter$$super$runTest(TestBase.scala:28)

  at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:200)

  at sc.core.tests.TestBase.runTest(TestBase.scala:28)

  at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1703)

  at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1703)

  at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)

  at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

  at scala.collection.immutable.List.foreach(List.scala:309)

  at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

  at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)

  at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)

  at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

  at scala.collection.immutable.List.foreach(List.scala:309)

  at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

  at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)

  at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)

  at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1703)

  at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)

  at org.scalatest.Suite$class.run(Suite.scala:1423)

  at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)

  at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1749)

  at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1749)

  at org.scalatest.SuperEngine.runImpl(Engine.scala:545)

  at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1749)

  at sc.core.tests.TestBase.org$scalatest$BeforeAndAfter$$super$run(TestBase.scala:28)

  at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:241)

  at sc.core.tests.TestBase.run(TestBase.scala:28)

  at org.scalatest.junit.JUnitRunner.run(JUnitRunner.scala:99)

  at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:288)

  at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:167)

  at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:134)

  at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)

  at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)

  at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

[Bill Venners]

Hi J,

It looks like we need to enhance our JUnitRunner to handle such exceptions. This is the first I'd heard of them, so it is unlikely to deal with those exceptions as yet! Can you let me know what version of ScalaTest you are using? We'll try and add that as an enhancement. It is the least we could do since they are polite enough to say "please" in the pleaseStop() method name.

Bill

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Bill Venners
Artima, Inc.
http://www.artima.com

[J Knurek]

Hi Bill,

Currently we're using 2.0. 

I'm trying to upgrade to 2.15, so that I can then upgrade and implement the fix. But I'm running into an issue with the Surefire plugin. 

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18-SNAPSHOT:test (default-test) on project spcore: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.18-SNAPSHOT:test failed: There was an error in the forked process

java.lang.NoClassDefFoundError: org/scalatest/Matchers$ResultOfCollectedString

at java.lang.Class.getDeclaredMethods0(Native Method)

at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)

at java.lang.Class.getMethod0(Class.java:2774)

at java.lang.Class.getMethod0(Class.java:2783)

at java.lang.Class.getMethod(Class.java:1663)

at org.apache.maven.surefire.util.ReflectionUtils.tryGetMethod(ReflectionUtils.java:57)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.isSuiteOnly(JUnit3TestChecker.java:64)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.isValidJUnit3Test(JUnit3TestChecker.java:59)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.accept(JUnit3TestChecker.java:54)

at org.apache.maven.surefire.common.junit4.JUnit4TestChecker.accept(JUnit4TestChecker.java:52)

at org.apache.maven.surefire.util.DefaultScanResult.applyFilter(DefaultScanResult.java:97)

at org.apache.maven.surefire.junit4.JUnit4Provider.scanClassPath(JUnit4Provider.java:230)

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:109)

at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

Caused by: java.lang.ClassNotFoundException: org.scalatest.Matchers$ResultOfCollectedString

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

... 16 more

My assumption is that it's related to this warning, but still trying to track it down....

[INFO] --- scala-maven-plugin:3.1.6:testCompile (default) @ spcore ---

[WARNING]  Expected all dependencies to require Scala version: 2.10.3

[WARNING]  com.fasterxml.jackson.module:jackson-module-scala_2.10:2.1.5 requires scala version: 2.10.0

[WARNING] Multiple versions of scala libraries detected!

[J Knurek]

this looks like the relevant bit of the stack trace for this upgrade issue

Caused by: java.lang.RuntimeException: There was an error in the forked process

java.lang.NoClassDefFoundError: org/scalatest/Matchers$ResultOfCollectedString

at java.lang.Class.getDeclaredMethods0(Native Method)

at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)

at java.lang.Class.getMethod0(Class.java:2774)

at java.lang.Class.getMethod0(Class.java:2783)

at java.lang.Class.getMethod(Class.java:1663)

at org.apache.maven.surefire.util.ReflectionUtils.tryGetMethod(ReflectionUtils.java:57)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.isSuiteOnly(JUnit3TestChecker.java:64)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.isValidJUnit3Test(JUnit3TestChecker.java:59)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.accept(JUnit3TestChecker.java:54)

at org.apache.maven.surefire.common.junit4.JUnit4TestChecker.accept(JUnit4TestChecker.java:52)

at org.apache.maven.surefire.util.DefaultScanResult.applyFilter(DefaultScanResult.java:97)

at org.apache.maven.surefire.junit4.JUnit4Provider.scanClassPath(JUnit4Provider.java:232)

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:109)

at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

What's interesting is that I'm getting this error with all Scalatest versions 2.1.x. But when I tried out building the latest into 2.2.0-SNAPSHOT I get a slightly different error (similar stacktrace though)

Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.18-SNAPSHOT:test failed: There was an error in the forked process

java.lang.VerifyError: Cannot inherit from final class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.getDeclaredMethods0(Native Method)

at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)

at java.lang.Class.getMethod0(Class.java:2774)

at java.lang.Class.getMethod0(Class.java:2783)

at java.lang.Class.getMethod(Class.java:1663)

at org.apache.maven.surefire.util.ReflectionUtils.tryGetMethod(ReflectionUtils.java:57)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.isSuiteOnly(JUnit3TestChecker.java:64)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.isValidJUnit3Test(JUnit3TestChecker.java:59)

at org.apache.maven.surefire.common.junit3.JUnit3TestChecker.accept(JUnit3TestChecker.java:54)

at org.apache.maven.surefire.common.junit4.JUnit4TestChecker.accept(JUnit4TestChecker.java:52)

at org.apache.maven.surefire.util.DefaultScanResult.applyFilter(DefaultScanResult.java:97)

at org.apache.maven.surefire.junit4.JUnit4Provider.scanClassPath(JUnit4Provider.java:230)

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:109)

at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

[Bill Venners]

Hi J,

This definitely looks like a version mismatch of some kind. Is it possible that you have some other ScalaTest version getting pulled into your build via a non-obvious transitive dependency? It has been a while, but there was one open source project that accidentally included ScalaTest as a required dependency of using that library. I would run a tool to see exactly what all your transitive dependencies are, and look to see if you see an extra ScalaTest in there somewhere.

Bill

[J Knurek]

Indeed - it looks like the swagger-core_2.10 library has a dependency on org.scala-lang:scalap:jar:2.10.0:compile 

I'll need to figure out the "correct" way to handle that, but for now I've forced it to use 2.10.3 and scalatest 2.1.5 is running.

I got this error with 2.2.0-SNAPSHOT

java.lang.IncompatibleClassChangeError: Class sc.core.tests.integration.InventoryTest does not implement the requested interface org.scalautils.Explicitly

Should be an easy fix, but wasn't expecting the breaking change.