ScalaTest 1.0
Ben James
Scala 2.7.6.
FIrst part of the error traceback is:
java.lang.NoClassDefFoundError: org/scalatest/Reporter$class
at sbt.impl.ScalaTestRunner$ScalaTestReporter.<init>
(TestFrameworkImpl.scala:56)
I looked in the ScalaTest 1.0 jar and indeed there is no Reporter
$class.class, where there was in ScalaTest 0.9.5
Are there plans to support ScalaTest 1.0 soon?
Thanks,
Ben
Bill Venners
Josh Cough created a class you can use to get it to work. We are still
negotiating whether to add this class to ScalaTest or sbt, banging the
dependency back and forth like playing ping pong.
Here is the link to the code:
https://scalatest.dev.java.net/source/browse/scalatest/branches/josh-sbt-2/app/src/main/scala/org/scalatest/sbt/ScalaTestRunner.scala?rev=2017&view=markup
For anyone that wants to use sbt with scalatest at this very second,
you can do this in two steps:
1) Put ScalaTestRunner in your own source code in the org.scalatest.sbt package.
2) Add this line to your sbt project file:
override def compileClasspath = super.compileClasspath +++
Path.fromFile(FileUtilities.sbtJar)
Josh said, though, that the last time I tried this was at least a week
back. Here is the code.
package org.scalatest.sbt
import _root_.sbt._
/**The test runner for ScalaTest suites. */
class ScalaTestRunner(val log: Logger, val listeners: Seq[TestReportListener],
val testLoader: ClassLoader) extends BasicTestRunner
{
import _root_.java.lang.reflect.Modifier
def runTest(testClassName: String): Result.Value = {
val testClass = Class.forName(testClassName, true,
testLoader).asSubclass(classOf[Suite])
if( isAccessibleSuite(testClass)){
val test = testClass.newInstance
val reporter = new ScalaTestReporter
test.run(None, reporter, new Stopper {}, Filter(), Map.empty,
None, new Tracker)
if (reporter.succeeded) Result.Passed else Result.Failed
} else{
Result.Passed
}
}
private val emptyClassArray = new Array[java.lang.Class[T] forSome {
type T }](0)
private def isAccessibleSuite(clazz: java.lang.Class[_]): Boolean = {
try {
classOf[Suite].isAssignableFrom(clazz) &&
Modifier.isPublic(clazz.getModifiers) &&
!Modifier.isAbstract(clazz.getModifiers) &&
Modifier.isPublic(clazz.getConstructor(emptyClassArray: _*).getModifiers)
} catch {
case nsme: NoSuchMethodException => false
case se: SecurityException => false
}
}
/**An implementation of Reporter for ScalaTest. */
private class ScalaTestReporter extends org.scalatest.Reporter with NotNull
{
import org.scalatest.events._
var succeeded = true
def apply(event: Event) {
event match {
// why log.info sometimes and fire(MessageEvent...) other times?
case rc: RunCompleted => log.info("Run completed.")
case rs: RunStarting => fire(MessageEvent("Run starting"))
case rs: RunStopped => {succeeded = false;
fire(ErrorEvent("Run stopped"))}
case ra: RunAborted => {succeeded = false;
fire(ErrorEvent("Run aborted"))}
// this one seems to be working really well
case ts: TestStarting => fire(TypedEvent(ts.testName, "Test
Starting", None)(None))
// not sure what to do here at all
case tp: TestPending =>
case tf: TestFailed => {succeeded = false;
fire(TypedErrorEvent(tf.testName, "Test Failed", None,
tf.throwable)(Some(Result.Failed)))}
// this one also seems to be working really well
case ts: TestSucceeded => fire(TypedEvent(ts.testName, "Test
Succeeded", None)(Some(Result.Passed)))
// need to check if there is a reason why this test was ignored
case ti: TestIgnored => fire(IgnoredEvent(ti.testName,
Some("test ignored")))
case sc: SuiteCompleted => fire(TypedEvent(sc.suiteName,
"Suite Completed", None)(None))
// why not sure Some(Result.Failed) here?
// also, why not say succeeded = false?
// seems like i should do both if the suite is aborted.
case sa: SuiteAborted => fire(TypedErrorEvent(sa.suiteName,
"Suite Aborted", Some(sa.message), sa.throwable)(None))
case ss: SuiteStarting => fire(TypedEvent(ss.suiteName, "Suite
Starting", None)(None))
// not actually sure if this is what i should do here...info
provided is really just...some random extra info provided by a test,
like a log statement
case ip: InfoProvided => fire(MessageEvent(ip.message))
Bill Venners
Artima, Inc.
http://www.artima.com
Ben James
On 13 Oct, 14:16, Bill Venners <b...@artima.com> wrote:
> Hi Ben,
>
> Josh Cough created a class you can use to get it to work. We are still
> negotiating whether to add this class to ScalaTest or sbt, banging the
> dependency back and forth like playing ping pong.
>
Mark Harrah
On Tuesday 13 October 2009, Bill Venners wrote:
> Josh Cough created a class you can use to get it to work. We are still
> negotiating whether to add this class to ScalaTest or sbt, banging the
> dependency back and forth like playing ping pong.
We agree that It is not good for sbt or ScalaTest to have to depend on
the other. So, I propose creating a separate test framework
interface. The purpose is to make it easy for for tools to support
running tests from any of the three Scala testing frameworks. The
needs of sbt are I think basic and framework agnostic. It needs the
interface to provide the ability to:
- send output to $logger (more later)
- run test $classname
- return the result of the test
The interface would be pure Java so that calling a test framework
through this interface would be binary compatible across Scala
versions and test framework versions. Then, sbt running Scala 2.7.5
can run ScalaTest 1.0 compiled against 2.7.5 or 2.8.0-whatever or
ScalaTest 1.1 against 2.8.1.
To be more specific, sbt needs the interface to provide:
- the framework name, i.e. "ScalaTest", "specs", or "ScalaCheck"
- the type name that identifies a test ("org.scalatest.Suite",
org.specs.Specification, or "org.scalacheck.Properties") and whether a
test is a class (ScalaTest) or a module (specs, ScalaCheck)
- a test runner that
+ logs output through a basic Logger interface ( def error(msg:
String), def info(msg: String), ...)
+ runs a test with a given name using the provided ClassLoader and
reports success, failure, skipped, or error
+ perhaps extra options to pass to the framework as a Seq[String]
This is very much intended to be a lightweight, common view of the
frameworks. It would solve an important problem for sbt and I hope
would not be too much of a burden on framework authors. Each
framework would have to:
- introduce a compile-time dependency on a small Java jar defining a
simple interface to the frameworks (wouldn't be necessary at runtime
unless using the interface)
- implement the interface
A sample, minimal interface is attached based on some of sbt's current
code. (meant to be illustrative- I didn't actually compile it)
Thanks for reading!
-Mark
Bill Venners
That's a darn good idea. I think you just hit the ping pong ball my
way and I swung and I missed it. I'd be happy to add something to
ScalaTest that extends such a Java interface.
Bill
Bill Venners
And not only that, I think this sounds like a great approach in
general as a way for Scala libraries and frameworks to interact with
each other, to depend on each other, given the binary compatibility
issues with Scala. A very good idea.
Josh Cough
Bill Venners
Thanks. We can implement it now and get some experience with it with
the different testing frameworks. Once we're all pretty sure we've got
the Java interface right, we can include the ScalaTest one in the
subsequent ScalaTest release.
Bill
Mark Harrah
sbt's existing interface, so feel free to improve it.
If something like this gets implemented by the test frameworks, I think I can
finally have sbt separate the version of Scala doing the compiling from the
version running sbt.
As for Java interfaces as a general technique, it is indeed how I will be
interfacing between the different components of sbt running under different
versions of Scala.
I will take care of ScalaTest 1.0 support in sbt.
-Mark
etorreborre
I'll implement the interface for specs-1.6.1-SNAPSHOT.
Eric.
On Oct 14, 12:54 pm, Mark Harrah <dmhar...@gmail.com> wrote:
> Great! Thanks, guys. As I said, the interface is just a sample based on
> sbt's existing interface, so feel free to improve it.
>
> If something like this gets implemented by the test frameworks, I think I can
> finally have sbt separate the version of Scala doing the compiling from the
> version running sbt.
>
> As for Java interfaces as a general technique, it is indeed how I will be
> interfacing between the different components of sbt running under different
> versions of Scala.
>
> I will take care of ScalaTest 1.0 support in sbt.
>
> -Mark
>
> On Tuesday 13 October 2009, Bill Venners wrote:
>
>
>
> > Hi Josh,
>
> > Thanks. We can implement it now and get some experience with it with
> > the different testing frameworks. Once we're all pretty sure we've got
> > the Java interface right, we can include the ScalaTest one in the
> > subsequent ScalaTest release.
>
> > Bill
>
> > > I'll implement the interface for ScalaTest.
>
> > >> Hi Mark,
>
> > >> And not only that, I think this sounds like a great approach in
> > >> general as a way for Scala libraries and frameworks to interact with
> > >> each other, to depend on each other, given the binary compatibility
> > >> issues with Scala. A very good idea.
>
> > >> Bill
>
Mark Harrah
Rickard Nilsson
This is certainly a great idea. I'll implement it in ScalaCheck 1.7-SNAPSHOT.
How should we coordinate the work around the common interfaces, should
we put up a new
tiny project to handle it, where we all (Mark, Bill, Erik, Rickard)
can commit changes?
Or do you have any other suggestions?
Regards,
Rickard
Quoting Mark Harrah <dmha...@gmail.com>:
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google
> Groups "simple-build-tool" group.
> To post to this group, send email to simple-b...@googlegroups.com
> To unsubscribe from this group, send email to
> simple-build-t...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/simple-build-tool?hl=en
> -~----------~----~----~----~------~----~------~--~---
>
>
Rickard Nilsson
> Hi,
>
> This is certainly a great idea. I'll implement it in ScalaCheck 1.7-SNAPSHOT.
>
> How should we coordinate the work around the common interfaces, should
> we put up a new
> tiny project to handle it, where we all (Mark, Bill, Erik, Rickard)
> can commit changes?
> Or do you have any other suggestions?
Sorry Mark, I missed your mail about the github project. I'll use it
from scala-tools.org when it is up there.
/ Rickard
Mark Harrah
On 10/14/09, Rickard Nilsson <rick...@gmail.com> wrote:
> Quoting Rickard Nilsson <rick...@gmail.com>:
>
>> Hi,
>>
>> This is certainly a great idea. I'll implement it in ScalaCheck
>> 1.7-SNAPSHOT.
Great, thanks!
>> How should we coordinate the work around the common interfaces, should
>> we put up a new
>> tiny project to handle it, where we all (Mark, Bill, Erik, Rickard)
>> can commit changes?
>> Or do you have any other suggestions?
>
> Sorry Mark, I missed your mail about the github project. I'll use it
> from scala-tools.org when it is up there.
I have no problem if someone else wants to host it or wants to do it
differently. I just wanted to get it moving. I have a first draft
that I am ready to push, but there are two naming questions. What
should the package name be and what should the artifact name be? I'm
working with 'sbt' as a package name and
org.scala-tools-testing#test-interface as organization/artifact IDs.
Thanks,
Mark
Mark Harrah
http://github.com/harrah/test-interface/
Take a look and let me know what you'd like to see done differently.
Thanks!
-Mark
Josh Cough
Josh Cough
public Result run(String testClassName, TestFingerprint fingerprint, Param[] params);
A user might want to communicate some restraints on which test they want to run. For example, they might want to only run tests in a particular group: Slow Tests. They could pass the groups in with the params. There might be a better way to do this, but at least this would make it possible, as it seems impossible (or quite difficult) given the current API.
I think the problem is actually a lot deeper than that shallow example. A general way to tap into, or control discovery would be very helpful, but I need to think it through a little more. I'll try to write more tonight.
Hopefully I'm not missing something obvious! Thanks
-Josh
Mark Harrah
On 10/16/09, Josh Cough <josh...@gmail.com> wrote:
> Hi guys. I'm having some problems with the API, but I think they are hard to
> see at first. One of the problems, I think, would be solved by changing the
> signature of the run method to:
>
> public Result run(String testClassName, TestFingerprint fingerprint,
> Param[] params);
>
>
> A user might want to communicate some restraints on which test they want to
> run. For example, they might want to only run tests in a particular group:
> Slow Tests. They could pass the groups in with the params. There might be a
> better way to do this, but at least this would make it possible, as it seems
> impossible (or quite difficult) given the current API.
testClassName is the name of the test class to run, not individual
properties/examples/whatever each test framework calls them. From my
point of view, sbt has already determined what are test classes or
modules and is asking the framework to run a specific test. It will
call 'run' for each test to run.
I think parameters could be useful in general though. Definitely
don't take this as a final API, just a draft.
> I think the problem is actually a lot deeper than that shallow example. A
> general way to tap into, or control discovery would be very helpful, but I
> need to think it through a little more. I'll try to write more tonight.
The API is meant to be a simple "run this test please" that leaves the
details of actually running the test to the framework. The client of
the framework is expected to handle providing the tests to run.
Implementing it should be similar to the ScalaTestRunner for sbt.
-Mark
Bill Venners
On Fri, Oct 16, 2009 at 4:49 PM, Mark Harrah <dmha...@gmail.com> wrote:
>
> Hi Josh,
>
> On 10/16/09, Josh Cough <josh...@gmail.com> wrote:
>> Hi guys. I'm having some problems with the API, but I think they are hard to
>> see at first. One of the problems, I think, would be solved by changing the
>> signature of the run method to:
>>
>> public Result run(String testClassName, TestFingerprint fingerprint,
>> Param[] params);
>>
>>
>> A user might want to communicate some restraints on which test they want to
>> run. For example, they might want to only run tests in a particular group:
>> Slow Tests. They could pass the groups in with the params. There might be a
>> better way to do this, but at least this would make it possible, as it seems
>> impossible (or quite difficult) given the current API.
>
> testClassName is the name of the test class to run, not individual
> properties/examples/whatever each test framework calls them. From my
> point of view, sbt has already determined what are test classes or
> modules and is asking the framework to run a specific test. It will
> call 'run' for each test to run.
>
When you say "run a specific test," do you mean "run a specific test
class"? Sounds like you are conflating the two:
/* Specifies how to identify classes that are tests.*/
public TestFingerprint[] tests();
One thing I tried very hard to do in ScalaTest was be clear on
terminology. A "test" in ScalaTest is anything that 1. has a name, 2.
can be started, and 3. either succeed, fail, or be pending. A "suite"
is a collection of tests. I think vague and conflicting usage of terms
makes an API harder to learn and use.
What "test" means is very confused, for example, in the JUnit API. A
test method is a test in JUnit. A collection of tests in a class is a
TestCase in JUnit 3 at least. A TestSuite is a collection of
TestCases, and both of those implement the Test interface. And the
convention is to name TestCase classes with names ending in Test. So
"test" in JUnit kind of means everything and therefore nothing.
You need not adopt ScalaTest's terminology, but if not, I'd recommend
you at least always say "test class" whenever you talk about that
concept. Because the main thing a test is in any of these frameworks,
JUnit, TestNG, ScalaTest, specs (so long as you substitute "test"
every time you see "example") is that named thing that can start and
either succeed or fail.
The way ScalaTest's terminology works is a suite of tests can be any
collection of tests. A "test class" can be a suite of the tests inside
it, be they test methods or test functions or something else. A class
whose instances can hold onto other classes that represent suites of
tests is also a suite because it is a collection of all those tests. A
"Spec" is a suite of tests that has the BDD style of input and output.
So in your API if you want to be clear you are talking just about
classes, I would call them "suite classes" in ScalaTest terminology.
If you'd rather use the word test, then I'd suggest calling them "test
classes" not just "tests."
Thanks.
Bill
----
Mark Harrah
Yes, I mean suite in ScalaTest terminology. We can go with suite,
since "suites" can be classes or modules and looking through the
documentation on the interface so far, it might be awkward sometimes
to refer to them as just test classes.
-Mark
Josh Cough
etorreborre
> I'd like to hear more ideas. I don't even know what a Param might contain at
> this point. Additonally, I'm also not sure what the best way for sbt to
thing we need since there's going to be a command-line interface
sending an Array[String] for the main function of every test framework
out there?
Eric.
On Oct 19, 5:50 am, Josh Cough <joshco...@gmail.com> wrote:
> I thought about this a bunch more over the weekend, and I think the
> discovery being handled by sbt is just fine. All we are doing is in
> ScalaTest is finding subclasses of Suite, and thats the same thing you'll be
> doing with TestFingerprint.
> But, the Params that I suggested is absolutely needed. You said:
>
> >The API is meant to be a simple "run this test please" that leaves the
> >details of actually running the test to the framework.
>
> tests should be run, or *which *tests to run inside a particular Suite. Here
Mark Harrah
>
>> I'm not sure that adding Params[] params to the run method is the best,
>> and
>> I'd like to hear more ideas. I don't even know what a Param might contain
>> at
>> this point. Additonally, I'm also not sure what the best way for sbt to
>
> Maybe a List[String] containing command-line parameters is the only
> thing we need since there's going to be a command-line interface
> sending an Array[String] for the main function of every test framework
> out there?
The parameters needed are extra options specific to the framework.
The test class to run is already identified and whether it is a class
or module. The runner needs to use the provided ClassLoader and
Logger. The run method will be called once for each test
class/module. It won't have all names passed at once. So, the extra
options might be similar to those passed to the main method, but the
interface can't be the same as calling the main method.
-Mark
Josh Cough
Mark, how does this effect things like test-only? You'd normally have something like this right?
sbt> test-only SomeClass AnotherClass YetAnotherClass
Now you'll have to recognize that the things on the end might not be classes.
sbt> test-only SomeClass AnotherClass YetAnotherClass -t things that aren't a class.
Finally, since these interfaces can likely be used by other tools, maybe we should change the package name? How about org.scalatools.testing ?
-Josh
Mark Harrah
> But, the Params that I suggested is absolutely needed. You said:
>
>>The API is meant to be a simple "run this test please" that leaves the
>>details of actually running the test to the framework.
>
> tests should be run, or *which *tests to run inside a particular Suite. Here
Right, I agree that how sbt gets the arguments from the user is the
tool's problem.
> A final note about discovery - we still might want to make discovery a part
> of the interfaces, or even have an implemented version as part of this
> project. I imagine other tools could make use of it as everyone has to do
> discovery. But, that is separate to the everything above.
The TestFingerprint interface could be more general than subclass
detection. I don't know what that would look like, though. If it
were made more general, I would wonder if your NetLogo tests should be
run by a custom framework.
-Mark
Mark Harrah
> able to reuse the code we have now (for scala test its
> org.scalatest.tools.Runner). This is exactly the way intellij does it as
> well. I'll proceed unless I hear otherwise.
I think String[] is fine for extra options, but it can't be the same
as just using the framework's main method. The runner needs to use
the provided ClassLoader and Logger, for example.
> Mark, how does this effect things like test-only? You'd normally have
> something like this right?
>
> sbt> test-only SomeClass AnotherClass YetAnotherClass
>
> Now you'll have to recognize that the things on the end might not be
> classes.
>
> sbt> test-only SomeClass AnotherClass YetAnotherClass -t things that aren't
> a class.
I don't think this would be a problem. Class names aren't unescaped
from things like $plus to +, so we can do something like separating
names from -- or some other syntax. I'll worry the details after
we've got an interface.
> Finally, since these interfaces can likely be used by other tools, maybe we
> should change the package name? How about org.scalatools.testing ?
Sure. Please feel free to modify the interface and post patches (or
pull requests or whatever).
-Mark
Josh Cough
>detection. I don't know what that would look like, though. If it
>were made more general, I would wonder if your NetLogo tests should be
>run by a custom framework.
But about the last sentence there, I'm not sure if I follow. Did you have an idea in mind that you can explain?
Josh Cough
>the provided ClassLoader and Logger, for example.
Understood. But, we should still be able to reuse a bunch of code.
>from things like $plus to +, so we can do something like separating
>names from -- or some other syntax. I'll worry the details after
>we've got an interface.
ok.
>pull requests or whatever).
Tom Adams
> >detection. I don't know what that would look like, though. If it
> >were made more general, I would wonder if your NetLogo tests should
> be
> >run by a custom framework.
>
> I'm also not sure what a more general TestFingerprint would look
> like, but I'll give it some thought.
> But about the last sentence there, I'm not sure if I follow. Did you
> have an idea in mind that you can explain?
I've not been following the discussion in detail, but here's my 2c.
Instinct [1] a Java BDD framework I wrote a while back has a bunch of
ways to detect what to run. It doesn't actually use subclasses, but
naming conventions, annotations, etc. I've been meaning to write an
sbt integration point to it for a while, so whatever gets added I'd be
happy to provide an implementation of this interface for Instinct (it
already has exactly this).
Tom
[1] http://code.google.com/p/instinct/
--
tom adams | e:tomjadams<at>gmail.com
Josh Cough
public interface Framework{public String name();
public boolean isSuite(Class c);
public Runner testRunner(ClassLoader testClassLoader, Logger logger);}
The TestFingerprint assumes that the framework can determine what is a suite without ever examining a class. This changes that, abandoning the first idea altogether. Maybe there is a nice middleground?
Josh Cough
Josh Cough
Each suite has N tests. Result is only Success, Error, Failure, Skipped. It needs to be an array of Results, but even that is not enough, because you need to know the names of the tests corresponding to the result. Here is a proposal:
interface TestResult {
public enum Status{ Success, Error, Failure, Skipped }
Status status();
String testName();
}
interface Runner {
...
public TestResult [] run(String suiteClassName, TestFingerprint fingerprint, String [] args)
...
}
I'm certainly open to other ideas, but this one at least seems to have the necessary information.
Also, Based on Bill's input, should I go ahead and rename TestFingerprint to SuiteFingerprint, and test to suite elsewhere (as I did above)? I used TestResult because in that case it does actually represent a single test.
-Josh
Josh Cough
I found yet another issue with Runner. It would be nice if tools had immediate info of when test is starting, is finished, and what its result is, as well as an entire suites. I suppose this issue could be resolved with the Logger, but thats probably not the best way. Waiting until the end of a Suite to report the results is not a good option. A Suite could have thousands of tests, and several long running tests. We have eventing build into ScalaTest, and my current ScalaTest/sbt integration code is also firing sbt events. Events are missing from this api, and they should probably be there. If we did have it, we could get get rid of the TestResult [] returned from the run method if we wanted, or we could just keep it. My first shot at this API looks like this:
interface EventType{
String description();
}
public interface TestEvent {
EventType eventType();
String testName();
}
interface TestEventListener{
void eventFired(TestEvent event)
}
Runner...
public TestResult [] run(String suiteClassName, TestFingerprint fingerprint, TestEventListener eventListener String [] args)
or
public void run(String suiteClassName, TestFingerprint fingerprint, TestEventListener eventListener String [] args)
I'm sure I'm missing something here, because if the TestEvent is a "test finished event", then we'll need the TestResult to go along with it. EventType isnt an enum because I'm not confident I can think of all possible Test events. I could only list those that we have in ScalaTest. Anyway, please help make this API better.
-Josh
etorreborre
I think that we may go through a too complicated road here, because
frameworks may have very different way to run and report results. For
example in specs you can nest examples and subexamples indefinitely
and I don't know how to use the current interface with that.
What would be simpler is:
Framework.java:
/** return a Runner using the ResultOutput to display
- descriptions
- test results
- run statistics
*/
public Runner testRunner(ClassLoader testClassLoader, Logger logger,
ResultOutput output);
ResultOutput.java
/** write to the console */
public void write(message: String)
Runner.java
/** return true if the tests are not passing for whatever reason */
public boolean run(String testClassName, TestFingerprint fingerprint,
String [] args);
Do you see any issue with this approach?
Eric.
On Oct 19, 3:03 pm, Josh Cough <joshco...@gmail.com> wrote:
> Sorry for so many mails, maybe I should have put this all in one mail, but I
> just keep thinking of new issues.
>
> I found yet another issue with Runner. It would be nice if tools had
> immediate info of when test is starting, is finished, and what its result
> is, as well as an entire suites. I suppose this issue could be resolved with
> the Logger, but thats probably not the best way. Waiting until the end of a
> Suite to report the results is not a good option. A Suite could have
> thousands of tests, and several long running tests. We have eventing build
> into ScalaTest, and my current ScalaTest/sbt integration code is also firing
> sbt events. Events are missing from this api, and they should probably be
> there. If we did have it, we could get get rid of the TestResult [] returned
> from the run method if we wanted, or we could just keep it. My first shot at
> this API looks like this:
>
> interface EventType{
> String description();
> }
>
> public interface TestEvent {
> EventType eventType();
> String testName();
>
> }
>
> interface TestEventListener{
> void eventFired(TestEvent event)
>
> }
>
> Runner...
> or
>
> I'm sure I'm missing something here, because if the TestEvent is a "test
> finished event", then we'll need the TestResult to go along with it.
> EventType isnt an enum because I'm not confident I can think of all possible
> Test events. I could only list those that we have in ScalaTest. Anyway,
> please help make this API better.
>
> -Josh
>
>
>
> On Sun, Oct 18, 2009 at 11:41 PM, Josh Cough <joshco...@gmail.com> wrote:
> > I found a significant problem with Runner.run: public Result run(String
> > suiteClassName, TestFingerprint fingerprint, String [] args)
>
> > Each suite has N tests. Result is only Success, Error, Failure, Skipped. It
> > needs to be an array of Results, but even that is not enough, because you
> > need to know the names of the tests corresponding to the result. Here is a
> > proposal:
>
> > interface TestResult {
> > public enum Status{ Success, Error, Failure, Skipped }
> > Status status();
> > String testName();
> > }
>
> > interface Runner {
> > ...
> > ...
> > }
>
> > I'm certainly open to other ideas, but this one at least seems to have the
> > necessary information.
>
> > Also, Based on Bill's input, should I go ahead and rename TestFingerprint
> > to SuiteFingerprint, and test to suite elsewhere (as I did above)? I used
> > TestResult because in that case it does actually represent a single test.
>
> > -Josh
>
>
> >> I pushed the Runner.run signature change, and moved the interfaces to the
> >> org.scalatools.testing package.
>
>
> >>> Thanks Tom. That makes sense. In that case we might need something in the
> >>> interface that takes a class and returns true if it is a suite. Would we
> >>> even need TestFingerprint if we did that? What if we did something like
> >>> this:
>
> >>> public interface Framework{
> >>> public String name();
>
> >>> public boolean isSuite(Class c);
> >>> public Runner testRunner(ClassLoader testClassLoader, Logger logger);
>
> >>> }
>
> >>> The TestFingerprint assumes that the framework can determine what is a
> >>> suite without ever examining a class. This changes that, abandoning the
> >>> first idea altogether. Maybe there is a nice middleground?
>
Mark Harrah
On 10/19/09, etorreborre <etorr...@gmail.com> wrote:
> I think that we may go through a too complicated road here, because
> frameworks may have very different way to run and report results. For
> example in specs you can nest examples and subexamples indefinitely
> and I don't know how to use the current interface with that.
>
> What would be simpler is:
>
> Framework.java:
>
> /** return a Runner using the ResultOutput to display
> - descriptions
> - test results
> - run statistics
> */
> public Runner testRunner(ClassLoader testClassLoader, Logger logger,
> ResultOutput output);
>
> ResultOutput.java
>
> /** write to the console */
> public void write(message: String)
>
> Runner.java
>
> /** return true if the tests are not passing for whatever reason */
> public boolean run(String testClassName, TestFingerprint fingerprint,
> String [] args);
>
> Do you see any issue with this approach?
The Logger is supposed to be used for the purpose of the ResultOutput.
If you would prefer ResultOutput, that is fine with me. The run
method above is basically all sbt needs. It just needs to know on a
test suite level whether the suite passed or failed so that it can
implement test-quick and test-failed. However, it would be useful as
a user to have the number of tests passed/failing/whatever and a
summary of failures be uniformly reported. I think that the rest of
the logging/output can be framework specific. So, I like returning a
TestResult[] as Josh suggests.
As Eric indicates, the problem with the event listener is that each
framework will have a different event hierarchy and events complicate
the interface. There could be a basic TestEvent interface and each
framework could implement its own hierarchy, but it should be
information that a client is not required to handle. That is, run
should still return TestResult[].
> On Oct 19, 3:03 pm, Josh Cough <joshco...@gmail.com> wrote:
>> I found yet another issue with Runner. It would be nice if tools had
>> immediate info of when test is starting, is finished, and what its result
>> is, as well as an entire suites. I suppose this issue could be resolved
>> with
>> the Logger, but thats probably not the best way.
I don't see a problem with this. I think it is the user that needs
the immediate feedback, not the tool. The framework can log to the
Logger as it runs and inform the tool of results after the suite
completes, .
-Mark
Mark Harrah
> interface that takes a class and returns true if it is a suite. Would we
> even need TestFingerprint if we did that? What if we did something like
> this:
>
> public interface Framework{
> public String name();
> public boolean isSuite(Class c);
> public Runner testRunner(ClassLoader testClassLoader, Logger
> logger);}
>
>
> The TestFingerprint assumes that the framework can determine what is a suite
> without ever examining a class. This changes that, abandoning the first idea
> altogether. Maybe there is a nice middleground?
a test suite?
The previous way was for the Framework to tell the Framework client
(such as sbt) what constitutes a test suite. ScalaTest would tell sbt
that subclasses of Suite are tests. sbt then does the actual
discovery of tests. I don't necessarily think that detecting
subclasses needs to be the only method of discovery, but it is the
most efficient way I know of.
> Also, Based on Bill's input, should I go ahead and rename TestFingerprint to
> SuiteFingerprint, and test to suite elsewhere (as I did above)? I used
> TestResult because in that case it does actually represent a single test.
-Mark
Josh Cough
>> On Oct 19, 3:03 pm, Josh Cough <joshco...@gmail.com> wrote:
>>> immediate info of when test is starting, is finished, and what its result
>>> is, as well as an entire suites. I suppose this issue could be resolved
>>> with the Logger, but thats probably not the best way.
>the immediate feedback, not the tool. The framework can log to the
>Logger as it runs and inform the tool of results after the suite
>completes, .
Now, lets also imagine that you have a Suite with 100k tests, and they take an hour to run. Logging would be helpful, but if you're in an IDE and you cant see that output, then well, this just doesn't work for you. You have to wait around for an hour.
Maybe its a pipedream that an IDE would use this, but its a pipedream I've had for years:
http://jackcoughonsoftware.blogspot.com/2007/12/unit-testing-in-scala-and-in-ides-in.html
Mark Harrah
than just sbt, we should involve the other tools and take this
discussion to a common mailing list, perhaps scala-tools or a new
group. Do you want to take care of this?
-Mark