Proposed enhancements to sbt's Framework API
Bill Venners
I wanted to get feedback on a proposed update to sbt's Framework API.
We developed this in concert with Mark Harrah as part of the ScalaTest
2.0 effort, because many of the deficiencies in ScalaTest's sbt
support were caused by deficiencies in the Framework API. It also
incorporates new directions that Mark wanted to take it, with the
framework returning tasks that can be tagged. The current state of the
code is here:
https://github.com/bvenners/test-interface/tree/master/src/org/scalasbt/testing
It is pretty thoroughly documented with Javadoc comments, but I'll
walk through some of the main ideas here:
1. The package is changed from org.scalatools.testing to
org.scalasbt.testing for two reasons. One is because scala-tools.org
is no longer a community resource. The other is so that the same class
can implement the new Framework interface as well as the old. That way
a test framework can support both the old and new Framework API in one
class, sbt can always look for that same class, and different versions
of sbt (old and new) will work with a version of the test framework
that supports both. Essentially sbt 0.12 and later will look to see if
the class implements the new Framework API, and if so, it would use
that. Else it would see if it implements the old Framework API, and if
so, would use that.
2. Framework.java is the starting point. sbt will call runner to get a
runner for a "run." One problem with the current version of Framework
that gave ScalaTest users trouble is that it has no notion of a run
starting and ending. That made it impossible for sbt users to access a
lot of nice reporters that ScalaTest has available through any other
means of using ScalaTest. In the new Framework API, a run starts when
sbt calls runner, and it ends when sbt calls done on that runner
(actually, it ends when the test framework returns from done.)
3. There is an addition of one more fingerprint, a DoNotDiscover
fingerprint. In ScalaTest we added an annotation that instructs
ScalaTest not to discover a test class. It's more efficient if sbt can
be told to do the same.
4. Runner has a way for sbt to ask it for a task to run a test class
it has discovered via a fingerprint, the task method with this
signature:
public Task task(String fullyQualifiedName, Fingerprint fingerprint);
It also has a way for sbt to ask for a task to rerun something that it
found out about through an event. One use case for this is to allow
sbt to rerun failed tests. That's this one:
public Task task(String fullyQualifiedName, boolean isModule,
Selector[] selectors);
5. The same four kinds of events can be fired as the original
Framework, which represent statuses of Success, Failure, Error, and
Skipped. But this has been refactored into an Event hierarchy, so that
only the data relevant to each event shows up in the event. One of the
kinds of objects that can show up in there is called a "selector,"
which is what sbt can pass to the second task method to rerun things.
6. There are four kinds of selectors. ScalaTest needs all four, but I
would expect the other frameworks probably only need SuiteSelector and
TestSelector. Suite here just means a collection of tests, usually
contained in a test class, but in ScalaTest this is more abstract and
a suite need not correspond to a class necessarily. In the other test
frameworks a suite will likely always correspond to a class. (Suite is
the collective noun for tests. It's gaggle of geese, school of fish,
suite of tests, etc.)
As an example, if a test class had 25 tests, 3 of which failed and the
rest of which succeeded, then sbt would receive 3 FailureEvents. The
fullyQualifiedName would be the same in those 3 events, but the
selector would be different. To rerun those, sbt would call task once,
passing in the common fullyQualifiedName and the three TestSelectors
in an array. sbt would get one task back that when executed, would (in
ScalaTest at least) just rerun those three tests in one instance of
the test class. If this is not easy in other test frameworks, they
could simply create three instances and run one test in each instance.
7. Tasks can be tagged. This allows sbt to control how to execute
things via tags. Mark has a page about it. Look for tagging tasks on
this page:
https://github.com/harrah/xsbt/wiki/Parallel-Execution
That's a quick overview. Please post questions to this list and I'll
try and answer as promptly as being on "vacation" with spotty internet
access allows.
Thanks.
Bill
----
Bill Venners
Artima, Inc.
http://www.artima.com
etorreborre
etorreborre
Stefan Zeiger
> I wanted to get feedback on a proposed update to sbt's Framework API.
I'll try to implement that after ScalaDays when I have some more time
for side projects.
> 2. Framework.java is the starting point. sbt will call runner to get a
> runner for a "run." One problem with the current version of Framework
> that gave ScalaTest users trouble is that it has no notion of a run
> starting and ending. That made it impossible for sbt users to access a
> lot of nice reporters that ScalaTest has available through any other
> means of using ScalaTest.
Same for JUnit. Every test class looks like a test run to JUnit at the
moment.
> It also has a way for sbt to ask for a task to rerun something that it
> found out about through an event. One use case for this is to allow
> sbt to rerun failed tests. That's this one:
>
> public Task task(String fullyQualifiedName, boolean isModule,
> Selector[] selectors);
Where do the selectors come from? Who constructs them?
> 6. There are four kinds of selectors. ScalaTest needs all four, but I
> would expect the other frameworks probably only need SuiteSelector and
> TestSelector. Suite here just means a collection of tests, usually
> contained in a test class, but in ScalaTest this is more abstract and
> a suite need not correspond to a class necessarily. In the other test
> frameworks a suite will likely always correspond to a class. (Suite is
> the collective noun for tests. It's gaggle of geese, school of fish,
> suite of tests, etc.)
In JUnit, a suite refers to a collection of several test classes. I
don't think any ordering or dependency is implied, so it should be
possible to just collect all classes in a suite and run them. That would
allow you to run a specific suite with the test-only command. At the
moment, JUnit suites are completely ignored because - without the
concept of a test run - they would cause tests to run multiple times
when they are discovered both individually and as part of a suite.
> As an example, if a test class had 25 tests, 3 of which failed and the
> rest of which succeeded, then sbt would receive 3 FailureEvents. The
> fullyQualifiedName would be the same in those 3 events, but the
> selector would be different. To rerun those, sbt would call task once,
> passing in the common fullyQualifiedName and the three TestSelectors
> in an array. sbt would get one task back that when executed, would (in
> ScalaTest at least) just rerun those three tests in one instance of
> the test class. If this is not easy in other test frameworks, they
> could simply create three instances and run one test in each instance.
I'll have to chek how that works with parameterized tests. I can
construct a Request (to run with JUnit) for a single test method instead
of a test class but I don't see any obvious way of running a specific
parameterized instance of a test method. What is a test framework
supposed to do if a specific fragment of a test run can be identified
but not launched individually? Only report fragments at the granularity
at which they can be run?
Cheers,
Stefan
Bill Venners
Sorry I didn't notice your reply until today.
On Tue, Apr 10, 2012 at 6:35 AM, Stefan Zeiger <sze...@novocode.com> wrote:
> On 2012-03-22 3:10, Bill Venners wrote:
>>
>> I wanted to get feedback on a proposed update to sbt's Framework API.
>
>
> I'll try to implement that after ScalaDays when I have some more time for
> side projects.
>
If you'll be at ScalaDays, we should sync up there about this.
>
>> 2. Framework.java is the starting point. sbt will call runner to get a
>> runner for a "run." One problem with the current version of Framework
>> that gave ScalaTest users trouble is that it has no notion of a run
>> starting and ending. That made it impossible for sbt users to access a
>> lot of nice reporters that ScalaTest has available through any other
>> means of using ScalaTest.
>
>
> Same for JUnit. Every test class looks like a test run to JUnit at the
> moment.
>
>
>> It also has a way for sbt to ask for a task to rerun something that it
>> found out about through an event. One use case for this is to allow
>> sbt to rerun failed tests. That's this one:
>>
>> public Task task(String fullyQualifiedName, boolean isModule,
>> Selector[] selectors);
>
>
> Where do the selectors come from? Who constructs them?
>
The selectors are fired in the events. So the test framework
constructs them when it fires the events.
>
>> 6. There are four kinds of selectors. ScalaTest needs all four, but I
>> would expect the other frameworks probably only need SuiteSelector and
>> TestSelector. Suite here just means a collection of tests, usually
>> contained in a test class, but in ScalaTest this is more abstract and
>> a suite need not correspond to a class necessarily. In the other test
>> frameworks a suite will likely always correspond to a class. (Suite is
>> the collective noun for tests. It's gaggle of geese, school of fish,
>> suite of tests, etc.)
>
>
> In JUnit, a suite refers to a collection of several test classes. I don't
> think any ordering or dependency is implied, so it should be possible to
> just collect all classes in a suite and run them. That would allow you to
> run a specific suite with the test-only command. At the moment, JUnit suites
> are completely ignored because - without the concept of a test run - they
> would cause tests to run multiple times when they are discovered both
> individually and as part of a suite.
>
Not sure I follow this. A good topic for ScalaDays.
>
>> As an example, if a test class had 25 tests, 3 of which failed and the
>> rest of which succeeded, then sbt would receive 3 FailureEvents. The
>> fullyQualifiedName would be the same in those 3 events, but the
>> selector would be different. To rerun those, sbt would call task once,
>> passing in the common fullyQualifiedName and the three TestSelectors
>> in an array. sbt would get one task back that when executed, would (in
>> ScalaTest at least) just rerun those three tests in one instance of
>> the test class. If this is not easy in other test frameworks, they
>> could simply create three instances and run one test in each instance.
>
>
> I'll have to chek how that works with parameterized tests. I can construct a
> Request (to run with JUnit) for a single test method instead of a test class
> but I don't see any obvious way of running a specific parameterized instance
> of a test method. What is a test framework supposed to do if a specific
> fragment of a test run can be identified but not launched individually? Only
> report fragments at the granularity at which they can be run?
>
Yes, if I understand your question correctly. If you can't rerun just
one test, you'd need to provide a selector for the next larger entity
that you can rerun.
Bill
> Cheers,
> Stefan
--
etorreborre
Bill Venners
Sorry, I don't get email from mailing lists so I didn't see this until
Mark told me it was here. I'll take a look at this today when I sit
down with Stephan and let you know.
Bill
Stefan Zeiger
>
> I wanted to get feedback on a proposed update to sbt's Framework API.
> We developed this in concert with Mark Harrah as part of the ScalaTest
> 2.0 effort, because many of the deficiencies in ScalaTest's sbt
> support were caused by deficiencies in the Framework API. It also
> incorporates new directions that Mark wanted to take it, with the
> framework returning tasks that can be tagged. The current state of the
> code is here:
I finally tried to implement this for JUnit today and ran into a few
issues. All of them should be solvable with the current state of the
test interface but I think it could be nicer, so here are some notes
(along with code, where applicable):
Having lots of classes with the same names in different packages (and
always wanting to use both at the same time) is quite annoying in Java
because you cannot rename imported names. OTOH, inventing new names or
arbitrary prefixes for everything is probably worse.
There are concrete classes in the new version (for events and
selectors). This makes it more difficult to fit your own implementation
into this system and properly reuse code, especially in Java where you
don't have traits or self-types. I'd prefer to have only interfaces and
enumerations in the API. Sensible default implementations could still be
provided but they should never be referenced by implementers (unless, of
course, you're creating an instance of such a class yourself). Here's
the "purified" API without classes:
https://github.com/szeiger/test-interface/commit/0445e29492ff1d5638392284b9bfa47abc65198a
Adding some adapters in order to take some of the boilerplate out of
writing a test framework which targets both versions of the test
interface:
https://github.com/szeiger/test-interface/commit/4a71b23e3f88288108b2cbabe01bbf80131f82bb
and
https://github.com/szeiger/test-interface/commit/532ef76d8d9f84e40d817e319fb9a3f1d21ba3c7
Give events and selectors a "display name" (required for the adapters):
https://github.com/szeiger/test-interface/commit/397e7ebc663d4f7c8a6348d08eeb6a49b3f5d602
RunStatus seems to be unused:
https://github.com/szeiger/test-interface/commit/b809e44dc588e2ccaf56a661ac9df52ec8deaabb
Who benefits from the strict one-or-two-level hierarchy for selectors?
Test frameworks shouldn't care. Are there any test runners which make
(or should make) use of this? We could simplify the API quite a bit by
making selectors fully hierarchical. No top-level
fullyQualifiedClassName, no isModule, no suite IDs or test names, no
Selector subclasses, just a Selector with a SelectorType (class, object,
method), a display name, a parent Selector and guaranteed equality
semantics. Mapping back from a Selector to some test artifact that can
be run is the task of the test framework which created the Selector, so
it can put whatever extra data it needs for that job into the Selector.
Here we go:
https://github.com/szeiger/test-interface/commit/89815947ecaf7625a87b73e54486f9d1fd53387e
That still doesn't give me a way to integrate JUnit test suites into the
interface but I wouldn't know what extra functionality to implement for
them anyway. Thanks to the new batch-oriented runner, I can discover
test suites (with fingerprints) and run the associated tests. This was
not possible with the old test-interface because it would have run tests
multiple times (as part of each suite that contains them in their
transitive closure, plus as individual tests). Now I can just run a test
the first time I encounter it and then mark it so it won't run again in
the same batch.
JUnit test suites are not hierarchical so they don't map to
test-interface's nested suites. OTOH, parameterized tests in JUnit could
be viewed as nested suites. It may not map cleanly to the
one-or-two-level system though. AFAICT there is no way to run a test
class or a single test method for a specific parameter set, so the
parameterization needs to go last in the hierarchy (even though the
parameters affect the test class and not the test method). As a matter
of fact, that's how JUnit and also my junit-interface print
parameterized test names:
package.and.class.Name.methodName[paremeter-index]. But the two-level
hierarchy expects the method name to go last. Actually, my recursive
hierarchy did that, too, so here's one more change to fix it:
https://github.com/szeiger/test-interface/commit/0606409a11541efaec417a471a40005280be5a8d
Cheers,
Stefan
Mark Harrah
Stefan Zeiger <szei...@googlemail.com> wrote:
> On 2012-03-22 3:10, Bill Venners wrote:
> > Hi All,
> >
> > I wanted to get feedback on a proposed update to sbt's Framework API.
> > We developed this in concert with Mark Harrah as part of the ScalaTest
> > 2.0 effort, because many of the deficiencies in ScalaTest's sbt
> > support were caused by deficiencies in the Framework API. It also
> > incorporates new directions that Mark wanted to take it, with the
> > framework returning tasks that can be tagged. The current state of the
> > code is here:
>
> I finally tried to implement this for JUnit today and ran into a few
> issues. All of them should be solvable with the current state of the
> test interface but I think it could be nicer, so here are some notes
> (along with code, where applicable):
Thanks for the feedback...
> Having lots of classes with the same names in different packages (and
> always wanting to use both at the same time) is quite annoying in Java
> because you cannot rename imported names. OTOH, inventing new names or
> arbitrary prefixes for everything is probably worse.
You are referring to having both the old and new interface included?
> There are concrete classes in the new version (for events and
> selectors). This makes it more difficult to fit your own implementation
> into this system and properly reuse code, especially in Java where you
> don't have traits or self-types. I'd prefer to have only interfaces and
> enumerations in the API. Sensible default implementations could still be
> provided but they should never be referenced by implementers (unless, of
> course, you're creating an instance of such a class yourself). Here's
> the "purified" API without classes:
> https://github.com/szeiger/test-interface/commit/0445e29492ff1d5638392284b9bfa47abc65198a
I tend to prefer interfaces as well. I forget if there was a good reason for concrete classes. I'll let Bill comment here.
> Adding some adapters in order to take some of the boilerplate out of
> writing a test framework which targets both versions of the test
> interface:
> https://github.com/szeiger/test-interface/commit/4a71b23e3f88288108b2cbabe01bbf80131f82bb
> and
> https://github.com/szeiger/test-interface/commit/532ef76d8d9f84e40d817e319fb9a3f1d21ba3c7
Seems useful to me...
> Give events and selectors a "display name" (required for the adapters):
> https://github.com/szeiger/test-interface/commit/397e7ebc663d4f7c8a6348d08eeb6a49b3f5d602
>
> RunStatus seems to be unused:
> https://github.com/szeiger/test-interface/commit/b809e44dc588e2ccaf56a661ac9df52ec8deaabb
>
> Who benefits from the strict one-or-two-level hierarchy for selectors?
> Test frameworks shouldn't care. Are there any test runners which make
> (or should make) use of this? We could simplify the API quite a bit by
> making selectors fully hierarchical. No top-level
> fullyQualifiedClassName, no isModule, no suite IDs or test names, no
> Selector subclasses, just a Selector with a SelectorType (class, object,
> method), a display name, a parent Selector and guaranteed equality
> semantics. Mapping back from a Selector to some test artifact that can
> be run is the task of the test framework which created the Selector, so
> it can put whatever extra data it needs for that job into the Selector.
> Here we go:
> https://github.com/szeiger/test-interface/commit/89815947ecaf7625a87b73e54486f9d1fd53387e
The main problem here is that this information needs to be serialized in some way. Think 'test-only selector-string'. As far as sbt is concerned, all of a Selector's structure needs to be exposed. Equality/hash code isn't that useful. sbt can't keep the Selector around or there will be class loader leaks. All types from a test framework need to be discarded at the end of task execution for this reason. Forking tests (in 0.12 or later) imposes similar constraints. So, any information to be retained, like Selector, needs to be in a standard form and preferably some subset may be referenced by a string by the user.
I'm ok with whatever representation gives me this information.
> That still doesn't give me a way to integrate JUnit test suites into the
> interface but I wouldn't know what extra functionality to implement for
> them anyway. Thanks to the new batch-oriented runner, I can discover
> test suites (with fingerprints) and run the associated tests. This was
> not possible with the old test-interface because it would have run tests
> multiple times (as part of each suite that contains them in their
> transitive closure, plus as individual tests). Now I can just run a test
> the first time I encounter it and then mark it so it won't run again in
> the same batch.
>
> JUnit test suites are not hierarchical so they don't map to
> test-interface's nested suites. OTOH, parameterized tests in JUnit could
> be viewed as nested suites. It may not map cleanly to the
> one-or-two-level system though. AFAICT there is no way to run a test
> class or a single test method for a specific parameter set, so the
> parameterization needs to go last in the hierarchy (even though the
> parameters affect the test class and not the test method). As a matter
> of fact, that's how JUnit and also my junit-interface print
> parameterized test names:
> package.and.class.Name.methodName[paremeter-index]. But the two-level
> hierarchy expects the method name to go last. Actually, my recursive
> hierarchy did that, too, so here's one more change to fix it:
> https://github.com/szeiger/test-interface/commit/0606409a11541efaec417a471a40005280be5a8d
The point of the overhaul is to better support features of test frameworks, so this kind of information is useful. I'll let Bill comment on how he thinks this should be integrated.
-Mark
> Cheers,
> Stefan
>
Stefan Zeiger
> Having lots of classes with the same names in different packages (and
> always wanting to use both at the same time) is quite annoying in Java
> because you cannot rename imported names. OTOH, inventing new names or
> arbitrary prefixes for everything is probably worse.
> You are referring to having both the old and new interface included?
I'm referring to implementing both the old and the new interface with
the same test framework.
>> Give events and selectors a "display name" (required for the adapters):
>> https://github.com/szeiger/test-interface/commit/397e7ebc663d4f7c8a6348d08eeb6a49b3f5d602
>>
>> RunStatus seems to be unused:
>> https://github.com/szeiger/test-interface/commit/b809e44dc588e2ccaf56a661ac9df52ec8deaabb
>>
>> Who benefits from the strict one-or-two-level hierarchy for selectors?
>> Test frameworks shouldn't care. Are there any test runners which make
>> (or should make) use of this? We could simplify the API quite a bit by
>> making selectors fully hierarchical. No top-level
>> fullyQualifiedClassName, no isModule, no suite IDs or test names, no
>> Selector subclasses, just a Selector with a SelectorType (class, object,
>> method), a display name, a parent Selector and guaranteed equality
>> semantics. Mapping back from a Selector to some test artifact that can
>> be run is the task of the test framework which created the Selector, so
>> it can put whatever extra data it needs for that job into the Selector.
>> Here we go:
>> https://github.com/szeiger/test-interface/commit/89815947ecaf7625a87b73e54486f9d1fd53387e
> The main problem here is that this information needs to be serialized in some way. Think 'test-only selector-string'.
OK, that changes things. The way I understood what Bill said so far is
that a test framework is always passed selectors that it constructed
itself. So you're saying that sbt actually has to construct the
selectors because test-only now takes a selector string instead of a
class? So how is that string parsed? How can it tell
my.package.class.method from mypackage.class.nestedSuite.method? Does it
use a different syntax for nested suites?
Apparently there are three different uses for selectors:
1. Telling the runner where an event occured (useful for creating a
tree-shaped report like Eclipse shows it for JUnit test runs).
2. Asking a test framework to run a set of tests specified by the user
with a glob expression ("test-only").
3. Asking a test framework to re-run a test for which it previously
generated an event.
#1 is easy. #3 should be subsumed under #2 in order to avoid class
loader leaks.
I see two possible solutions:
1. Use the simple hierarchical selector model that I proposed and give
the test framework full control over the pattern matching. You could do
this by passing an extra argument containing the selector pattern string
to Framework.runner. sbt would then iterate over all tests and pass them
to the Runner which performs the pattern matching and simply ignores the
non-matching ones. A proper test framework implementation would have to
accept at least the display names for its own selectors as input patterns.
2. Define a fixed format for the display name, e.g. "suite1.test". It
could still support arbitrary nesting, e.g. "suite1/suite2.test",
"suite1/suite2/suite3.test" where suite1 would be a test class name and
the meaning of suite2, suite3 and test would be up to the test framework.
In either case, using classes instead of interfaces for selectors makes
sense. If a test framework cannot add its own data to a selector,
there's no need to make it look like it can. I'd even make the classes
final.
-sz
Mark Harrah
This isn't really determined yet, so your feedback is helpful. I do want to integrate the more fine-grained interface into the command line somehow, such as through test-only.
> Apparently there are three different uses for selectors:
> 1. Telling the runner where an event occured (useful for creating a
> tree-shaped report like Eclipse shows it for JUnit test runs).
> 2. Asking a test framework to run a set of tests specified by the user
> with a glob expression ("test-only").
> 3. Asking a test framework to re-run a test for which it previously
> generated an event.
>
> #1 is easy. #3 should be subsumed under #2 in order to avoid class
> loader leaks.
Yes, this is correct. If we need to properly separate these uses into different types, that might be possible.
> I see two possible solutions:
>
> 1. Use the simple hierarchical selector model that I proposed and give
> the test framework full control over the pattern matching. You could do
> this by passing an extra argument containing the selector pattern string
> to Framework.runner. sbt would then iterate over all tests and pass them
> to the Runner which performs the pattern matching and simply ignores the
> non-matching ones. A proper test framework implementation would have to
> accept at least the display names for its own selectors as input patterns.
>
> 2. Define a fixed format for the display name, e.g. "suite1.test". It
> could still support arbitrary nesting, e.g. "suite1/suite2.test",
> "suite1/suite2/suite3.test" where suite1 would be a test class name and
> the meaning of suite2, suite3 and test would be up to the test framework.
>
> In either case, using classes instead of interfaces for selectors makes
> sense. If a test framework cannot add its own data to a selector,
> there's no need to make it look like it can. I'd even make the classes
> final.
A combination of 1 and 2 might also be possible. I like 2 because sbt can provide tab completion for it. 1 may be useful for things not easily covered by 2, though.
-Mark
> -sz
Bill Venners
Originally the role of Selectors was only #3. I have been unaware of a
requirement to do anything like #1, and that's also why there was no
displayName. There was no need for that. Is there a need for that?
Essentially, "display" info gets sent to the logger. The events
contain just info used for rerunning, allowing sbt to keep track of
what's run, etc., not for displaying anything to users.
In general people do want to say they run just this test, or run tests
that match this wildcard, but I don't believe Mark and I had talked
about how to do that. I'm sure they'd like to be able to do that from
sbt as well. It can be done with selectors, except for the wildcard
part. ScalaTest 2.0 can run tests based on a wildcard, but I'm not
sure the others can. If that's something you want to add, I'd put it
in as a fifth subtype of Selector. sbt would need to define some kind
of syntax to differentiate a test name and a test name wildcard from a
fully qualified class name. Trouble is that * is a valid test name
character. In ScalaTest you say -s fully.qualified.SuiteClassName -t
"long winded test name" -z "wildcard". There is no glob character. A
-z gets will select any test that includes the given string.
Frankly I'd like to give ScalaTest users access to this Runner
functionality from sbt too, but it would mean sbt needs to define a
syntax for it. And it also means that possibly the framework API would
need to have a way to indicate to sbt it can't do what was asked.
> #1 is easy. #3 should be subsumed under #2 in order to avoid class loader
> leaks.
>
> I see two possible solutions:
>
> 1. Use the simple hierarchical selector model that I proposed and give the
> test framework full control over the pattern matching. You could do this by
> passing an extra argument containing the selector pattern string to
> Framework.runner. sbt would then iterate over all tests and pass them to the
> Runner which performs the pattern matching and simply ignores the
> non-matching ones. A proper test framework implementation would have to
> accept at least the display names for its own selectors as input patterns.
>
> 2. Define a fixed format for the display name, e.g. "suite1.test". It could
> still support arbitrary nesting, e.g. "suite1/suite2.test",
> "suite1/suite2/suite3.test" where suite1 would be a test class name and the
> meaning of suite2, suite3 and test would be up to the test framework.
>
> In either case, using classes instead of interfaces for selectors makes
> sense. If a test framework cannot add its own data to a selector, there's no
> need to make it look like it can. I'd even make the classes final.
>
> -sz
>
I agree the Selector classes should be final. I'll make them final.
And I'll also remove RunStatus, which was a leftover.
Bill Venners
On Mon, Apr 23, 2012 at 3:34 PM, Mark Harrah <dmha...@gmail.com> wrote:
> On Mon, 23 Apr 2012 18:58:32 +0200
> Stefan Zeiger <szei...@googlemail.com> wrote:
>
>> On 2012-03-22 3:10, Bill Venners wrote:
>> There are concrete classes in the new version (for events and
>> selectors). This makes it more difficult to fit your own implementation
>> into this system and properly reuse code, especially in Java where you
>> don't have traits or self-types. I'd prefer to have only interfaces and
>> enumerations in the API. Sensible default implementations could still be
>> provided but they should never be referenced by implementers (unless, of
>> course, you're creating an instance of such a class yourself). Here's
>> the "purified" API without classes:
>> https://github.com/szeiger/test-interface/commit/0445e29492ff1d5638392284b9bfa47abc65198a
>
> I tend to prefer interfaces as well. I forget if there was a good reason for concrete classes. I'll let Bill comment here.
>
I changed both the Selector and Event subclasses to final classes. My
thought here was that the client would simply instantiate these guys,
not extend them.
One insight is that our goal with making this a Java, not Scala, API
in the first place was to eliminate a Scala version dependency. I.e.,
it was for binary compatibility, not because most test frameworks
would be implementing it in Java. In the JUnit case it makes sense
you'd impelement it in Java because you get binary compability. For
ScalaTest, specs, ScalaCheck, etc., it makes more sense to implement
it in Scala, because it goes right into the same jar file that has the
rest of the test framework, which already has a Scala version
dependency.
Also, I wanted the client to be able to use the same class to
implement both the old and new Framework interfaces if they wanted. I
wanted to do that in ScalaTest, for example, but it isn't strictly
necessary. You could have two different classes. I figured clients
would support both for some time, then someday drop support for the
older one (at which point versions of sbt older than 0.13 or whatever
version gets this new Framework API would no longer work with that
test framework).
As to Eric's question about the two different task methods. One was
intended to be used by sbt as a result of discovery or the user
explicitly naming test classes after test-only. That's the one that
has the Fingerprint in it. The other (the one with Selectors) was
intended to be used only when rerunning tests reported about via
Events. So that task method doesn't take a Fingerprint. sbt may not
know the Fingerprint, because it is getting the selectors out of
events, and in fact there may not be a valid Fingerprint. What about
the Fingerprint do you need to know when Selectors are being used?
Please include me in the CC list when you reply because I don't get
emails from this list.
Stefan Zeiger
> In general people do want to say they run just this test, or run tests
> that match this wildcard, but I don't believe Mark and I had talked
> about how to do that. I'm sure they'd like to be able to do that from
> sbt as well. It can be done with selectors, except for the wildcard
> part. ScalaTest 2.0 can run tests based on a wildcard, but I'm not
> sure the others can. If that's something you want to add, I'd put it
> in as a fifth subtype of Selector. sbt would need to define some kind
> of syntax to differentiate a test name and a test name wildcard from a
> fully qualified class name. Trouble is that * is a valid test name
> character. In ScalaTest you say -s fully.qualified.SuiteClassName -t
> "long winded test name" -z "wildcard". There is no glob character. A
> -z gets will select any test that includes the given string.
>
> Frankly I'd like to give ScalaTest users access to this Runner
> functionality from sbt too, but it would mean sbt needs to define a
> syntax for it. And it also means that possibly the framework API would
> need to have a way to indicate to sbt it can't do what was asked.
The way junit-interface does that at the moment is that you can use
test-only with a pattern to filter on test classes and test methods
(functionality provided by sbt) but you need to use a custom option for
more expressive regexp filtering which includes the ability to filter on
parameter sets for parameterized tests:
-tests=<REGEXPS> Run only the tests whose names match one of the specified
regular expressions (in a comma-separated list). Non-matched tests are
ignored. Only individual test cases are matched, not test classes. Use
sbt's "test-only" command instead to match test classes.
It would be nice if sbt could support a selector hierarchy to which I
can map parameterized JUnit tests so you could use the standard
test-only filter patterns for everything.
-sz
Bill Venners
Is it better to let the test frameworks themselves provide such
features via arguments placed after the -- mark? I can see already
that what you're doing with JUnit doesn't relate to what ScalaTest is
doing regarding regexps for test names. I'd bet ScalaCheck doesn't
allow you to do any kind of selecting of property methods in its
Properties trait. This smells more like a framework-specific feature.
So sbt could just let people specify class names to test-only before
the --, and then after the --, individual test frameworks could give
people access to their various kinds of features, if any, for
selecting only specific tests.
Bill
Bill Venners
I spoke with Stefan today and we figured out that the Selector API as it currently exists would allow him to rerun failed parameterized tests, so that was cool.
We also discussed the question of how sbt should enable individual tests to be specified via test-only, and felt that what makes sense is that sbt could allow wildcard class names to the left of --, then use that to narrow discovery and call into the task method that takes a fingerprint. Then to the right of --, individual test frameworks could allow users to express interest in specific tests in test-framework-specific ways if they want. Does that work for you?
I also refactored Event per Stefan's suggestion:
https://github.com/bvenners/test-interface/blob/master/src/org/scalasbt/testing/Event2.java
I wanted to ask if this is OK with you.(Note Event was an interface like this in the previous Framework API.) But let me know. If so I'll replace the Event class hierarchy with this interface.
Eric, I didn't hear back from you on how you are using the Fingerprint such that you'd want it even when being asked to rerun failed tests via the Runner.task method that takes a Selector. Can you elaborate (and please CC me).
Thanks.
Bill
etorreborre
etorreborre
Mark Harrah
Bill Venners <bi...@artima.com> wrote:
> Hi Mark,
>
> I spoke with Stefan today and we figured out that the Selector API as it
> currently exists would allow him to rerun failed parameterized tests, so
> that was cool.
>
> We also discussed the question of how sbt should enable individual tests to
> be specified via test-only, and felt that what makes sense is that sbt
> could allow wildcard class names to the left of --, then use that to narrow
> discovery and call into the task method that takes a fingerprint. Then to
> the right of --, individual test frameworks could allow users to express
> interest in specific tests in test-framework-specific ways if they want.
> Does that work for you?
> I also refactored Event per Stefan's suggestion:
>
> https://github.com/bvenners/test-interface/blob/master/src/org/scalasbt/testing/Event2.java
>
> I wanted to ask if this is OK with you.(Note Event was an interface like
> this in the previous Framework API.) But let me know. If so I'll replace
> the Event class hierarchy with this interface.
-Mark
Mark Harrah
etorreborre <etorr...@gmail.com> wrote:
> Hi,
>
> I think that having the ability to select a class as it is with "test-only
> *ClassName* -- arguments" provides enough flexibility.
> The only improvement I'd like to have is to get rid of '-only'.
>
> - "test" will test everything
> - "test -- arguments" will test everything passing arguments to the runner
> - "test *Class*" will just execute one test class
> - "test *Class* -- arguments" will execute one test class with some
> arguments passed to the runner
>
> Mark what would be the drawbacks of doing this?
-Mark
Bill Venners
Hi Bill,In specs2 there are 2 kinds of "suites" you might want to execute:- normal specifications (inheriting from the same base class)- files runner with your own customized way of discovering and filtering specifications and group them as one big specification (inheriting from another class)Those 2 suites have 2 different fingerprints and depending on the fingerprint I know how to run each one. Now if I need to re-run a failed test using a selector, how can I practically create the proper suite using the class name and selector only?
There's some logic I don't get here. In the first case, I need a class name and a fingerprint to know how to execute a suite, and in the second case, a class name only (+ isModule) is enough. Are you using different kinds of fingerprints in ScalaTest? And if you did how wouldn't you have this kind of issue? Do you have a public url where I could check your source code and maybe understand this point better?
Bill
--
Bill Venners
On Thu, 26 Apr 2012 13:36:12 -0700This is how test-only works currently. One goal of the new interface was to provide tab-completion at a finer granularity than just the class name. Test frameworks can already interpret arguments after -- however they want, but sbt cannot provide any tab completion for this.
Bill Venners <bi...@artima.com> wrote:
> Hi Mark,
>
> I spoke with Stefan today and we figured out that the Selector API as it
> currently exists would allow him to rerun failed parameterized tests, so
> that was cool.
>
> We also discussed the question of how sbt should enable individual tests to
> be specified via test-only, and felt that what makes sense is that sbt
> could allow wildcard class names to the left of --, then use that to narrow
> discovery and call into the task method that takes a fingerprint. Then to
> the right of --, individual test frameworks could allow users to express
> interest in specific tests in test-framework-specific ways if they want.
> Does that work for you?
I see. In our discussions somehow I missed that tab completion with greater granularity was a goal. I do remember vaguely some discussions, but I definitely never had that in mind for the Selectors. Since we missed the 0.12 cutoff date, why don't we think about this further?
Is what's needed some way for sbt to ask the framework for test names of a class plus a way to specify a set of test names for a class when asking the framework for a task? I think the asking for test names would be before a run started, so this getTestNames would be a method on Framework I think. You'd have to give us a class loader, and I maybe the test framework needs a way to *optionally* reply. But if you could get that info, then I think you could come up with whatever syntax you wanted to let people specify test names, including your own wildcard syntax, and the test frameworks would just be optionally given a subset of complete test names for each fully qualified class name when you call the task method that takes a fingerprint.
Bill
Bill Venners
> I also refactored Event per Stefan's suggestion:>That looks fine.
> https://github.com/bvenners/test-interface/blob/master/src/org/scalasbt/testing/Event2.java
>
> I wanted to ask if this is OK with you.(Note Event was an interface like
> this in the previous Framework API.) But let me know. If so I'll replace
> the Event class hierarchy with this interface.
Bill