Re: custom reporter for sbt
Robey Pointer
On Jan 13, 2012, at 3:20 PM, Robey Pointer wrote:
> I've been playing with a custom scalatest reporter for sbt 7, which will just draw a progress bar as tests run:
>
> [info] == mytest ==
> [info] Running org.scalatest.tools.Runner -p target/test-classes -r com.twitter.libkestrel.MyReporter
>
> Running 114 tests:
> [################ ] 31/114
>
> ...
>
> [info] == mytest ==
> [info] Running org.scalatest.tools.Runner -p target/test-classes -r com.twitter.libkestrel.MyReporter
>
> Running 114 tests:
> [############################################################] 114/114
> Finished in 0:01.744
>
> [info] == mytest ==
> [success] Successful.
>
> It was pretty easy to get hooked in, but I'm hooking in at a very high level:
>
> lazy val mytest = task { args =>
> val myargs = List("-p", "target/test-classes", "-r", "com.twitter.libkestrel.MyReporter")
> runTask(Some("org.scalatest.tools.Runner"), testClasspath, args ++ myargs).dependsOn(testCompile)
> }
>
> One bad side effect of hooking in via Runner is that after the tests complete, my mac jumps back to my primary desktop, launches a GUI java app, and immediately kills it. It's a bit jarring. :)
>
> Is there a better way to invoke a scalatest runner? Or maybe a better way to replace the default reporter when running tests from within sbt?
>
> My current reporter code is here, if anyone's interested in playing with it: https://github.com/robey/libkestrel/blob/report/src/test/scala/com/twitter/libkestrel/MyReporter.scala
>
Bill Venners
I do have a dots reporter planned for 1.8. Right now the only reporter
you can use with sbt is the one that writes to sbt's logger. There's
an impedence mismatch for some reporters, because the Framework
interface sbt uses to integrate with test frameworks doesn't include
the notion of a run. Several of ScalaTest's reporters do something at
the start of a run, such as ensure a directory is created (for
example, for writing JUnit XML files) or open a file. A custom
reporter such as yours should work fine, because it is writing to the
standard output. But since I'm working on 1.8, maybe I should just do
dots next and make sure that's available from sbt.
Is the GUI app that's getting launched the ScalaTest GUI? Doesn't seem
like it should start that since you gave it a custom reporter class.
What GUI is being launched then killed right away?
Bill
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
--
Bill Venners
Artima, Inc.
http://www.artima.com
Bill Venners
We tried integrating something like dots and/or your progress bar last
night. I think what we discovered is that dots isn't such a good idea.
One reason is that both sbt and ant at least don't print anything
unless you send a \n, or append a \n, so my concept of printing one
green or yellow dot for each non-failed test doesn't really work well
with those tools. And it doesn't look good interspersed with any print
statements being printed out by the tests. The progress bar you came
up with does work with those tools, but I think that's best done via a
custom reporter like you've written, at least for now.
So the question comes back to how to run a custom reporter through
sbt. As I mentioned in my previous email, the existing Framework
interface does not include the notion of a run. So there's no
expectedTestCount coming in to custom reporters, and your reporter
needs that to make a progress bar. I've been working with Mark Harrah
on an enhancement to Framework that would introduce the notion of a
run, so that you can do things like your custom reporter with sbt's
native test and test-only commands. But that won't help you if you're
using sbt 7 (and it won't available for quite a while).
What might be the best option is if we make an sbt task for the
current Framework interface that gives more complete access to
ScalaTest. We did this a few months back as an experiment. It is in a
branch somewhere. I can try and resurrect it and deploy it as a
snapshot. It would have to use ScalaTest's discovery and
parallelization, but would at least grant users a way to access to all
ScalaTest functionality from sbt, including plugging in custom
reporters like yours. I'll email again once we get it deployed as a
snapshot.
Bill
On Tue, Feb 14, 2012 at 5:57 PM, Robey Pointer <ro...@twitter.com> wrote:
Bill Venners
Chee Seng got our plugin working last night with sbt 0.7.7. What
version of Scala and ScalaTest are you using? And what version of sbt
exactly?
Thanks.
Bill
Robey Pointer
> Hi Robey,
>
> I do have a dots reporter planned for 1.8. Right now the only reporter
> you can use with sbt is the one that writes to sbt's logger. There's
> an impedence mismatch for some reporters, because the Framework
> interface sbt uses to integrate with test frameworks doesn't include
> the notion of a run. Several of ScalaTest's reporters do something at
> the start of a run, such as ensure a directory is created (for
> example, for writing JUnit XML files) or open a file. A custom
> reporter such as yours should work fine, because it is writing to the
> standard output. But since I'm working on 1.8, maybe I should just do
> dots next and make sure that's available from sbt.
Yeah, I'm happy to use a custom reporter. It's probably best, actually, because we could just include it in our set of standard extensions to sbt, and extend it as time goes on. (The one we have for ruby has some nice features like calling out the 10 slowest tests, and I think the progress bar should be optional -- if you have a bunch of failing tests, you may want to see the long view.)
Right now we're using sbt 7, and migrating from scala 2.8.1 to 2.9.1.
Our plans to upgrade to sbt 11 are in a bit of flux right now because it doesn't support nested repos yet, so not all internal projects can use it. I plan to at least migrate to sbt 11 in the near future for libkestrel & kestrel, so it would be fine with me if the custom reporter was only supported under sbt 11. (Mark McBride has done an excellent job of porting most of our sbt 7 extensions to sbt 11 already.)
> Is the GUI app that's getting launched the ScalaTest GUI? Doesn't seem
> like it should start that since you gave it a custom reporter class.
> What GUI is being launched then killed right away?
I can't tell what it is. It seems to start and immediately die. It just uses the java cup-of-coffee icon, so it's clearly something from java. I'll try to think of some way I can catch it and see what the heck it is.
Bill Venners
We have prepped two sbt plugins, one for sbt .7 and the other for sbt
.11. Both provide the same syntax. We built this a few months ago to
experiment with a nice "external DSL" for using ScalaTest with sbt.
This is a rough draft of our idea and will undergo syntax changes. It
may also have a lot of bugs. It is in just-fleshing-out-ideas shape
right now. Ultimately we want a similar DSL to work with test and
test-only, but that will require enhancements to sbt's Framework API.
I think it may make sense to also deploy an sbt plugin for people on
older versions of sbt (and for the time period until Framework gets
updated) which offers the same DSL.
To use this you'll need to build it locally for now, but that's easy
if you follow these steps:
######### // First, build the plugin and deploy it locally:
######### $ svn checkout
https://scalatest.googlecode.com/svn/branches/sbt-plugin-dot-11
A sbt-plugin-dot-11/test-st-plugin
A sbt-plugin-dot-11/test-st-plugin/build.sbt
A sbt-plugin-dot-11/test-st-plugin/project
A sbt-plugin-dot-11/test-st-plugin/project/plugins.sbt
A sbt-plugin-dot-11/test-st-plugin/src
A sbt-plugin-dot-11/test-st-plugin/src/test
A sbt-plugin-dot-11/test-st-plugin/src/test/scala
A sbt-plugin-dot-11/test-st-plugin/src/test/scala/ConsoleProgressBarReporter.scala
A sbt-plugin-dot-11/test-st-plugin/src/test/scala/AdderTest.scala
A sbt-plugin-dot-11/test-st-plugin/src/test/scala/WidgetTest.scala
A sbt-plugin-dot-11/test-st-plugin/src/main
A sbt-plugin-dot-11/test-st-plugin/src/main/scala
A sbt-plugin-dot-11/test-st-plugin/src/main/scala/Adder.scala
A sbt-plugin-dot-11/test-st-plugin/src/main/scala/Widget.scala
A sbt-plugin-dot-11/sbt-plugin
A sbt-plugin-dot-11/sbt-plugin/build.sbt
A sbt-plugin-dot-11/sbt-plugin/src
A sbt-plugin-dot-11/sbt-plugin/src/main
A sbt-plugin-dot-11/sbt-plugin/src/main/scala
A sbt-plugin-dot-11/sbt-plugin/src/main/scala/ScalaTestSbtPlugin.scala
Checked out revision 3424.
######### $ cd sbt-plugin-dot-11
######### $ ls
sbt-plugin/ test-st-plugin/
######### $ cd sbt-plugin
######### $ ls
build.sbt src/
######### $ sbt
[info] Set current project to sbt-plugin-11 (in build
file:/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/)
######### > reload
[info] Set current project to sbt-plugin-11 (in build
file:/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/)
######### > update
[info] Updating
{file:/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/}default-cce2cb...
[info] Resolving org.scala-tools.sbt#sbt_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#main_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#actions_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#classfile_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#io_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#control_2.9.1;0.11.1 ...
[info] Resolving org.scala-lang#scala-library;2.9.1 ...
[info] Resolving org.scala-tools.sbt#interface;0.11.1 ...
[info] Resolving org.scala-tools.sbt#logging_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#process_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#classpath_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#launcher-interface_2.9.1;0.11.1 ...
[info] Resolving org.scala-lang#scala-compiler;2.9.1 ...
[info] Resolving org.scala-tools.sbt#incremental-compiler_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#collections_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#api_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#persist_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.9.0;0.4.0 ...
[info] Resolving org.scala-tools.sbt#compile_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#ivy_2.9.1;0.11.1 ...
[info] Resolving org.apache.ivy#ivy;2.2.0 ...
[info] Resolving com.jcraft#jsch;0.1.31 ...
[info] Resolving commons-httpclient#commons-httpclient;3.1 ...
[info] Resolving commons-logging#commons-logging;1.0.4 ...
[info] Resolving commons-codec#commons-codec;1.2 ...
[info] Resolving org.scala-tools.sbt#completion_2.9.1;0.11.1 ...
[info] Resolving jline#jline;0.9.94 ...
[info] Resolving org.scala-tools.sbt#run_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#task-system_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#tasks_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#tracking_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#cache_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#testing_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] Resolving org.scala-tools.sbt#compiler-interface;0.11.1 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_0;0.11.1 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_9_0;0.11.1 ...
[info] Done updating.
[success] Total time: 2 s, completed Feb 17, 2012 5:36:56 PM
######### > compile
[info] Compiling 1 Scala source to
/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/target/scala-2.9.1/sbt-0.11.1/classes...
[success] Total time: 5 s, completed Feb 17, 2012 5:37:04 PM
######### > package
[info] Packaging
/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/target/scala-2.9.1/sbt-0.11.1/sbt-plugin-11-0.3.0.jar
...
[info] Done packaging.
[success] Total time: 0 s, completed Feb 17, 2012 5:37:08 PM
######### > publish-local
[info] Packaging
/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/target/scala-2.9.1/sbt-0.11.1/sbt-plugin-11-0.3.0-sources.jar
...
[info] Done packaging.
[info] Generating API documentation for main sources...
[info] Wrote /Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/target/scala-2.9.1/sbt-0.11.1/sbt-plugin-11-0.3.0.pom
[info] :: delivering :: org.scalatest#sbt-plugin-11;0.3.0 :: 0.3.0 ::
release :: Fri Feb 17 17:37:13 PST 2012
[info] Packaging
/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/target/scala-2.9.1/sbt-0.11.1/sbt-plugin-11-0.3.0.jar
...
[info] Done packaging.
[info] delivering ivy file to
/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/target/scala-2.9.1/sbt-0.11.1/ivy-0.3.0.xml
model contains 2 documentable templates
[info] API documentation generation successful.
[info] Packaging
/Users/bv/nobkp/delus/sbt-plugin-dot-11/sbt-plugin/target/scala-2.9.1/sbt-0.11.1/sbt-plugin-11-0.3.0-javadoc.jar
...
[info] Done packaging.
[info] published sbt-plugin-11 to
/Users/bv/.ivy2/local/org.scalatest/sbt-plugin-11/scala_2.9.1/sbt_0.11.1/0.3.0/poms/sbt-plugin-11.pom
[info] published sbt-plugin-11 to
/Users/bv/.ivy2/local/org.scalatest/sbt-plugin-11/scala_2.9.1/sbt_0.11.1/0.3.0/jars/sbt-plugin-11.jar
[info] published sbt-plugin-11 to
/Users/bv/.ivy2/local/org.scalatest/sbt-plugin-11/scala_2.9.1/sbt_0.11.1/0.3.0/srcs/sbt-plugin-11-sources.jar
[info] published sbt-plugin-11 to
/Users/bv/.ivy2/local/org.scalatest/sbt-plugin-11/scala_2.9.1/sbt_0.11.1/0.3.0/docs/sbt-plugin-11-javadoc.jar
[info] published ivy to
/Users/bv/.ivy2/local/org.scalatest/sbt-plugin-11/scala_2.9.1/sbt_0.11.1/0.3.0/ivys/ivy.xml
[success] Total time: 2 s, completed Feb 17, 2012 5:37:15 PM
######### > exit
######### // Now, you can try out the plugin (and your custom
reporter) in the sibling project:
######### $ cd ..
######### $ ls
sbt-plugin/ test-st-plugin/
######### $ cd test-st-plugin/
######### $ sbt
[info] Loading project definition from
/Users/bv/nobkp/delus/sbt-plugin-dot-11/test-st-plugin/project
[info] Updating
{file:/Users/bv/nobkp/delus/sbt-plugin-dot-11/test-st-plugin/project/}default-bf6a2e...
[info] Resolving org.scalatest#sbt-plugin-11;0.3.0 ...
[info] Resolving org.scala-tools.sbt#sbt_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#main_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#actions_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#classfile_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#io_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#control_2.9.1;0.11.1 ...
[info] Resolving org.scala-lang#scala-library;2.9.1 ...
[info] Resolving org.scala-tools.sbt#interface;0.11.1 ...
[info] Resolving org.scala-tools.sbt#logging_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#process_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#classpath_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#launcher-interface_2.9.1;0.11.1 ...
[info] Resolving org.scala-lang#scala-compiler;2.9.1 ...
[info] Resolving org.scala-tools.sbt#incremental-compiler_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#collections_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#api_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#persist_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.9.0;0.4.0 ...
[info] Resolving org.scala-tools.sbt#compile_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#ivy_2.9.1;0.11.1 ...
[info] Resolving org.apache.ivy#ivy;2.2.0 ...
[info] Resolving com.jcraft#jsch;0.1.31 ...
[info] Resolving commons-httpclient#commons-httpclient;3.1 ...
[info] Resolving commons-logging#commons-logging;1.0.4 ...
[info] Resolving commons-codec#commons-codec;1.2 ...
[info] Resolving org.scala-tools.sbt#completion_2.9.1;0.11.1 ...
[info] Resolving jline#jline;0.9.94 ...
[info] Resolving org.scala-tools.sbt#run_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#task-system_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#tasks_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#tracking_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#cache_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#testing_2.9.1;0.11.1 ...
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] Resolving org.scala-tools.sbt#compiler-interface;0.11.1 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_1;0.11.1 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_0;0.11.1 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_9_0;0.11.1 ...
[info] Done updating.
[info] Set current project to A Project (in build
file:/Users/bv/nobkp/delus/sbt-plugin-dot-11/test-st-plugin/)
######### // The "test" command still works as usual, strings get
printed to sbt's logger:
######### > test
[info] Updating
{file:/Users/bv/nobkp/delus/sbt-plugin-dot-11/test-st-plugin/}default-ad01be...
[info] Resolving org.scala-lang#scala-library;2.9.1 ...
[info] Resolving org.scalatest#scalatest_2.9.1;1.7.1 ...
[info] Done updating.
[info] Compiling 2 Scala sources to
/Users/bv/nobkp/delus/sbt-plugin-dot-11/test-st-plugin/target/scala-2.9.1/classes...
[info] Compiling 3 Scala sources to
/Users/bv/nobkp/delus/sbt-plugin-dot-11/test-st-plugin/target/scala-2.9.1/test-classes...
[info] WidgetTest:
[info] - colour
[info] - disposition
[info] AdderTest:
[info] - 1 + 1 = 2
[info] Passed: : Total 2, Failed 0, Errors 0, Passed 2, Skipped 0
[success] Total time: 3 s, completed Feb 17, 2012 5:37:46 PM
######### // But you can also now use the ScalaTest plugin, which you
access by typing "st" instead of "test"
######### // I set up "stargs" in the project file to use your custom
reporter by default:
######### > st
Running 3 tests:
[#################### ] 1/3
[######################################## ] 2/3
[############################################################] 3/3
Finished in 0:01.605
[success] Total time: 2 s, completed Feb 17, 2012 5:37:50 PM
######### // If you say "-- stdout", it will override stargs and use
ScalaTest's stdout reporter instead:
######### > st -- stdout
Run starting. Expected test count is: 3
DiscoverySuite:
AdderTest:
- 1 + 1 = 2
WidgetTest:
- colour
- disposition
Run completed in 625 milliseconds.
Total number of tests run: 3
Suites: completed 3, aborted 0
Tests: succeeded 3, failed 0, ignored 0, pending 0
All tests passed.
[success] Total time: 1 s, completed Feb 17, 2012 5:37:57 PM
######### // If you want to specify specific tests, list them after
"st" before "--" (if you have a "--"):
######### > st com.example.test.AdderTest -- stdout
Run starting. Expected test count is: 1
AdderTest:
- 1 + 1 = 2
Run completed in 49 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 0, ignored 0, pending 0
All tests passed.
[success] Total time: 0 s, completed Feb 17, 2012 5:38:12 PM
######### // If you leave off the "--", it will once again use stargs
from build.sbt (which is your custom reporter)
######### > st com.example.test.AdderTest
Running 1 tests:
[############################################################] 1/1
Finished in 0:00.048
[success] Total time: 0 s, completed Feb 17, 2012 5:38:15 PM
>
Looks better in color. Anyway, voila! This allows you to use a custom
reporter from sbt. The syntax is definitely going to change. It is too
verbose and I want the sbt DSL and the Maven plugin and ant task and
ScalaTest shell syntax to all be as consistent as possible, so it is
easy to hop from one to another. I plan to release both the sbt DSL
and the Maven plugin final when I release ScalaTest 2.0 final, so
until then the syntax is subject to change. I'm very interested in
hearing suggestions on how to make it nicer. I don't have
documentation yet on what it looks like, but you can look at the
strings being parsed in
sbt-plugin-dot-11/sbt-plugin/src/main/scala/ScalaTestSbtPlugin.scala
to find out.
The plugin that works with sbt7 works the same way. Just do the same
thing but for this repo URL:
https://scalatest.googlecode.com/svn/branches/sbt-plugin-dot-7
Both of these plugins use reflection to call into ScalaTest, so they
should work for most any version of ScalaTest.
Bill
Robey Pointer
Bill Venners
On Fri, Apr 6, 2012 at 10:58 AM, Robey Pointer <ro...@twitter.com> wrote:
> Tried this out last night and this morning (under sbt 11 and scala 2.9), and
> it works great!
>
> The plugin looks pretty intense, though, making me wonder if it's all
> *rrrrreallllly* necessary. It might be just as good to have a way to pass
> parameters to the scalatest runner from sbt. For example, it would be okay
> to have something like
>
> ScalaTest.args += "-rcom.twitter.scalatest.BarReporter"
>
> Also it would be cool if there was an option to make "st" just be called
> "test". I'm sure there's some awesome way I could do that in my project file
> using functional combinators of some sort, but I wasn't able to puzzle it
> out within 5 minutes.
>
Ah, well, the main problem has been the existing Framework API, which
doesn't have a notion of a run. So we can't fire RunStarting and
RunCompleted events to the reporters. Some of them really need that,
and yours for example needs the expectedTestCount included in the
RunStarting event. Other custom reporters might need dispose so it
knows to close a file, for example. But for us to know when to fire
close, we'd need to know when the run ends, and right now sbt doesn't
tell us.
The new Framework API, which will be supported by sbt 0.12 should
solve most of these problems. And we'll have a nice DSL like the
plugin has just for using sbt's test and test-only directly. But I
think for people who can't upgrade to 0.12, an sbt ScalaTest plugin is
going to be the best way.
Now there's one other wrench in the works with regards to progress
bars, which is that sbt isn't going to ever send an expectedTestCount.
The reason is that sbt wants to be able fire off tests in one thread
while it may still be compiling or discovering tests in another. That
is more efficient, but it means we don't have an expectedTestCount
before we start running. (Some tests may not yet have been discovered
or even compiled when the run starts.)
So anyway, the new Framework API doesn't have a way for us to produce
an expectedTestCount in RunStarting. So I think our RunStarting will
probably fire a 0 expectedTestCount. But in 2.0, we're thinking of
adding an event called ExpectedTestCountUpdated, which would allow us
to dynamically increase the expected test count as sbt gives us more
suites to run. It could make progress bars kind of hop around or speed
up and slow down a but, but at least they would work with sbt. By
using a plugin, you're using ScalaTest's discovery, not sbt's, and
ScalaTest does do full discovery before starting to run. Less
efficient, perhaps, but makes for nice progress bars.
Bill
Robey Pointer
Bill Venners
On Sun, Apr 8, 2012 at 11:34 AM, Robey Pointer <ro...@twitter.com> wrote:
> I see… Those all sound like good ideas: having sbt send updates on the fly
> ("actually the total is 105") and getting hooked into test-only. Sucks that
> it seems to be a long way off.
>
> We won't really know until we can try out these things in real life, but I'm
> guessing that receiving test-count updates should be fine. Finding tests
> ought to happen much much faster than actually running them, so in the worst
> case, we could display a little twirly bar until sbt has finished counting.
> (I assume it would send some kind of signal when it's finished counting.)
>
Actually we hadn't thought of that, but I think that makes sense, and
just in time. I'll run that past Mark Harrah and see if he can send a
signal. Sbt won't be counting, but the test framework can count easily
enough, but sbt could probably tell us it is done with discovery (if
we provide a way in the new Framework API).
I was planning on adding DiscoveryStarting and DiscoveryCompleted
events in 2.0, but hadn't thought about them being used this way.
DiscoveryCompleted could give a final count. (The way I had been
thinking of using the DiscoveryStarting/Completed events was so a UI
could show a spinning disk during Discovery. Right now on the
ScalaTest GUI, for example, it looks like the whole thing is stuck
until discovery completes and the tests start running.)
Anyway, great suggestion.
Bill