Running SpecificationWithJUnit 1.6.0 specs with sbt
etorreborre
I've tried to compile specs 1.6.0 with sbt and everything works fine.
However when I try to launch my test suite only 125 tests are
executed. I suspect that Specifications inheriting the class
SpecificationWithJUnit are not executed. I quickly tried to understand
why from your source code but couldn't find an obvious reason.
Do you have any idea?
Thanks,
Eric.
Mark Harrah
If you can send me an example project or even just an example test class, I'll
make it into proper sbt test and fix the issue.
-Mark
etorreborre
class MySpec extends org.specs.SpecificationWithJUnit {
"this" should {
"work" in { 1 must_== 1 }
}
}
where SpecificationWithJUnit is defined as:
import org.junit.runner.RunWith
import org.specs.runner.{ JUnitSuiteRunner, JUnit }
@RunWith(classOf[JUnitSuiteRunner])
abstract class SpecificationWithJUnit extends Specification with JUnit
{ ... }
E.
Mark Harrah
Mark Harrah
could be plain classes. I only ever saw modules in the documentation
so sbt only looks for modules that inherit from Specification. If you
make your test a module it will work.
-Mark
etorreborre
inheriting from org.spex.Specification:
class matchingSpec extends spex.Specification { ... } for example.
Specs support both specifications as modules and as classes. The
latter is even mandatory if you want to run your specification as a
JUnit test.
How can I help you to progress with this thing? Is that possible for
you to incorporate all cases? Or maybe we drop this for now and we
focus on getting the Java interface idea working?
E.
On Oct 15, 9:38 am, Mark Harrah <dmhar...@gmail.com> wrote:
> sbt isn't finding that test because I didn't realize that specs tests
> could be plain classes. I only ever saw modules in the documentation
> so sbt only looks for modules that inherit from Specification. If you
> make your test a module it will work.
>
> -Mark
>
Mark Harrah
>
> Actually it is correctly instantiating classes like the ones
> inheriting from org.spex.Specification:
>
> class matchingSpec extends spex.Specification { ... } for example.
sbt's module detection or something else is going on in your setup.
> Specs support both specifications as modules and as classes. The
> latter is even mandatory if you want to run your specification as a
> JUnit test.
> How can I help you to progress with this thing? Is that possible for
> you to incorporate all cases? Or maybe we drop this for now and we
> focus on getting the Java interface idea working?
and ScalaTest support and it would be wrong to leave out specs! The
Java interface will pass responsibility to framework authors for
defining the superclass and whether tests are modules or classes.
-Mark
etorreborre
I committed my sbt project definition to svn (http://code.google.com/p/
specs/source/checkout) so you should be able to run a "sbt
test" (after maybe a tweak or two?).
Then you'll see that only 125 tests are executed on my project instead
of the 1000+.
As I wrote, it seems like the type of the class makes a difference in
what is run and what is not. Is there a debug trace in sbt to see why
a test class wasn't instantiated? I can see that all the test classes
are tested for inclusion in the includeTest method but maybe,
afterwards, there is an instantiation issue? You may also want to have
a look at my org.specs.util.Classes trait which I use to instantiate
Specification classes if that helps.
Thanks again for your help.
On Oct 15, 1:11 pm, Mark Harrah <dmhar...@gmail.com> wrote:
Mark Harrah
On 10/15/09, etorreborre <etorr...@gmail.com> wrote:
>
> Mark,
>
> I committed my sbt project definition to svn (http://code.google.com/p/
> specs/source/checkout) so you should be able to run a "sbt
> test" (after maybe a tweak or two?).
I did 'update' and then 'compile' and I get some errors in
org/specs/runner/ScalaTest.scala around line 164 that look like that
code hasn't been updated for the Reporter in ScalaTest 1.0. If I
comment those out, I get some other errors related to overriding.
> Then you'll see that only 125 tests are executed on my project instead
> of the 1000+.
>
> As I wrote, it seems like the type of the class makes a difference in
> what is run and what is not. Is there a debug trace in sbt to see why
> a test class wasn't instantiated? I can see that all the test classes
> are tested for inclusion in the includeTest method but maybe,
> afterwards, there is an instantiation issue? You may also want to have
> a look at my org.specs.util.Classes trait which I use to instantiate
> Specification classes if that helps.
I think I have this fixed (in addition to fixes for ScalaCheck 1.6 and
ScalaTest 1.0). I'd like to test it on specs to be sure, though. Can
you take care of the compile errors?
Thanks,
Mark
Bill Venners
On Thu, Oct 15, 2009 at 10:15 AM, Mark Harrah <dmha...@gmail.com> wrote:
>
> Hi Eric,
>
> On 10/15/09, etorreborre <etorr...@gmail.com> wrote:
>>
>> Mark,
>>
>> I committed my sbt project definition to svn (http://code.google.com/p/
>> specs/source/checkout) so you should be able to run a "sbt
>> test" (after maybe a tweak or two?).
>
> I did 'update' and then 'compile' and I get some errors in
> org/specs/runner/ScalaTest.scala around line 164 that look like that
> code hasn't been updated for the Reporter in ScalaTest 1.0. If I
> comment those out, I get some other errors related to overriding.
>
changes. I can go in and try and update that class for you and send
you a patch if you want.
Bill
> Thanks,
> Mark
>
> >
>
--
Bill Venners
Artima, Inc.
http://www.artima.com
etorreborre
project I committed.
I'm going to do a fresh checkout to fix that and I'll come back to
you. I can also test your changes on sbt 0.5.6-snapshot if you deploy
them (or just commit to svn).
E.
etorreborre
> changes. I can go in and try and update that class for you and send
> you a patch if you want.
1.6.1 of specs to be compatible with ScalaTest 1.0 and publish it.
E.
etorreborre
box and I get 125 tests being executed.
Mark Harrah
> box and I get 125 tests being executed.
be supported in trunk now. specs tests that are classes should be
supported, but that didn't get me up to 1000+ tests, only 225. I'll
try again tomorrow.
-Mark
Mark Harrah
On 10/15/09, Mark Harrah <dmha...@gmail.com> wrote:
> On 10/15/09, etorreborre <etorr...@gmail.com> wrote:
>>
>> Mark I fixed my sbt project so now everything compiles ok out of the
>> box and I get 125 tests being executed.
>
> Getting there... ScalaTest 1.0, specs 1.6, and ScalaCheck 1.6 should
> be supported in trunk now. specs tests that are classes should be
> supported, but that didn't get me up to 1000+ tests, only 225. I'll
> try again tomorrow.
There are a few problems that I have come across. First, the counting
granularity for sbt's specs support is too coarse- it is per class (or
whatever the top level is called in specs), not per example (again,
whatever it is called). That probably accounts for the lower number
of tests run than expected. I'll fix this.
Second, spex.Specification extends ScalaTest's Suite class, so sbt
picks up a subclass of spex.Specification as both a ScalaTest and
specs test and runs it once through specs and once through ScalaTest.
The tests that verify that certain code is only executed once then
fail. Not sure what to do about this one yet.
Third, Mockito doesn't clean up after its ThreadLocals, leading to
OOME: PermGen after a few 'test' executions. Not sure what to do
about this yet either.
-Mark
Bill Venners
On Fri, Oct 16, 2009 at 4:57 PM, Mark Harrah <dmha...@gmail.com> wrote:
>
> Eric,
>
> On 10/15/09, Mark Harrah <dmha...@gmail.com> wrote:
>> On 10/15/09, etorreborre <etorr...@gmail.com> wrote:
>>>
>>> Mark I fixed my sbt project so now everything compiles ok out of the
>>> box and I get 125 tests being executed.
>>
>> Getting there... ScalaTest 1.0, specs 1.6, and ScalaCheck 1.6 should
>> be supported in trunk now. specs tests that are classes should be
>> supported, but that didn't get me up to 1000+ tests, only 225. I'll
>> try again tomorrow.
>
> There are a few problems that I have come across. First, the counting
> granularity for sbt's specs support is too coarse- it is per class (or
> whatever the top level is called in specs), not per example (again,
> whatever it is called). That probably accounts for the lower number
> of tests run than expected. I'll fix this.
>
> Second, spex.Specification extends ScalaTest's Suite class, so sbt
> picks up a subclass of spex.Specification as both a ScalaTest and
> specs test and runs it once through specs and once through ScalaTest.
> The tests that verify that certain code is only executed once then
> fail. Not sure what to do about this one yet.
>
some other kind of test framework you recognize, be it specs or JUnit
or TestNG or something else, that in that case you don't run it with
ScalaTest but with the other one, whatever it is. I tried to design
ScalaTest so it could be integrated with other test frameworks, so
people can get a uniform report from heterogeneous tests. The only
other thing people might want is a single switch to say if it is both
ScalaTest and something else, do the opposite: always run it with
ScalaTest. But I'd wait on that until people ask and give you a good
justification for such a knob. I'd think the default would be to just
run with ScalaTest only when it is a Suite and nothing else you
recognize.
> Third, Mockito doesn't clean up after its ThreadLocals, leading to
> OOME: PermGen after a few 'test' executions. Not sure what to do
> about this yet either.
>
is make a bug report to the Mockito project. I have never gotten
around to it yet.
Bill
> -Mark
Mark Harrah
On 10/16/09, Bill Venners <bi...@artima.com> wrote:
> On Fri, Oct 16, 2009 at 4:57 PM, Mark Harrah <dmha...@gmail.com> wrote:
>> On 10/15/09, Mark Harrah <dmha...@gmail.com> wrote:
>> Second, spex.Specification extends ScalaTest's Suite class, so sbt
>> picks up a subclass of spex.Specification as both a ScalaTest and
>> specs test and runs it once through specs and once through ScalaTest.
>> The tests that verify that certain code is only executed once then
>> fail. Not sure what to do about this one yet.
>>
> I'd suggest that if a class is both an org.scalatest.Suite as well as
> some other kind of test framework you recognize, be it specs or JUnit
> or TestNG or something else, that in that case you don't run it with
> ScalaTest but with the other one, whatever it is. I tried to design
> ScalaTest so it could be integrated with other test frameworks, so
> people can get a uniform report from heterogeneous tests. The only
> other thing people might want is a single switch to say if it is both
> ScalaTest and something else, do the opposite: always run it with
> ScalaTest. But I'd wait on that until people ask and give you a good
> justification for such a knob. I'd think the default would be to just
> run with ScalaTest only when it is a Suite and nothing else you
> recognize.
In sbt, you declare your test frameworks in a list, so perhaps I
should just use the first framework that accepts the test. The
default could put specs before ScalaTest. Users could reorder the
list to match their preference.
>> Third, Mockito doesn't clean up after its ThreadLocals, leading to
>> OOME: PermGen after a few 'test' executions. Not sure what to do
>> about this yet either.
>>
> This is a problem for me in ScalaTest too. I think what we need to do
> is make a bug report to the Mockito project. I have never gotten
> around to it yet.
I'm never sure whether I diagnose PermGen errors properly until I
actually fix them, so it is good to know you tracked it down to the
same thing. I'll try to come up with a patch for this.
-Mark
Bill Venners
On Sat, Oct 17, 2009 at 11:36 AM, Mark Harrah <dmha...@gmail.com> wrote:
>
>>> Third, Mockito doesn't clean up after its ThreadLocals, leading to
>>> OOME: PermGen after a few 'test' executions. Not sure what to do
>>> about this yet either.
>>>
>> This is a problem for me in ScalaTest too. I think what we need to do
>> is make a bug report to the Mockito project. I have never gotten
>> around to it yet.
>
> I'm never sure whether I diagnose PermGen errors properly until I
> actually fix them, so it is good to know you tracked it down to the
> same thing. I'll try to come up with a patch for this.
>
What you seem to be saying is you run out of memory? My problem, which
I also haven't had time to track down, is that it appears to me that
Mockito is caching loaded classes by string name rather than class
instance. What ScalaTest does is that each time it does a run, it
creates a new class loader for what I call the "run path." (The run
path differs from the class path in that the class path is loaded from
the same class loader for the lifetime of the Runner VM (via the usual
class path class loader) and the run path is loaded from a new class
loader for each run. This allows you to make changes to fix bugs,
recompile, then just push rerun and those changes will be picked up.
When I do this with Mockito tests, I get class cast exceptions.
So the upshot is you can't do reruns in ScalaTest's GUIs with tests
that use Mockito, without potentially getting false failures from
ClassCastExceptions. Not a killer but I do need to submit an bug fix
or enhancement request to the Mockito folks.
etorreborre
> Third, Mockito doesn't clean up after its ThreadLocals, leading to
> OOME: PermGen after a few 'test' executions. Not sure what to do
> about this yet either.
On the Mockito mailing-list (but that was last July) it says that the
generated classes are not cached (http://groups.google.com/group/
mockito/browse_thread/thread/3ac9af08f5dfa999):
>Mockito shouldn't affect your PermGen. Mockito does generate classes
>on the fly but the generated classes are cached. Can't imagine what
>else could fill up permgen.
I'll try to investigate this with the latest sbt snapshot.
Eric.
> On 10/15/09, Mark Harrah <dmhar...@gmail.com> wrote:
Mark Harrah
First, this is my evaluation after memory profiling. I haven't fixed
the problem, so I'm not 100% sure this is the cause...
On 10/19/09, etorreborre <etorr...@gmail.com> wrote:
>
> Hi Mark, Bill,
>
>> Third, Mockito doesn't clean up after its ThreadLocals, leading to
>> OOME: PermGen after a few 'test' executions. Not sure what to do
>> about this yet either.
>
> I don't see that when I run the tests with the Maven surefire plugin.
> On the Mockito mailing-list (but that was last July) it says that the
> generated classes are not cached (http://groups.google.com/group/
> mockito/browse_thread/thread/3ac9af08f5dfa999):
the jvm shuts down after each time you run Maven. This is an issue of
class loaders not being garbage collected because of a ThreadLocal not
being cleaned up.
>>Mockito shouldn't affect your PermGen. Mockito does generate classes
>>on the fly but the generated classes are cached. Can't imagine what
>>else could fill up permgen.
problem is that ThreadLocal is a Java class loaded by the top-level
class loader. If you (Mockito, here) use ThreadLocal, you must
explicitly clear the value you set when you are done. Otherwise, the
top-level class loader now has a reference to an instance of one of
your classes. The class loader used to load your classes cannot be
garbage collected because there is a strong reference from the root
set to your class.
A tool like sbt (or maybe ScalaTest's graphical runner?) loads the
Mockito classes each test run. Because the classes can't be
collected, PermGen fills up.
> I'll try to investigate this with the latest sbt snapshot.
'test' on specs several times. If I make the PermGen about 60M
(should be plenty large enough for normal operation), I can get
PermGen errors just by running the propSpec test multiple times.
Running the literate tests, for example, won't give the errors.
-Mark
etorreborre
the reset method from mockito to avoid this phenomenon.
I'll try that and tell you.
Oh I see, the clas
> Hi Eric,
>
> First, this is my evaluation after memory profiling. I haven't fixed
> the problem, so I'm not 100% sure this is the cause...
>
etorreborre
specifications being executed with sbt-0.5.6-SNAPSHOT now :-)
Thanks,
Eric.
Mark Harrah
in the source code, so I wouldn't expect reset to help.
-Mark
Mark Harrah
The situation is improved if the three static ThreadLocals in Mockito
are replaced by synchronized WeakHashMaps keyed on the current Thread.
I can run 'test-only *propSpec*' more times before getting OOME:
PermGen and can continue to run tests successfully after getting the
OOME error (so it is probably now a soft OOME). It isn't the full
answer since the OOME still happens, but I think it shows that the
ThreadLocals are a problem.
-Mark
On 10/20/09, Mark Harrah <dmha...@gmail.com> wrote:
> I looked at the source for Mockito and it never clears the ThreadLocal
> in the source code, so I wouldn't expect reset to help.
>
> On 10/20/09, etorreborre <etorr...@gmail.com> wrote:
>>
>> Thanks Mark for the detailed explanation. I'm wondering if I could use
>> the reset method from mockito to avoid this phenomenon.
>>
>> I'll try that and tell you.
>>
Mark Harrah
I think this is a different problem. I'm not familiar with
ScalaTest's GUI runner, but if you can tell me how to reproduce the
problem, I'll take a look at it.
-Mark