Running a test class multiple times with a different configuration
Ken Egervari
different configurations. Is there a facility in scalatest to do this,
or any advice on how to go about it?
One idea that comes to mind is to write an abstract class with all the
tests, and then sub-classes can specify the configuration, but this
could get a tad bloated. If I have 8 classes under test, I will have 8
* N subclasses to deal with, and N is going to be 8 or 10 in the long-
term.
The N configurations will probably be the same for all the tests, so
there's not really a need to duplicate it in subclasses. I'd actually
like to specify the N configurations in 1 spot, and then somehow
relate those configurations to sub-set of my spec classes. These
configurations can run once when the Spec class is instantiated by the
test runner.
If there's no facility to do something like this, would it be
difficult to create something that would work in an IDE and with sbt
out of the box?
Thanks!
Bill Venners
Can you clarify what you mean by "configuration"? Is it different
objects the test runs need? Different external setup? Something else?
Thanks.
Bill
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
>
--
Bill Venners
Artima, Inc.
http://www.artima.com
Ken Egervari
Thanks very much for the response.
What I'd like to do is run the same tests against different databases,
as well as execute a specific SQL file contains the DDL for that
database. That way if I want to run the tests for a new database, I
can just specify a new configuration (the particular database
properties as well as the DDL file) and all the tests will run for
that configuration.
It would be great to specify the configurations in one location, and
then map them to some test classes. If there's a way I can do this
easily on my own, that'd be great ;) It'd be best to not have to add
anything to ScalaTest unless you think this is something of value. I'm
probably in the minority.
I do want to find a way to make sure all the tests run when I do a
root-level package run in IDEA or execute "sbt test" though. I'd
rather not have to specify some fancy build options or arguments to
make this work if it's possible, as that could be confusing and
annoying to anyone else working on the project.
I hope that outlines the problem a bit better ;) Thanks for any advice
you could offer.
Ken
Alex Boisvert
Here's a short example,
Bill Venners
Do you want to pick only one configuration for each run? In other
words, do you want a suite of tests that you can either run with an
in-memory database or a staging database, etc.? If that's what you're
after then you probably want to use ScalaTest's ConfigMap. This is
something that gets passed around for the purpose of configuring
tests. You could put entries in there for which DDL to use and which
database to use, for example, and your tests could use that. You can
pass any objects in the config map, and you can put strings in it via
Runner's command line.
On the other hand, you may want to reuse the same test code with
multiple configurations during the same run. If that's what your
after, one thing to look at is ScalaTest's support for shared tests.
Look in the ScalaDoc at the trait your using (Spec or FlatSpec, etc.).
That isn't the only way to do that kind of thing, though. If, for
example, you want to run each test in a suite of tests twice, you can
override withFixture and call test() twice. You can change the
configuration before each invocation of test(). Or if you'd rather set
one configuration, then run all the tests in that suite, then set
another configuration, then run all the tests again, you'd do that by
overriding runTests and implementing that behavior. If you want to do
this kind of thing in multiple suites, you'd pull that overridden
withFixture or runTests method out into a trait that you mix into any
suite you want to have this behavior.
Does something in those ideas sound like a good fit for your needs?
Bill
Bill Venners
Thanks for your post. That's a nice use of MultipleFixtureWordSpec.
I may be misunderstanding Alex's request, but I think he wants to run
the same tests with different fixtures. Whereas
MultipleFixtureWordSpec is more for helping you run different tests in
the same suite with different fixtures. I tried to address both of
these needs in ScalaTest.
So for example if you have tests A, B, and C, and fixtures X, Y, and
Z. Then MultipleFixtureWordSpec can help you do something like AX, BY,
and CZ. I.e, each test can declare which fixture it needs. Whereas
shared tests would help you do AX, BX, CX, AY, BY, CY, AZ, BZ, CZ.
Each test would be run multiple times, once with each fixture.
Bill
Ken Egervari
product mapping from {ABC} x {XYZ} is exactly what I want ;) I will
definitely look into what you've said in your first post as that
sounds exactly like what i need.
You see, I'm doing a project called ScalaDBTest, which I guess is a
sister project to yours. I just started yesterday, but I already got a
small grammar build for a dsl and it reads from files and puts them
into the database already. The aim of the project was to make
something that was much easier to use out of box (better defaults and
features), have it be much easier to read the test data files, and
perform faster than dbunit.
My project is here: http://github.com/egervari/scaladbtest
So your advice will go a long way to making this bad boy work for
various databases ;) Right now it just works with hsqldb (at least
that's all i've tested it on... it could actually work on others
perhaps but I've not tested it).
Thanks again Bill! If I have any trouble, I'll post again ;) Oh, if
you have any ideas/features for things that you've been craving for a
dbunit-like framework to have, now it's a good chance since I intend
to make something much better for the java/scala community.
Ken Egervari
a bunch of other stuff).
I started playing around and withFixture seemed to be a nice and
simple way to go about it... but then after 5 minutes I realized the
timing of this method call is very bad and won't suit my purposes.
Often times, objects that need a "dataSource" get a reference to it
before withFixture is called, so this won't work unless I somehow have
every test hook into this one with another hook method.
That's not really too painful, but I can this to work a lot more
transparently if I can set the "dataSource" before any shared state
tries to access it.
So I'm investigating overriding runTests(...) instead. However, I
don't think this method is getting called... or my implementation is
being skipped? I put all kinds of code in there like println()'s and
it appears that none of it is getting executed :(
Maybe runTests won't solve my problem anyway, because maybe it still
won't get called in time... which means I have two approaches - write
a hook method that all test suites that need dataSource's can
implement to make sure they get a valid reference to the dataSource...
or use a ConfigMap. ConfigMap seems like the "right" solution, but
it's going to complicate my build process and that's not something I
really want to invest in right now.
I'll play around with runTests a bit, just to see how it works and if
I can get the framework to call into it. If not, I'll do the hook
method approach to ensure that all interested parties get a valid
reference to the dataSource.
Ken Egervari
to remove duplication, for example:
describe("Test Data") {
val table = new Table(testData, "single_id_table")
<------------------
describe("when constructed") {
.....
As you can see from above, testData is going to be null if I use
withFixture. If I can find some way to fix this easily, then every
thing's going to work ;)
The idea:
override protected def withFixture(test: NoArgTest) {
runTestWithDataSourceAndDdl(test, TestContext.hsqldbDataSource,
"hsqldb.sql")
}
private def runTestWithDataSourceAndDdl(test: NoArgTest, dataSource:
DataSource, ddlFilename: String) {
this.dataSource = dataSource;
this.jdbcTemplate = new SimpleJdbcTemplate(dataSource)
this.testData = new TestData(dataSource)
createTables(ddlFilename)
test()
cleanUpTables()
}
def cleanUpTables() {
jdbcTemplate.update("delete from two_string_table")
jdbcTemplate.update("delete from single_id_table")
jdbcTemplate.update("delete from date_table")
}
Ken Egervari
initializeDataSourceReferences(dataSource: DataSource) which is a
protected method any Spec suite can override. Each database/ddl
configuration will call this method before it runs the test.
An example of the individual suite code looks like this:
var testData: TestData = _
var table: Table = _
override protected def setDataSource(dataSource: DataSource) {
testData = new TestData(dataSource)
}
...
}
This works, pretty much.
The only problem with this approach is that if i have multiple nested
describe() blocks under a parent describe() block, the usual use-case
is that "table" will need to be redefined N times for N nested
describe blocks, because the conditions on how the table was
constructed changes. Which means I'll have to create N vars for each
of these conditions before each test, because I have no idea which
describe() block I'm going to be in.
So it works, but it's smelly ;)
I also had to duplicate/unfactor several shared states and put them
the individual it() tests because they depended on this fixture. If
they exist outside the it() method, the dataSource is null... and I
don't really know a good way to avoid this duplication other than
putting it in the fixture too... but that's also smelly because it's
only used for 2 out of 19 tests.
Did I just royally mess this up and was there a better solution, or is
this pretty much the way to do it?
Bill Venners
Sorry I just noticed these emails, which means I somehow managed to go
several hours without checking email. No wonder I feel relaxed.
Anyway, do you mind posting the entire class? I'm not sure I can grok
exactly what you're trying to do, but I am hopeful there's a
sweet-smelling way to do it.
If you don't want to post the whole thing, let me ask a couple pointed
questions. Does each test need two fixture objects, testData and
table? Sounds like it. And then you want to run tests multiple times
such that testData is initialized with a different data source each
time? And from your earlier emails I think you want to run each test
with each datasource. Is that correct?
Ah, but there are also some other fixture objects you called "shared
states" in one of your emails, and those are only needed by some (2 of
the total 19) methods.
If I've got this right, then, you actually want to do AX, BX, AY, BY
(the different data sources) thing, as well as the AX, BY thing (the
shared states needed by 2 tests).
Can you confirm that's correct (and post the whole class if you don't
mind)? If so I can make some suggestions.
Thanks.
Bill
Ken Egervari
Thanks again for the response.
Yes, you are right about needing that cross product {ABC}x{XYZ} and
also needing shared state due to nested describe blocks. It is a very
complicated setup ;)
A good example to demonstrate the current problem is the Spec Suite
for the "Table" Class:
http://github.com/egervari/scaladbtest/blob/master/src/test/scala/scaladbtest/model/TableSpec.scala
As you browse this code, you'll notice that I make use of a lot of
shared state. In some cases, I instantiate Table and pass in
"testData", which is going to be null... but these individual tests
just happen to not require this reference so it's okay for now
(although that's definitely a smell... because if I try and use it
later, the test is going to fail for no reason other than testData
being null).
If you go to line 56:
http://github.com/egervari/scaladbtest/blob/master/src/test/scala/scaladbtest/model/TableSpec.scala#L56
... you'll see that both of these it() test blocks use the same "val
table" definition. I'd really like this kind of use case to be shared
while also not having to use the "catch all" fixture at the top of the
Spec... because it really doesn't apply to all the tests - just these
2.
On line 81:
http://github.com/egervari/scaladbtest/blob/master/src/test/scala/scaladbtest/model/TableSpec.scala#L81
... you'll see another table that's using the shared state way of
doing things. This is okay for now, but like the other ones, it could
break if I add more tests that actually need a real testData object.
On line 102 - easily the worse example:
http://github.com/egervari/scaladbtest/blob/master/src/test/scala/scaladbtest/model/TableSpec.scala#L102
... you'll notice a massive table definition. If I need to write more
tests with this same table definition, it's going to get pretty messy,
or at the very least, require some indirection which will make the
tests harder to read.
Here's all the tests in my model package:
http://github.com/egervari/scaladbtest/tree/master/src/test/scala/scaladbtest/model/
Here's the base class for all the Spec Suites that require a data
source in order to run:
http://github.com/egervari/scaladbtest/blob/master/src/test/scala/scaladbtest/DataSourceSpecSupport.scala
I hope that provides all the necessary info ;) Thanks very much for
the advice Bill.
Ken
Ken Egervari
actual project. The dbunit file had 502 lines, 29921 characters. My
format that I've been toying around with is only 420 lines... but more
remarkably, is only 20852 characters. So that's nearly a 1/3 reduction
in boilerplate.
The tests run in 42 seconds too, while using dbunit, they run in 58
seconds for 1093 tests (a lot of which use the database). Don't ask
why mine is faster - I did nothing special. I was going to investigate
ways to get the inserts to be faster still.
I also like the defaults on scaladbtest better. You don't have to
specify null values for polymorphic columns, you can specify default
values for columns that are the same across all/most records in your
test data, and a bunch of other features. And I've really just
scratched the surface since the project is like 3 days old and is
already better than dbunit (although I won't claim it's as robust...
that's silly).
It also takes less code/conceptual load to get it to work in a project
- even a java project no less.
Ken