table-driven property checks

[sami.d...@gmail.com]

Hi,

I just tried ScalaTest 1.5.RC2 (which did not break any of my existing
tests , BTW) table-driven property checks.
The issue I am hitting may not be directly related to ScalaTest, but
after investigating the issue, I came to the conclusion that I may be
missing the big-picture. So I am posting here in case I missed a
ScalaTest-specific design decision.
So, I don't expect to get the magic answer that will fix my problem,
but I would be really happy to hear of anything about scalatest that
might help me continue my investigation.

The problem I am trying to solve :
    I have some test code that I'd like to run against several databases

The solution I picked (please feel free to give your opinion on this
if it does not make sense):
    Using table-driven property checks and a big forAll {} loop that
will run every test for each database
The code is available from :
https://github.com/iglootools/ddddotron/blob/master/integration-tests/src/test/scala/org/iglootools/ddddotron/tests/RepositorySpec.scala

The problem :

val databases = Table(
    ("name", "dataSource", "txManager"),
    ("HSQLDB", getBean[DataSource]("hsqldbDataSource"),
getBean[PlatformTransactionManager]("hsqldbTxManager")),
    ("PostgreSQL", getBean[DataSource]("postgresDataSource"),
getBean[PlatformTransactionManager]("postgresTxManager"))
  )

If I ONLY add one of these databases (HSQLDB or PostgreSQL line), and
run all the tests, everything is fine. The code is working fine for
both db (separately)
The cleanup code gets executed before each test, and la vie est belle.

override def beforeEach() {
new SimpleJdbcTemplate(currentDataSource.get).update("""DELETE
FROM event""")
currentEventStore.get.deleteAllSnapshots
}

Now, if I run the tests using the 2 databases, I run into a very weird error :
I end up getting tons of DuplicateKeyException that reflect the fact
that my database is not cleaned-up between each test. Here is
additional information :

- The code does not even get to the second database before crashing.
The simple fact of adding the PostgreSQL line makes the HSQL tests
fail. (whereas they work fine with only one db in the Tables())
- The cleanup code is clearly executed between each test (I added some
tracing to check), but does not seem to have any effect when I add the
additional line
- I still need to further investigate transaction demarcation, but
from what I saw, no matter whether I create transactions or not, the
end-result is the same.

Regards,
Sami

[Bill Venners]

Hi Sami,

Well beforeEach is short for beforeEachTest. I actually decided to leave "Test" off the end because it is shorter and because BDD folks favor the word "example" over "test." But that may be unfortunately confusing the situation with table-driven testing. Possibly. I'm not sure this is what's going on because I don't see you're forAll's, but it looks like you're hoping beforeEach would be run before each row of the table is checked via the forAll. If that's true, then it won't work unless you're forAll makes tests. If your forAll is just sitting inside a test, then the beforeEach will happen once before the test, and inside the test you'll get two "property checks". Again if this is what's goin gon, the solution is to put the cleanup code inside the forAll expression, like:

forAll (databases) { (name, currentDataSource, txManager) =>
  // Clean up any previous stuff

  new SimpleJdbcTemplate(currentDataSource.get).update("""DELETE FROM event""")
  currentEventStore.get.deleteAllSnapshots

 //  do your property check stuff here
}

Am I on the right track?

Bill

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

[sami.d...@gmail.com]

Hi Bill,

thanks for your answer !

From what I could see with the logging, it looks like beforeEach and
afterEach are executed once for each it() {}
( I am nesting forAll(), describe() and it(), is that OK ?)

I only have one gigantic forAll, and I expect all the tests to be
executed for each database. And in fact, it seems to be the case

sami

[Bill Venners]

Hi Sami,

On Thu, May 19, 2011 at 4:35 PM, sami.d...@gmail.com <sami.d...@gmail.com> wrote:

Hi Bill,

thanks for your answer !

From what I could see with the logging, it looks like beforeEach and
afterEach are executed once for each it() {}
( I am nesting forAll(), describe() and it(), is that OK ?)

Yes, beforeEach/afterEach is executed once for each test, which in a Spec maps to an it() clause. You can nest forAll, describe, and it just like you're doing, no problem.
 

I only have one gigantic forAll, and I expect all the tests to be
executed for each database. And in fact, it seems to be the case

Yes, that's how it is intended to work. Users have a choice of whether to run property checks inside tests, or use the table to register tests. In the latter case you must make sure each test gets a unique name, like you're doing with your "describe(name)" clause.

In the former case, each test would be testing both databases and you can't use beforeEach/afterEach for cleanup between databases. That would look like:

describe("Using both databases") {
  it("some test") {
    forAll (database) { (name, dataSource, txMgr) =>
    }
  }

  it("another test") {
    forAll (database) { (name, dataSource, txMgr) =>
    }
   }
}

But the other way is to use the rows of the table to actually define your tests. That would look like what you have done:

forAll (databases) { (name, dataSource, txMgr) => {
  describe("Using the " + name + " database") {
    it("some test") {
    }

   it("another test") {
   }
  }
}

This gives you one set of tests per database in your table, and each test has the database name embedded in its test name, so all test names are unique still. And this way beforeEach/afterEach can be used for setup and teardown.

The only change I'd suggest is that when you use a table to generate tests like this, you really need  not use forAll, which is more geared for property checks. A table is a Seq, so you can just use a for expression if you want, like this:

for ((name, dataSource, txMgr) <- databases) {
  describe("Using the " + name + " database") {
    it("some test") {
    }

    it("another test") {
    }
  }
}

Bill
 

[sami.d...@gmail.com]

OK, looks good, thanks !

So, I seem to use ScalaTest correctly. The origin of my problem is
probably in my use of JDBC/Spring then !

Thank you !
Sami

[sami.d...@gmail.com]

FYI,

My problem was clearly lying in JDBC/Spring. I am still unsure of why
exactly, but now that I use a different application context for each
database, along with a different Spec for each database (that inherits
from the main RepositorySpec), everything is alright.

Thanks again for helping me drill down the issue.
Sami