Data-driven testing in scalatest vs. TestNG
Bill Lear
write and annotate a test generation method, and then "attach" it to a
test method with another annotation. For example, in Java, using
TestNG, you would write something like the following (abbreviated):
@DataProvider(name = "generateFibData")
Object[][] generateFibData() {
return new Object[][] {
{ 0, 0 }, { 1, 1 }, { 2, 1 }, { 3, 2 }, { 4, 3 }, { 5,
5 }, { 6, 8 }
};
}
@Test(dataProvider = "generateFibData")
public void test(int n, int expected) {
assertEquals(expected, Fibonacci.compute(n));
}
whereas, in scalatest, you might do this:
property("Fibonacci test") {
val data = Table(("n", "expected fibonacci number"),
(0, 0), (1, 1), (2, 1), (3, 2), (4, 3), (5, 5), (6, 8))
forAll (data) { (n: Int, expected: Int) =>
Fibonacci.compute(n) should equal (expected)
}
}
I'm trying to figure out the argument for doing things the "TestNG
way", and why the scalatest way (obviously, doable in Java as well) is
not simpler and clearer. I know one difference is that the Java
example above, when run through Maven, gives you 7 test results ---
one for each row of data --- whereas the scalatest version yields only
1.
Bill
Bill Venners
Sorry it took me a couple days to get back to you on this. I wanted to
code up an example. Good thing you asked and I did, too, because I
found a big fat bug in 1.5-snapshot. I wasn't catching
DiscardedEvaluationException in any table-driven property check! Can't
believe I didn't think to write a test for that. Anyway, I'll fix
that, but on to the matter at hand.
The short answer is I felt that usually people would prefer to see a
property check, whether it be table- or generator-driven as something
you do inside a test. Otherwise you'll end up with an explosion of
tests. I've BCC'd Cedric, so as not to reveal his email. Maybe he can
chime in on why TestNG does it on a per test basis.
Moreover, you can do it in ScalaTest on a per test basis if you want.
Here's an example in which a property is checked inside a single test:
class Fraction(n: Int, d: Int) {
require(d != 0)
require(d != Integer.MIN_VALUE)
require(n != Integer.MIN_VALUE)
val numer = if (d < 0) -1 * n else n
val denom = d.abs
override def toString = numer + " / " + denom
}
import org.scalatest.FunSuite
import org.scalatest.prop.TableDrivenPropertyChecks
import org.scalatest.matchers.ShouldMatchers
class FractionSuite extends FunSuite with ShouldMatchers with
TableDrivenPropertyChecks {
test("just one test") {
val fractions =
Table(
("n", "d"), // First tuple defines column names
( 1, 2), // Subsequent tuples define the data
( -1, 2),
( 1, -2),
( -1, -2),
( 3, 1),
( -3, 1),
( 3, -1),
( -3, -1)
)
forAll (fractions) { (n: Int, d: Int) =>
whenever (d != 0 && d != Integer.MIN_VALUE
&& n != Integer.MIN_VALUE) {
val f = new Fraction(n, d)
if (n < 0 && d < 0 || n > 0 && d > 0)
f.numer should be > 0
else if (n != 0)
f.numer should be < 0
else
f.numer should be === 0
f.denom should be > 0
}
}
}
}
Running this gives you one test for the entire property check:
Run starting. Expected test count is: 1
FractionSuite:
- just one test
Run completed in 121 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 0, ignored 0, pending 0
All tests passed.
Here it is refactored to create one test per row:
class Fraction(n: Int, d: Int) {
require(d != 0)
require(d != Integer.MIN_VALUE)
require(n != Integer.MIN_VALUE)
val numer = if (d < 0) -1 * n else n
val denom = d.abs
override def toString = numer + " / " + denom
}
import org.scalatest.FunSuite
import org.scalatest.prop.TableDrivenPropertyChecks
import org.scalatest.matchers.ShouldMatchers
class FractionSuite extends FunSuite with ShouldMatchers with
TableDrivenPropertyChecks {
val fractions =
Table(
("n", "d"), // First tuple defines column names
( 1, 2), // Subsequent tuples define the data
( -1, 2),
( 1, -2),
( -1, -2),
( 3, 1),
( -3, 1),
( 3, -1),
( -3, -1)
)
for ((n, d) <- fractions) {
test("fraction " + n + "/" + d + " normalizes correctly") {
whenever (d != 0 && d != Integer.MIN_VALUE
&& n != Integer.MIN_VALUE) {
val f = new Fraction(n, d)
if (n < 0 && d < 0 || n > 0 && d > 0)
f.numer should be > 0
else if (n != 0)
f.numer should be < 0
else
f.numer should be === 0
f.denom should be > 0
}
}
}
}
Running this gives you 8 tests:
Run starting. Expected test count is: 8
FractionSuite:
- fraction 1/2 normalizes correctly
- fraction -1/2 normalizes correctly
- fraction 1/-2 normalizes correctly
- fraction -1/-2 normalizes correctly
- fraction 3/1 normalizes correctly
- fraction -3/1 normalizes correctly
- fraction 3/-1 normalizes correctly
- fraction -3/-1 normalizes correctly
Run completed in 145 milliseconds.
Total number of tests run: 8
Suites: completed 1, aborted 0
Tests: succeeded 8, failed 0, ignored 0, pending 0
All tests passed.
As usual in ScalaTest, you can have it your way. The way it works is
that a Table is also a Seq, so you can use it in a for expression.
I've pulled out the tuple elements into n and d, and then define a
test whos name includes those values. You need to do that because
every test in a ScalaTest suite must have a unique name.
Thanks for asking. Now I must attend to a bug fix.
Bill
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
>
--
Bill Venners
Artima, Inc.
http://www.artima.com
Bill Venners
I see. I haven't done anything with just rerunning failures yet, other
than in the GUI.
In the ScalaTest case I do try to give people all the info they'd need
to figure out what went wrong when they are doing property checks
inside of tests. Here's what I got yesterday that made me realize I
had a big bug to fix, for example:
Run starting. Expected test count is: 1
FractionSuite:
- just one test *** FAILED ***
DiscardedEvaluationException was thrown during property evaluation. (z.scala:37)
Message: "None"
Occurred at table row 6 (zero based, not counting headings), which
had values (
n = -3,
d = 0
)
Run completed in 1 second, 71 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 0, failed 1, ignored 0, pending 0
*** 1 TEST FAILED ***
It tells you what exception was thrown, what line of code it was on,
what the message was, if any, which table row it happened on and what
the named values were on that row.
Bill
2011/5/11 Cédric Beust ♔ <ced...@beust.com>:
> Hi Bill and Bill,
> The main reason for using a @DataProvider annotation is, as you correctly
> guessed, for accurate reporting.
> It's very easy to emulate the way data providers work inside the test method
> itself (like ScalaTest does) but in this case, the testing framework has no
> idea that several invocations were made, so the testing framework will
> usually only report one test result, and if it's a fail, it won't be able to
> tell you which value failed.
> It will also not be able to generate a testng-failed.xml, which is another
> TestNG feature that users like a lot (in this case, TestNG will generate
> this XML file so that only the failing value will be rerun).
> Hope this helps, and feel free to ask if you have more questions.
> --
> Cédric