parametrized @DataProvider

[Jörg Gellien]

Hi all,

I am experimenting with DataProvider for a few days and now I found an
extension I like to discuss.
What I have:
I am using two data repositories containing testdata. The first is a
xml based repository
structured according to testclass/testmethod execution for now used in
my selfwritten JUnit Extension.
The second one is using a database where I provide testdata by doing a
simple JDBC select and inject these into the testmethods.
Because I want to use a generic approach to retrieve special testdata
on a per method basis in a testclass I need to specify a "dataQuery"
term and provide it to the @DataProvider method.
This would allow me to integrate my existing repositories easily in a
TestNG environment.

Something like:
@DataProvider(name="myDataProvider")
public Object[][] retrieveTestData(String dataQuery){
// evaluate dataQuery and do whatever is neccessary to provide
testdata
return ...;
}

@Test (dataProvider="myDataProvider", dataQuery="select id, name,
address from customer where id < 10")
public void checkValidAddressOnCustomer(int id, String name, String
address){
// check whatever is required
}

@Test (dataProvider="myDataProvider", dataQuery="select id, name, age
from customer where id < 10")
public void checkForYoungCustomers(int id, String name, int age){
// check whatever is required
}

Because I am rather new to Java annotations I am not sure if this is
the right place to provide such a dataQuery term.
And because I like to provide testdata on a per method basis it will
not be enough to use an equivalent to
@Parameters on @DataProviders defined in testng.xml.

Are there others interested in such an extension of TestNG or doesn't
this make sense to you?

Best regards

Joerg Gellien

[Cédric Beust ♔]

Hi Jorg,

Parameterized data providers...  quite an intriguing idea, and it does make a lot of sense.

But before we start discussing changing the current implementation, I believe you can already achieve something similar with the current TestNG.

What do you think of the following:

private Object[][] privateProvider(String query) {
  // use the query to get the data
}

@DataProvider(name = "providerFoo")
public Object[][] providerFoo() {
  return privateProvider("SELECT FOO...");
}


@DataProvider(name = "providerBar")
public Object[][] providerBar() {
  return privateProvider("SELECT BAR...");
}

@Test(providerName = "providerFoo")
...

@Test(providerName = "providerBar")
...

This approach respects the Don't Repeat Yourself principle and it doesn't require extra annotations.

What do you think?

--
Cedric

--
Cédric

[Jörg Gellien]

Hi Cedric,

actually I started with your proposal.
And found it not appropriate for the way I write my tests.
The datadriven approach makes a lot of sense to me during lets call it
functional testing of services.
Based on the requirements provided by domain specialists on such
services, e.g. CustomerService, I have to check a set of functions like
credibillity, valid address and others. For every of these functions
the analysts provide a set of valid and a set of invalid input data,
like customer data and other.

So starting with this simple szenario I begin with 1 testclass with 4
testmethods
(checkValidCredibility(), checkInvalidCredibility(),
checkValidAddress(), checkInvalidAddress()). - For now I am not talking
about unexpected behavior.

Because every scenario takes about 5 datasets as a starter I must write
4 DataProvider methods with special dataquery appropriate for this
case.

I am rather lazy and this makes me think about a way to avoid double
coding of the DataProviders. Every one of these only distinguishes in
the query term provided to my repository.
Actually I wrote my own extension of JUnit
(http://ddtunit.sourceforge.net) on this part but I found your way of
handling test selection and execution a lot more flexible and elegant
to that of JUnit.

So by providing a parametrized DataProvider in TestNG you can easily
integrate every data repository that works querybased. For every
supported repository there must be only one implementation.
This could be even adapted to Excel spreadsheet datasets like often
provided by people who are more familiar with office products than
database queries and perhaps writing xml data files. -> In this
scenario you could use the column names in a selfdefined query
structure and use Apache POI to access the datafiles.

If you want to add a new datadriven test just do it by adding the
appropriate annotation or change it if a new DataProvider is more
appropriate to a specific kind of test.

Please talk back ;-)

Best regards

Jörg

[Jörg Gellien]

Hi All,
here comes some more flesh to the bones.
After giving theoretical background on using @DataProvider with
database access I will now explain a scenario actually used in my
projects.

Requirements:
- hsql 1.8.0 (www.hsqldb.org)
For ease of use the standalone server and the example testdata were
used.
- a patched version of TestNG 4.2 that supports one additional variable
in @Test annotation: dataQuery and a @DataProvider method that can take
zero/one parameter: String dataQuery

While scanning the list for useful hints on DataProviders I stumbled
over @DataProver Iterators.
http://groups.google.com/group/testng-dev/browse_thread/thread/37b21d037e2f3ae7

Hey that is actually a good one. So I directly incorporated this in the
example.

The scenario:
Test functionality of a Customer service by providing a set of customer
information through database access.

What we need:
1) database hsqldb
=> extract from zip and add <instdir>/data
=> hsqldb started in server mode using default port.
<instdir>/demo/runServer.bat
=> run <instdir>/runManagerSwing.bat => Menu Options->Insert Test
Data, Menu Options->Commit
2) @DataProvider method
3) @Test methods using @DataProvider with additional parameter
dataQuery.

Here it comes: the @DataProvider:

@DataProvider(name = "jdbcDataProvider")
public Iterator retrieveDBData(String dataQuery) {
Iterator rowIterator = null;
Properties props = new Properties();
props.put("user", "sa");
props.put("password", "");
Connection connect = null;
try {
connect = DriverManager.getConnection(
"jdbc:hsqldb:hsql://localhost/", props);
Statement stmt = connect.createStatement();
ResultSet resultSet = stmt.executeQuery(dataQuery);
rowIterator = new JDBCResultSetIterator(resultSet);
} catch (SQLException ex) {
throw new RuntimeException("Error creating result set",
ex);
} finally {
if (connect != null) {
try {connect.close();} catch (SQLException ex) {}
}
}
return rowIterator;
}

And now some testing:
For example validating customer information:

@Test(dataProvider = "jdbcDataProvider", dataQuery = "SELECT id,
firstname, lastname FROM Customer")
public void validateCustomers(int count, String firstName, String
lastName) {
System.err.println(count + ") " + firstName + " " + lastName +
" ...");
}

and another test on the same testdata base:

@Test(dataProvider = "jdbcDataProvider", dataQuery = "select
firstname, street, city from Customer;")
public void validateAddress(String firstName, String street, String
city) {
System.err.println(firstName + " is living in " + city + ", " +
street);
}

There is no need for special wrappers on the @DataProvider. Just adding
the meta information on using external data and a specification saying
what data you want to use.

Because of the data iterator the footprint of this test keeps rather
small. It was implemented straight from the example MyIterator in
TestNG itself.

One open point I like to discussion is the way how to provide
information on parametrized @DataProviders. With my scenario there are
two possibilities of using this DataProvider:
Zero or one parameter. And it does not conform to the usual way of
using Parameters.

Any ideas how to fit the above scenario into the general schema of
TestNG annotations?

Best regards

Joerg

P.S.: If you are interested in the JDBCResultSetIterator just mail.

[faustothegrey]

Hi all,

I originally thought to integrate JTestCase with TestNG using
DataProviders,
but it turn out to be unnecessary.

JTestCase and TestNG can be integrated seamlessly like in the (working)
example reported at the bottom.

Naturally there is a lot to do as to otpimaze and specializing this
integration.

Also there is an issue "when to fail in the method" that i'll explain
in a following post.

-- TestNG JTestCase integration example --

@Test(groups = { "functest" }, enabled = true)
public class TestNG2JTestCase2 {
private static JTestCase _jtestcase = null;

@Configuration(beforeTestClass = true)
public static void setupClass() {
ppp("SETTING UP THE CLASS TestNG2JTestCase");
try {
String dataFile = "sample/tests/test-data.xml";
_jtestcase = new JTestCase(dataFile, "CalculatorTest");
} catch (Exception e) {
e.printStackTrace();
}

}

@Configuration(afterTestClass = true)
public static void tearDownClass1() {
ppp("TEARING DOWN THE TestNG2JTestCase");
}

@Test(groups = { "odd" })
public void testMethod1() {
try {
Vector testCases = _jtestcase
.getTestCasesInstancesInMethod("testCalculate");
// for each test case
System.out.println("Number of test case:: " + testCases.size());
for (int i = 0; i < testCases.size(); i++) {

// retrieve name of test case
TestCaseInstance testCase = (TestCaseInstance) testCases
.elementAt(i);

System.out.println("**********************************************************");
System.out.println("* Test Method: CalculatorTest::testCalculate");
System.out.println("* Test Case: " + i);

System.out.println("**********************************************************\n");

// get hashed params for this test case
HashMap params = testCase.getTestCaseParams();

Integer var1Int = (Integer) params.get("var1");
Integer var2Int = (Integer) params.get("var2");
Integer var3Int = (Integer) params.get("var3");
Integer var4Int = (Integer) params.get("var4");

String opt = (String) params.get("opt");

int var1 = var1Int.intValue();
int var2 = var2Int.intValue();
int var3 = var3Int.intValue();
int var4 = var4Int.intValue();

// Now comes to what we need to test.
// we don't want Exceptions to break our tests, so we catch
// Exceptions here.
try {
// ...All testing related to Calculate here...
Calculator calculator = new Calculator();
int result = calculator.calculate(var1+var3, var2+var4, opt);

// asserting result:
assert testCase.assertTestVariable("result",
(new Integer(result))) : testCase.getFailureReason() ;
// assertTrue("Assertion Failed : method=" +
// testCase.getMethod()
// + "testcase=" + testCase.getTestCaseName(), succeed);
// else
// fail("***Fail to test Calculate when asserting result!");

} catch (JTestCaseException e) {
e.printStackTrace();
// fail("***Unexpected exception is throwed from JTestCase:
// " + e.getMessage());
} catch (Exception e) {
System.out.println("---Test case throws exception as expected: "+
e.getMessage());
}
}
} catch (Exception e) {
e.printStackTrace();
}

}

}

[Jörg Gellien]

Hi Fausto,

one point for processing ;-)
one thing that disturbes me is that you provide kind of your own
execution model on top of TestNG without an integration to it.
(Oh what a sentence).
In my understanding of TestNG it is designed to use method parameters
if there are parametrized tests.
(Correct me Cedric if I am wrong)
This feature is fully supported by the existing Runners and Reporters.
For multiple execution of one Java testmethod you have @Factory or my
favorit @DataProvider.
With these you can structure your testcode much clearer by putting the
loop over existing testdata into one separate method
annotated by @DataProvider and provide for example a TestCaseInstance
as parameter to the testmethod.
Such an integration will automaticly resolve your problem on reporting
execution details.

Best regards
Jörg

[Cédric Beust ♔]

Hi Fausto,

Interesting, but I have to confess I'm not really convinced.

First of all, I think JTestCase's approach to data-driven testing is a bit heavy and not very flexible:

- The data can only be specified in XML
- It has to reference Java elements in XML, which is awkward to do and also a design smell to me (annotations are a better match for this kind of thing)

TestNG's DataProviders are a more flexible answer to this problem in my opinion because they hook seamlessly into the Java programming model by passing this data as parameters to test methods, which is very natural, and also because they are defined in Java, which presents several advantages:

- You benefit from the type checking performed by the compiler (do some refactoring on a JTestCase class and you'll need to update all your XML files by hand)
- You can keep the test and the way its data is generated close to each other (typically, in the same class)
- You can generate test data in many different ways, and not just from XML (Java code, properties file, XML file, database, network, etc...)

Is there any particular reason why you want to add JTestCase support to TestNG?  If it's for migration reasons (you already have plenty of JTestCase XML files), it should be very easy to write a generic @DataProvider(name = "JTestCaseProvider") that would parse these XML files and pass the data as method parameters, "the TestNG way".

What do you think?

--

Cédric

[faustothegrey]

Hi guys,
thank you for feedback. And ..ehehe, this kind of skepticism was
expected.

Joerg :

>one thing that disturbes me is that you provide kind of your own
>execution model on top of TestNG without an integration to it.

Nope. That's exactly the point. I just let TestNG tell me WHAT test to
do and JTestCase
do its work of abstracting the test case . They're "orthogonal", in
some sense.

Cedric:

>> First of all, I think JTestCase's approach to data-driven testing is a bit
>> heavy and not very flexible:
>> - The data can only be specified in XML
>>- It has to reference Java elements in XML, which is awkward to do and also
>> a design smell to me (annotations are a better match for this kind of thing)

Uhmm. But try to use annotation to assert that your multijoned SQL
query for
"goofy goof" user should produce a profile record with a sys admin
qualification ...

JTestCase can help you in keeping separate your test code from test
case
implemetation, when it is desiderable; which is the opposite with
@DataProvider does.
On the other DataProvider have all the advantage you listed.
What's best ? The one that fits your needs ... :)

>> With these you can structure your testcode much clearer by putting the
>> loop over existing testdata into one separate method
>> annotated by @DataProvider and provide for example a TestCaseInstance
>> as parameter to the testmethod.

>>XML files), it should be very easy to write a generic @DataProvider(name =

>> "JTestCaseProvider") that would parse these XML files and pass the data as
>> method parameters, "the TestNG way".

Ok. Perfect. Let's make a @JTestCaseDataProvider(file="testfile.xml")
that
passes a TestCaseInstance to the annotated method looping over it.

I volunteer for some implemenation. Anyone can kick me started ?

Fausto

[Cédric Beust ♔]

On 12/22/05, faustothegrey <f.l...@inera.it > wrote:

Uhmm. But try to use annotation to assert that your multijoned SQL
query for "goofy goof" user should produce a profile record with a sys admin
qualification ...

I was just saying that annotations are usually the right choice when you are trying to express metadata that applies to a Java package, class or method.

It's clearer, is type checked and will not break in the face of refactoring (JTestCase will).

JTestCase can help you in keeping separate your test code from test
case implemetation, when it is desiderable; which is the opposite with
@DataProvider does.

I'm not sure I agree with that.  @DataProviders let you specify your data from anywhere, not just an XML file.

Ok. Perfect. Let's make a @JTestCaseDataProvider(file="testfile.xml")
that passes a TestCaseInstance to the annotated method looping over it.

That's not exactly what I meant. I don't think you need a new annotation for this.

Just use a regular @DataProvider that will parse the JTestCase .xml file (Java and test data information) and return that data as method parameters.

--
Cédric

[faustothegrey]

Hi Cedric,

Feel like I haven't been much TestNGally-correct lately :) .

I don't want to give any "lecture" to you of course, but bear with me
for another couple of sentences. Then I'll quite down.

@DataProvider can pull information from the outside but don't have
JTestCase
TestCaseInstance abstraction, that lets you specify test parameters and
asserts
externally, equivalenty to DDTUnit. That's not just pulling in data for
testing. Ok
you can do it manually anyway, but would you do it if you can download
the
jar and go ....

>> was just saying that annotations are usually the right choice when you are
>> trying to express metadata that applies to a Java package, class or method.

Of course, by definition. For expressing test case params and asserts I
don't see
it like metadata. It's test case expressed in a separate xml file.
About refactoring,
JTestCase breaks down if you change the method name, but an entry point
is needed.

Is it possible to independently develop an annotation or some form of
@DataProviders
that relate to JTestCase or DDTUnit as a third-party plugin ? If it is
possibile,
it would be a greater boost for the JTestCase (and DDTUnit i guess)
community
to pass to TestNG from JUnit background ..

And, if JTestCase user will find out that DataProvider are better than
JTestCase itself,
well that's life ... ;)

Regards,
Fausto

[Cédric Beust ♔]

On 12/22/05, faustothegrey <f.l...@inera.it> wrote:

Hi Cedric,

Feel like I haven't been much TestNGally-correct lately :) .

I don't want to give any "lecture" to you of course, but bear with me
for another couple of sentences. Then I'll quite down.

Oh don't worry!  I certainly didn't feel you were lecturing me.  We're just exchanging viewpoints, and there's always something to learn for both parties in such discussions, so don't be shy!

The worst that can happen is that we will disagree, but there's nothing wrong with that :-)

@DataProvider can pull information from the outside but don't have
JTestCase TestCaseInstance abstraction, that lets you specify test parameters and
asserts externally, equivalenty to DDTUnit. That's not just pulling in data for
testing. Ok you can do it manually anyway, but would you do it if you can download
the jar and go ....

>>  was just saying that annotations are usually the right choice when you are
>>  trying to express metadata that applies to a Java package, class or method.

Of course, by definition. For expressing test case params and asserts I
don't see it like metadata. It's test case expressed in a separate xml file.

You can call it whatever you want, but the bottom line is that JTestCase needs to invent its own syntax to point to a precise location in Java:

<package name="org.foo">
  <class name="...">
  </...>
</package>

... etc.  You get the idea.

The problem with that, as I said earlier, is that it's not type safe and it will break if you refactor.  Annotations are usually a better way to express this information for that reason (and yes, I'm aware that testng.xml has the same problem, but it's supplied as a complement to annotations, not a substitute).

About refactoring, JTestCase breaks down if you change the method name, but an entry point
is needed.

Not sure what you mean here...

Is it possible to independently develop an annotation or some form of
@DataProviders that relate to JTestCase or DDTUnit as a third-party plugin ? If it is
possibile, it would be a greater boost for the JTestCase (and DDTUnit i guess)
community to pass to TestNG from JUnit background ..

Adding an annotation to TestNG means that you are changing its behavior, and the current plug-in API doesn't exactly support that.

There is a cleaner solution, though:  apt, the Annotation Processor Tool, which was shipped with JDK5.  It's the recommended way to add the treatment of your own annotations to Java programs.  I'm still not quite sure how that would work with TestNG exactly, but you might want to do some research in that direction if you are interested.  And please, keep me (us) posted of your efforts.  Adding annotations to TestNG is not something I had planned for but I'm curious where this will go.

And, if JTestCase user will find out that DataProvider are better than
JTestCase itself, well that's life ... ;)

:-)

--
Cédric

[faustothegrey]

Hi again

>>parties in such discussions, so don't be shy!

ok, thanks. Really I feel i'm overly bold here because
i'm not an TestNG expert so ( my fault .. my fault )

>>You can call it whatever you want, but the bottom line is that JTestCase
>>needs to invent its own syntax to point to a precise location in Java:

Ah, now I see! JTestCase doesn't point to the classes. In the test case
class, for each method, you specify the name of the method manually.
TestCaseInstance testInst =
JTestCase.getTestCasesInMethod("methodname"). "methodnane" is declared
in the xml file like
<method name="methodname">. The xml tag names are deciving, I hereditad
them :<. Far from ideal but it shields you from refractoring breakage
as you said. The entry point I was referring is "methodname", as you
can image.

>> There is a cleaner solution, though: apt, the Annotation Processor Tool,

uhmm, not sure i got your suggestion. I'll try to think about it anyway
or else I will ask for your help. What i was thinking about was
extending @DataProvider annotation. But not sure one can do it. Yeah,
need some more researching.

>> Adding annotations to TestNG is not something I had planned
>> for but I'm curious where this will go.

It's when you know that what you've done was allright. I mean,
making things you didn't plan for.

It's bed time for me now, so bye, thank you for the nice discussion
and hope we can talk again in the future.

Fausto

[Alexandru Popescu]

#: faustothegrey changed the world a bit at a time by saying on 12/23/2005 2:18 AM :#

I've been quietly watching this thread, cause I've havent got the time to investigate JTestCase. But
as far as I got it from this thread JTestCase externalize the data, the assertions and for doing it
you need a point to reference the Java code. It looks to me much like creating a new language. Am
I wrong?

./alex
--
.w( the_mindstorm )p.

[Cédric Beust ♔]

On 12/22/05, faustothegrey <f.l...@inera.it> wrote:

Ah, now I see! JTestCase doesn't point to the classes. In the test case
class, for each method, you specify the name of the method manually.
TestCaseInstance testInst =
JTestCase.getTestCasesInMethod("methodname"). "methodnane" is declared
in the xml file like
<method name="methodname">. The xml tag names are deciving, I hereditad
them :<. Far from ideal but it shields you from refractoring breakage

Mmmh, I confess I'm a bit lost.  Is "methodname" a real Java method?  If it is, then renaming it will break JTestCase because the name in the XML file won't be renamed as well, right?

Also, how does getTestCasesInMethod() find test cases inside a method?  (and what does that even mean?  test cases *inside* a method?)

>> There is a cleaner solution, though:  apt, the Annotation Processor Tool,
uhmm, not sure i got your suggestion. I'll try to think about it anyway
or else I will ask for your help. What i was thinking about was
extending @DataProvider annotation. But not sure one can do it. Yeah,
need some more researching.

I think (hope ;-)) you will find that modifying TestNG is fairly easy (especially with the latest annotation refactoring), but right now, I would probably not include a JTestCase patch into the TestNG core.

Which brings me to the other subject:  offering an API for people to add annotations (and their processing) inside TestNG, which is a very hard problem because it's not really clear where the extension points in TestNG should be... 

It's bed time for me now, so bye, thank you for the nice discussion
and hope we can talk again in the future.

Sure, any time!

--
Cédric

[Jörg Gellien]

Hi Alexandru,

you are right. - It is a simple language to describe object
instanciation using xml tailored to the needs of testcase execution
models like JUnit. (Holds true for JTestCase and DDTUnit as well)

Jörg

[Jörg Gellien]

Hi All,
quite a nice discussion - I enjoyed reading it.
Here is a litte clarification (hopefully)
Example (taken from DDTUnit - but hold true to JTestCase as well):
XML Resource:
...
<cluster id="myClass>
<group id="myMethod">
<test>
<objs><obj id="myObject" type="string">
Hallo World</obj>
<objs>
<!-- complex objects are possible -->
<asserts/> <!-- won't use for now -->
</test>
</group>
</cluster>
...
The mapping to testclass goes like this:
xml - Java
cluster -> testclass
group -> testmethod
test -> parameterset accessible inside of testmethod
(There is one testmethod invocation per test,
or for JTestCase one iteration inside explicit
loop in testmethod)

Every <test> inside of a <group> contains the same object structure.
(Number of objects, object typing) So these can be used in one Java
method invocation.
To specify a connection between <group> and Java testmethod you need a
rule
- the simplest: group@id -> Java method name

Now I am back to the topic why I started this thread:
How can I provide "selection criteria" of
xml-resource:cluster[@id="myClass"]/group[@id="myMethod"]
on a TestNG @Test method and iterate over existing <test> data?

The first starter was to use @DataProvider method and a field in the
testclass to specify the cluster@id.

Because <group> structures vary (like stated above, e.g. number of
<obj>)
Cedric proposed to use one parametrized method for actual selection of
group@id
and 1 @DataProvider per @Test:

@DataProvider(name = "providerBar")
public Object[][] providerBar() {
return privateProvider("SELECT BAR...");

}
@Test(providerName = "providerBar")

This kind of works, but ... it is not convenient if you are talking
about more than a handful of testcases.

So if we could provide parameters to a @DataProvider it will be
possible to define 1 @DataProvider
that will take special parameters from @Test annotation.

I would see this as a closure on @DataProvider not a special case
extension for DDTUnit or JTestCase.
- My first example used database select criteria as parameter.

Hope I could make my point ;-)

Best regards
Jörg

[Alexandru Popescu]

See down the page.

#: Jörg Gellien changed the world a bit at a time by saying on 12/23/2005 12:41 PM :#

IIRC you can provide parameters to a @DataProvider method. Gonna check this later.

./alex
--
.w( the_mindstorm )p.

> I would see this as a closure on @DataProvider not a special case
> extension for DDTUnit or JTestCase.
> - My first example used database select criteria as parameter.
>
> Hope I could make my point ;-)
>
> Best regards
> Jörg
>
>
>
>
>

> !DSPAM:43abd48a674831134112123!
>
>

[Jörg Gellien]

Hi Alexandru,

perhaps I overlooked it or Cedric included it lately. (I checked
Sources 4.2)
If you find anything let me know.

Thanks for the good support of the TestNG team!

Jörg

[faustothegrey]

Hi All,

First of all sorry Joerg to have distracted the topic from DDTUnit to
JTestCase
but the same arguments are valid for both frameworks, i guess. I'll
refer
to JTestCase because i know it better (hope we can continue our
discussion
about DDTUnit-JTestCase integration).

Then I'll +1 to your proposal, only i'm not sure if you are still
thinking to
pass a TestCaseInstance to the @DataProvider -ed method.

Apart from that, DataProvider already have all the necessary
functionality.

Alexander:

>> you need a point to reference the Java code. It looks to me much like creating a new >> language.

Interesting remark. If possible, can you explicit it a little further.

Cedric:

>> Mmmh, I confess I'm a bit lost. Is "methodname" a real Java method? If it

No, i doesn't have to be. I just have to match the name attribute of a
method
element in the JTestCase xml defin. file, like <method
name="methodname">
In Java, you retrieve the params and asserts with the call
TestCaseInstance[] testcases =
JTestCase.getTestCasesForMethod("methodname")
anywhere in your code.

>>Also, how does getTestCasesInMethod() find test cases inside a method? (and
>>what does that even mean? test cases *inside* a method?)

We refer to a test case as a pair of <param> and <assert> group.
For example, Calculator.makeOR(int var1, int var2)
may have this definition
<method name="notexactlymakeORmethodnamebuttheentrypoint">
<test-case name="or_0_1">
<param name="var1">0</param>
<param name="var2">1</param>
<assert name="result">1</assert>
</test-case>
<test-case name="or_1_0">
<param name="var1">1</param>
<param name="var2">0</param>
<assert name="result">1</assert>
</test-case>

ect.

>> probably not include a JTestCase patch into the TestNG core.

Of course, no need changing the core.

>>Which brings me to the other subject: offering an API for people to add
>>annotations (and their processing) inside TestNG, which is a very hard
>>problem because it's not really clear where the extension points in TestNG
>>should be...

To do that, i think you need a annotation registry that people can
augment
(here is a reference to a type safe registry implementation i
particularly
like
http://www.onjava.com/pub/a/onjava/2004/12/22/towardsbugfree.html)
and a hook to execute the externally provided implemenation to the
Invoker
perhaps in the same place where you have the dataprovider hook.

Invoker.java

lines 469-484

// Do we have a @DataProvider? If yes, then we have several
// sets of parameters for this method
//
Method dataProvider =
Parameters.findDataProvider(testMethod.getMethod(),
m_annotationFinder);

if (null != dataProvider) {
int parameterCount =
testMethod.getMethod().getParameterTypes().length;
// Map[] allParameters = new Map[parameterCount];
// String[] parameterNames = new String[parameterCount];
for (int i = 0; i < parameterCount; i++) {
String n = "param" + i;
allParameterNames.put(n, n);
}
// allParameterNames = parameterNames;
allParameterValues = MethodHelper.invokeDataProvider(dataProvider);

Just a thought.

Fausto

[Jörg Gellien]

Hi Fausto,
first of all surely no offence taken.
My intention for this thread is to find a way to incorporate queryable
data repositories to TestNG. And of cause DDTUnit and JTestCase are
quite near in idea and representation.

>Then I'll +1 to your proposal, only i'm not sure if you are still thinking to
>pass a TestCaseInstance to the @DataProvider -ed method.

Thanks for the vote.
If you like to use the asserts from JTestCase I would provide the
TestCaseInstance, if you only want to use the param objects I would
provide them through @DataProvider to the @Test method signature. - The
loop will be done in @DataProvider method.

>>one thing that disturbes me is that you provide kind of your own
>>execution model on top of TestNG without an integration to it.
>
>Nope. That's exactly the point. I just let TestNG tell me WHAT test to
>do and JTestCase
>do its work of abstracting the test case . They're "orthogonal", in
>some sense.

Just a remark on that one. - You are right when stating that TestNG
will select the methods to execute.
What TestNG or its test runner and reporter is not aware of is the
explicite loop with its own exception handling. And so it is missing
some information which is relevant to be consistent.

but please do not missinterprete - we are sitting in the same boat and
heading in the same direction ;-) ->surly I take up the thread with our
discussion. on DDTUnit-JTestCase.

best regards to all
and a merry christmas

Jörg

[faustothegrey]

Hi Joerg.

>> first of all surely no offence taken.

Ok thanks.

>> heading in the same direction ;-) ->surly I take up the thread with our
>> discussion. on DDTUnit-JTestCase

Looking forward to that !

>>What TestNG or its test runner and reporter is not aware of is the
>>explicite loop with its own exception handling. And so it is missing
>>some information which is relevant to be consistent.

Well, in the example I wrote some posts ago I did see TestNG
report JTestCase asertion failures. JTestCase reports assertion
failures
using an utility method getFailureReason(). I was amused to see that
changing JTestCase xml files I can make TestNG report green or red
and the correct advice. A screenshot would be more impressive.

The only possible problem was about loop breaking in the first
assertion
failure, which is JTestCase problem, not a TestNG -JTestCase
integration
mismatch. In the contrary, using DataProvider that loop over
TestCaseInstances
would solve jtestcase problem !

To express my point even more clearly, to me it looks that the
following
snippet of code are, in the TestNG view, completely equivalent, because
the
two are making, at the end of the day, just an assert:

@Parameters({ "first-name" })
@Test
public void testSingleString(String firstName) {

System.out.println("Invoked testString " + firstName);

/**********************
* An assert *
**********************/
assert "Cedric".equals(firstName);
}

public void testCedric(String firstName) {

Vector testCases = _jtestcase
.getTestCasesInstancesInMethod("testCedricMethod");

// for each test case
System.out.println("Number of test case:: " + testCases.size());
for (int i = 0; i < testCases.size(); i++) {

// retrieve name of test case
TestCaseInstance testCase = (TestCaseInstance) testCases
.elementAt(i);

String firstName = testCase.getParameter("firstname");

firstName = firstName.toLowerCase();

/**********************
* Just another assert *
**********************/
assert testCase.assertTestVariable("result",
firstName) : testCase.getFailureReason() ;
}
}

<method name="testCedricMethod">
<test-case>
<param name="firstname">Cedric</param>
<assert name="result">cedric</assert>
</test-case>
</method>

The difference between TestNG and JTestCase being that while TestNG is
a
framework for unit testing, JTestCase is a tool, or language, or
abstraction for
externalizing a test case.

Merry Christmas to all and thanks for the nice discussions.

Fausto