[testng-users] HTML Report, JUnit Report

[Prakash]

Hello all,

I am a new user to TestNG.

I created a sample test suite that calls few test classes like this

public static void main(String[] args) {
TestListenerAdapter tla = new TestListenerAdapter();
TestNG testng = new TestNG();
testng.setTestClasses(new Class[] { CarTest.class,
CustomerTest.class });
testng.addListener(tla);
testng.run();
}

CarTest.class has @Dataprovider defined and everything works fine.

@Test(dataProvider = "test-data")
public void testWithParameters(String make, String model, int year,
String expected, boolean result)
{
Car car = new Car(make, model, year);

if (result)
Assert.assertEquals(car.toString(), expected);
else
Assert.assertFalse(car.toString().equals(expected));
}

@DataProvider(name = "test-data" )
public Object[] [ ] rangeData( )
{
return new Object[ ] [ ] {
{ "Honda", "Accord", 2010, "Honda Accord 2010", true},
{ "Honda", "Accord", 2010, "Honda Accord 2110", true},
{ "Toyota", "Camry", 2010, "Toyota Camry 2010", true},
{ "Toyota", "Camry", 2010, "Toyota Camri 2010", true},
{ "BMW", "325", 2010, "BMW 325 2010", true},
};
}

When all iterations of data provided test passes, the HTML report
displays the complete details i.e. all five assertion details.
When i make one of the iteration's assertion fail, the HTML report
displays only the two failed assertions and one of the passed
assertion. I don't know why two other passed assertion detail is not
displayed in the final report. Can someone help me how to get all
assertion details.

I have one more question, how can i get a JUnit style report when i am
running the tests from Java code like above and not using testNG.xml.

Thanks
Prakash

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[prasanna bhat]

Hi Prakash,

You might want to take a look at this plugin... https://reportng.dev.java.net/

For JUnit style reports refer (http://testng.org/doc/documentation-main.html#logging-junitreports) or you get many ant tasks if you google for it.

Thanks,
Prasanna

[Cédric Beust ♔]

Hi Prakash,

On Sun, May 2, 2010 at 8:39 AM, Prakash <rushto...@gmail.com> wrote:

When all iterations of data provided test passes, the HTML report
displays the complete details i.e. all five assertion details.
When i make one of the iteration's assertion fail, the HTML report
displays only the two failed assertions and one of the passed
assertion. I don't know why two other passed assertion detail is not
displayed in the final report. Can someone help me how to get all
assertion details.

This should not be happening:  if an invocation of a data provider fails, it should be reported as such and all the others should be successes.  I wrote a quick test to confirm:

PASSED: ctest1(42)

PASSED: ctest1(43)

PASSED: ctest1(45)

PASSED: ctest1(46)

FAILED: ctest1(44)     <-- I failed that one

I also verified that the HTML report shows this result as well.

Can you send me the code of your test class showing the bug?

 

I have one more question, how can i get a JUnit style report when i am
running the tests from Java code like above and not using testNG.xml.

TestNG generates an XML file that's compatible with JUnitReports, take a look at this section of the documentation.

Hope this helps.

--
Cédric

[Prakash]

Thanks for the reply Prasanna,

I downloaded reportng and i got JUnit style reports working.

but i still don't get to see all the successful parameterized
assertions/methods displayed in the report output even when using
JUnit style report.
All successful parameterized assertions appear on default index.html
output, but not in default emailable-report.html and JUnit styled
report.

Anyone else had this issue and went past this?

On May 2, 12:03 pm, prasanna bhat <prasannabha...@gmail.com> wrote:
> Hi Prakash,
>

> You might want to take a look at this plugin...https://reportng.dev.java.net/

> > testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>
> > .

> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to testng-users...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/testng-users?hl=en.

[Prakash]

Thank you Cedric, here is my code,

Car.java

public class Car
{
private String make;
private String model;
private int year;

public Car(String make, String model, int year)
{
this.make = make;
this.model = model;
this.year = year;
}

public String toString()
{
return make + " " + model + " " + year;
}
}

CarTest.java

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class CarTest
{
@Test(groups = { "functest" })
public void testToStringPositive()
{
Car car = new Car("Honda", "Accord", 2010);

Assert.assertEquals(car.toString(), "Honda Accord 2010");

}

@Test(groups = { "functest" })
public void testToStringNegative()
{
Car car = new Car("Honda", "Accord", 2010);

Assert.assertFalse(car.toString().equals("Honda Accord 2011"));

}

@Test(dataProvider = "test-data")
public void testWithParameters(String make, String model, int year,
String expected, boolean result)
{
Car car = new Car(make, model, year);

if (result)
Assert.assertEquals(car.toString(), expected);
else
Assert.assertFalse(car.toString().equals(expected));
}

@DataProvider(name = "test-data" )
public Object[] [ ] rangeData( )
{
return new Object[ ] [ ] {
{ "Honda", "Accord", 2010, "Honda Accord 2010", true},
{ "Honda", "Accord", 2010, "Honda Accord 2110", true},
{ "Toyota", "Camry", 2010, "Toyota Camry 2010", true},
{ "Toyota", "Camry", 2010, "Toyota Camri 2010", true},
{ "BMW", "325", 2010, "BMW 325 2010", true},
};
}
}

Controller or Main class

import org.testng.TestNG;
import org.testng.reporters.JUnitXMLReporter;
import org.uncommons.reportng.HTMLReporter;

public class SampleTestSuite
{

public static void main(String[] args)
{

HTMLReporter rep = new HTMLReporter();
JUnitXMLReporter jxr2 = new JUnitXMLReporter();

TestNG testng = new TestNG();
testng.setTestClasses(new Class[] { CarTest.class,
CustomerTest.class });

testng.addListener(rep);
testng.addListener(jxr2);
testng.run();
}
}

Please let me know if i have went wrong somewhere.

Thanks
Prakash

On May 2, 6:20 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Hi Prakash,
>

> On Sun, May 2, 2010 at 8:39 AM, Prakash <rushtoprak...@gmail.com> wrote:
>
> > When all iterations of data provided test passes, the HTML report
> > displays the complete details i.e. all five assertion details.
> > When i make one of the iteration's assertion fail, the HTML report
> > displays only the two failed assertions and one of the passed
> > assertion. I don't know why two other passed assertion detail is not
> > displayed in the final report. Can someone help me how to get all
> > assertion details.
>
> This should not be happening:  if an invocation of a data provider fails, it
> should be reported as such and all the others should be successes.  I wrote
> a quick test to confirm:
>
> PASSED: ctest1(42)
> PASSED: ctest1(43)
> PASSED: ctest1(45)
> PASSED: ctest1(46)
> FAILED: ctest1(44)     <-- I failed that one
>
> I also verified that the HTML report shows this result as well.
>
> Can you send me the code of your test class showing the bug?
>
>
>
> > I have one more question, how can i get a JUnit style report when i am
> > running the tests from Java code like above and not using testNG.xml.
>
> TestNG generates an XML file that's compatible with JUnitReports, take a
> look at this section of the

> documentation<http://testng.org/doc/documentation-main.html#logging-junitreports>
> .

[Cédric Beust ♔]

Prakash,

JUnitReports doesn't support parameterized tests (because JUnit 3 doesn't support it either), so you will only get a subset of the results there.

If you need the full report, you should either use the default HTML reports or the excellent ReportNG, as Prasanna pointed out.

-- 

Cédric

--
Cédric

[Cédric Beust ♔]

Prakash,

I tried your code and I think I'm getting the correct report:  2 passed, 3 failed.  I'm attaching the HTML.

Are you seeing something different?

-- 

Cedric

[Prakash]

Exactly Cedric, I am getting same results in default index.html, but
not in emailable-report.html.

I do not have option to send attachments.

> > testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>
> > .

> > > For more options, visit this group athttp://
> > groups.google.com/group/testng-users?hl=en.
>

> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>
> > .

> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.
>
> --
> Cédric
>
> --
> You received this message because you are subscribed to the Google Groups "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to testng-users...@googlegroups.com.
> For more options, visit this group athttp://groups.google.com/group/testng-users?hl=en.
>

>  report.html
> 10KViewDownload

[Cédric Beust ♔]

I'm seeing the correct results in emailable-report.html as well (see attachment).

-- 

Cédric

[Prakash]

Cedric,

The method testWithParameters was run 5 times(5 rows of data), the
attached report only accounts/shows for three scenarios ( 2 failed, 1
passed), it is missing 2 more passed scenario.

Expected->Actual

2 failed
Camry->Camri
2010->2110

3 passed
Camry->Camry
Accord->Accord
BMW->BMW

The report shows only one scenario passed (BMW), the other two
scenarios that passed for Camry and Accord are missing.

Please let me know if i am missing to see something here.

> > >  report.html
> > > 10KViewDownload
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.
>
> --
> Cédric
>
> --
> You received this message because you are subscribed to the Google Groups "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to testng-users...@googlegroups.com.
> For more options, visit this group athttp://groups.google.com/group/testng-users?hl=en.
>

>  Picture 2.png
> 63KViewDownload

[prasanna bhat]

Hi Cedric,

Unfortunately even i'm facing the same issue now. I executed the sample code as posted by Prakash and i found that  the number of tests executed doesn't match the total number of tests run and  the number of test methods passed  is incorrect but the number of tests failed is correct.  (By the way i was using reportng to generate reports).

I wrote a sample class containing a test method and a DataProvider method and i was successfully able to reproduce the issue. But i have also noticed that when all the test methods pass this issue is not seen.

I checked the testng-results.xml file and it was clear that the  <test-method> entries were missing for some of the executed test methods and hence they are not seen in the generated reports.

And also if you notice the screen shot of the emailabe-report you have attached in the previous update the testWithParameters method is shown as executed only once where as it got executed thrice.

Can you please throw more light on this??


Thanks,
Prasanna

[prasanna bhat]

To make it more clear i'm attaching the report generated by reportNG.

And the TestSuite.txt file contains the following content:

-------------------------------------------------------------------------------
Test set: TestSuite
-------------------------------------------------------------------------------
Tests run: 10, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 1.14 sec <<< FAILURE!
testIsBetween(deshaw.automation.bo.TestDataProviderExample)  Time elapsed: 0.015 sec  <<< FAILURE!
java.lang.AssertionError: expected:<false> but was:<true>
    at org.testng.Assert.fail(Assert.java:87)
    at org.testng.Assert.failNotEquals(Assert.java:441)
    at org.testng.Assert.assertEquals(Assert.java:111)
    at org.testng.Assert.assertEquals(Assert.java:242)
    at org.testng.Assert.assertEquals(Assert.java:252)

testIsBetween(deshaw.automation.bo.TestDataProviderExample)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: expected:<false> but was:<true>
    at org.testng.Assert.fail(Assert.java:87)
    at org.testng.Assert.failNotEquals(Assert.java:441)
    at org.testng.Assert.assertEquals(Assert.java:111)
    at org.testng.Assert.assertEquals(Assert.java:242)
    at org.testng.Assert.assertEquals(Assert.java:252)

Thanks,
Prasanna

[Cédric Beust ♔]

Prasanna,

ReportNG is not part of the TestNG distribution, you should report problems to Dan Dyer.

However, if testng-results.xml contains incorrect information, then this is a TestNG bug.  Can you email me (privately if it's too big or sensitive) a test cast that allows me to reproduce this bug?  The example with the five cars seems to work for me...

Thanks.

-- 

Cédric

[prasanna bhat]

Thanks for your reply Cedric. I have sent you a private mail (ced...@beust.com) with testng-results.xml and TestSuite.txt files attached to it.

i have sent you two pairs of files for two executions:
1)  For the same piece of code which Prakash had posted in the previous updates.
2)  I have executed my own class with a test method and data provider as given below.

    @DataProvider(name="range-provider")
    public static Object[][] rangeData() {
        int lower=5;
        int upper=10;
        return new Object[][] {
                {lower-1, lower, upper, false},
                {lower+1, lower, upper, true},
                {lower-2, lower, upper, false},
                {lower+2, lower, upper, true},
                {lower-3, lower, upper, false},
                {lower+3, lower, upper, true},
                {lower-4, lower, upper, true}, //error case
                {lower+4, lower, upper, true},
                {lower-5, lower, upper, true}, //error case
                {lower+5, lower, upper, true}
               
        };
    }
   
    private boolean isBetween(int n, int lower, int upper){           
        if(n >= lower && n<=upper)
            return true;

        return false;
    }
   
    @Test(enabled=true, dataProvider="range-provider")
    public void testIsBetween(int n, int lower, int upper, boolean expected) {
        System.out.println("Recieved "+n+" "+lower+"-"+upper+"Expected: "+expected);

        Assert.assertEquals(expected, isBetween(n, lower, upper));
    }

Btw, I'm using maven-surefire-plugin version 2.4.2 and TestNG version 5.11.

Thanks,
Prasanna

2010/5/4 Cédric Beust ♔ <ced...@beust.com>

[prasanna bhat]

Hi Cedric,

Were you able to reproduce it??

Thanks,
Prasanna

[Prakash]

Hello

Do we have any update on this issue?

Thanks
Prakash

On May 5, 10:11 am, prasanna bhat <prasannabha...@gmail.com> wrote:
> Hi Cedric,
>
> Were you able to reproduce it??
>
> Thanks,
> Prasanna
>

> >>> On Tue, May 4, 2010 at 8:54 PM, prasanna bhat <prasannabha...@gmail.com>wrote:
>
> >>>> Hi Cedric,
>
> >>>> Unfortunately even i'm facing the same issue now. I executed the sample
> >>>> code as posted by Prakash and i found that  the number of tests executed
> >>>> doesn't match the total number of tests run and  the number of test methods

> >>>> passed  is incorrect but the number of tests failed is correct.  (*By
> >>>> the way i was using reportng to generate reports*).

>
> >>>> I wrote a sample class containing a test method and a DataProvider
> >>>> method and i was successfully able to reproduce the issue. But i have also
> >>>> noticed that when all the test methods pass this issue is not seen.
>

> >>>> I checked the *testng-results.xml* file and it was clear that the
> >>>> <test-method> entries were *missing* for some of the executed test

> >>>> methods and hence they are not seen in the generated reports.
>
> >>>> And also if you notice the screen shot of the emailabe-report you have

> >>>> attached in the previous update the *testWithParameters* method is

> >>>> shown as executed only once where as it got executed thrice.
>
> >>>> Can you please throw more light on this??
>
> >>>> Thanks,
> >>>> Prasanna
>

> ...
>
> read more »

[Cédric Beust ♔]

Sorry Prakash, I haven't had the time to look into this.

-- 

Cédric

--
Cédric