Wrong index.html in testng reports

[Bogdan Calmac]

As some of you might remember from a thread a few of month ago (http://forums.opensymphony.com/thread.jspa?messageID=73140) I am using testng without a testng.xml configuration file, but rather by using the testSuite and testName attributes of @Test.

I don't know if it's related to this pattern of use, but the index page of the HTML reports seems to be cumulative, so instead of:

suite 1 (1/0/0)
suite 2 (1/1/0)
suite 3 (42/0/0)

it looks like:

suite 1 (1/0/0)
suite 2 (2/1/0) // 1/0/0 added from above, should be 1/1/0
suite 3 (44/1/0) // 2/1/0 added from above, should be 42/0/0

It looks like there's a counter which is not reset when advancing to a new suite. See the attached reports for the details. On the other hand, the report page for each individual suite is fine; only the top-level index page is wrong.

I've tested this on 5.2 and HEAD and the problem is the same. For reference, I define the suite and name with:

@Test(suiteName="com.knoa.commons.sql", testName="DataSourceTest")
public class DataSourceTest

and the testng invocation from ant is:

<testng dumpCommand="${testngDumpCommand}"
workingDir="${targetTest}"
outputDir="${testngDir}"
verbose="${testngVerbose}"
haltOnFailure="${testngHaltOnFailure}"
failureProperty="${testngFailureProperty}">
<classpath>
<pathelement location="${targetTestClasses}"/>
<pathelement location="${targetClasses}"/>
<path refid="path.testLib"/>
</classpath>
<jvmarg value="${testngDebugArg}"/>
<jvmarg value="-Xmx${testngMaxMemory}"/>
<classfileset dir="${targetTestClasses}">
<patternset refid="testngClasses"/>
</classfileset>
</testng>

thanks,

Bogdan Calmac
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=92922#92922

[Alexandru Popescu]

On 10/12/06, Bogdan Calmac <testng...@opensymphony.com> wrote:
> As some of you might remember from a thread a few of month ago (http://forums.opensymphony.com/thread.jspa?messageID=73140) I am using testng without a testng.xml configuration file, but rather by using the testSuite and testName attributes of @Test.
>
> I don't know if it's related to this pattern of use, but the index page of the HTML reports seems to be cumulative, so instead of:
>
> suite 1 (1/0/0)
> suite 2 (1/1/0)
> suite 3 (42/0/0)
>
> it looks like:
>
> suite 1 (1/0/0)
> suite 2 (2/1/0) // 1/0/0 added from above, should be 1/1/0
> suite 3 (44/1/0) // 2/1/0 added from above, should be 42/0/0
>
> It looks like there's a counter which is not reset when advancing to a new suite. See the attached reports for the details. On the other hand, the report page for each individual suite is fine; only the top-level index page is wrong.
>

This is the expected behavior.

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

[Bogdan Calmac]

Hmm, ok, let's take an example. Say there are 100 suites and suites 1 and 78 have an error. In the index page all of them will be red (because there was an error in 1) and then you will have to carefully scroll until 78 to find the other error. Do you think this is an intuitive presentation?

just my 2c,

Bogdan.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=92937#92937

[Alexandru Popescu]

On 10/12/06, Bogdan Calmac <testng...@opensymphony.com> wrote:
>

> Hmm, ok, let's take an example. Say there are 100 suites and suites 1 and 78 have an error. In the index page all of them will be red (because there was an error in 1) and then you will have to carefully scroll until 78 to find the other error. Do you think this is an intuitive presentation?
>

I guess you may be right from this perspective, but once you decided
to run all of them together the result is passed only if all have
passed. Probably a special marker for the ones containing
failures/skipped would make sense.

Also, I haven't seen yet somebody with 100 suites :-).

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

[Cédric Beust ♔]

On 10/11/06, Bogdan Calmac <testng...@opensymphony.com> wrote:

Hmm, ok, let's take an example. Say there are 100 suites and suites 1 and 78 have an error. In the index page all of them will be red (because there was an error in 1) and then you will have to carefully scroll until 78 to find the other error. Do you think this is an intuitive presentation?

Fair point...  Give us some time and Alexandru and I will think about this a bit more...

--
Cédric

[Alexandru Popescu]

I was thinking about 2 possible approaches (but I haven't checked yet
the code to see if they are possible). Here they are:

1/ list each suite with its result color, its own counters and on top
of them include a summary with the total counters and the color of the
overall result

2/ list each suite with the overall result, but put special markers
for those having failures/skipped; each suite is listing its own
counters and on top of them include a summary

What do you guys think about these?

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

[Bogdan Calmac]

I would go with 1. It is both more flexible and intuitive.

Otherwise, the current implementation and 2. assume that the suites are dependent on each other, which is not always the case. I mean, if 1 fails and all the others are considered failed, then we could as well just not run them at all.

cheers,

bogdan.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=93091#93091

[BK Lau]

> I was thinking about 2 possible approaches (but I
> haven't checked yet
> the code to see if they are possible). Here they are:
>
> 1/ list each suite with its result color, its own
> counters and on top
> of them include a summary with the total counters and
> the color of the
> overall result
>

Alex:

I brought this same issue some months ago.
I agreed with Bogdan that it is confusing i.e. not intuitive.
Choice#1 is probably the best.

> 2/ list each suite with the overall result, but put
> special markers
> for those having failures/skipped; each suite is
> listing its own
> counters and on top of them include a summary
>
> What do you guys think about these?
>
> ./alex
> --
> .w( the_mindstorm )p.
> TestNG co-founder

> lipseTestNG Creator
>
> On 10/12/06, Cédric Beust ♔ <cbe...@google.com>

> wrote:
> >
> >
> > On 10/11/06, Bogdan Calmac
> <testng...@opensymphony.com> wrote:
> > >
> > > Hmm, ok, let's take an example. Say there are 100
> suites and suites 1 and
> > 78 have an error. In the index page all of them
> will be red (because there
> > was an error in 1) and then you will have to
> carefully scroll until 78 to
> > find the other error. Do you think this is an
> intuitive presentation?
> >
> > Fair point... Give us some time and Alexandru and
> I will think about this a
> > bit more...
> >
> > --

> > Cédric

> >
> > >
> >
>
> >
---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=93145#93145

[Alexandru Popescu]

On 10/12/06, Bogdan Calmac <testng...@opensymphony.com> wrote:
>

> I would go with 1. It is both more flexible and intuitive.
>

Oke... I will give it a shot and see if I can get it working.

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

> Otherwise, the current implementation and 2. assume that the suites are dependent on each other, which is not always the case. I mean, if 1 fails and all the others are considered failed, then we could as well just not run them at all.

[Alexandru Popescu]

The current SVN trunk should contain an improved SuiteHTMLReporter
that would work as described in my suggestion 1. Can you please give
it a try and let me know if it works better than the previous?

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

[Bogdan Calmac]

Yep, it works according to solution 1. above, first a summary line and then individual lines for each suite.

Thanks guys.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=93927#93927