I don't know if it's related to this pattern of use, but the index page of the HTML reports seems to be cumulative, so instead of:
suite 1 (1/0/0)
suite 2 (1/1/0)
suite 3 (42/0/0)
it looks like:
suite 1 (1/0/0)
suite 2 (2/1/0) // 1/0/0 added from above, should be 1/1/0
suite 3 (44/1/0) // 2/1/0 added from above, should be 42/0/0
It looks like there's a counter which is not reset when advancing to a new suite. See the attached reports for the details. On the other hand, the report page for each individual suite is fine; only the top-level index page is wrong.
I've tested this on 5.2 and HEAD and the problem is the same. For reference, I define the suite and name with:
@Test(suiteName="com.knoa.commons.sql", testName="DataSourceTest")
public class DataSourceTest
and the testng invocation from ant is:
<testng dumpCommand="${testngDumpCommand}"
workingDir="${targetTest}"
outputDir="${testngDir}"
verbose="${testngVerbose}"
haltOnFailure="${testngHaltOnFailure}"
failureProperty="${testngFailureProperty}">
<classpath>
<pathelement location="${targetTestClasses}"/>
<pathelement location="${targetClasses}"/>
<path refid="path.testLib"/>
</classpath>
<jvmarg value="${testngDebugArg}"/>
<jvmarg value="-Xmx${testngMaxMemory}"/>
<classfileset dir="${targetTestClasses}">
<patternset refid="testngClasses"/>
</classfileset>
</testng>
thanks,
Bogdan Calmac
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=92922#92922
This is the expected behavior.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
just my 2c,
Bogdan.
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=92937#92937
I guess you may be right from this perspective, but once you decided
to run all of them together the result is passed only if all have
passed. Probably a special marker for the ones containing
failures/skipped would make sense.
Also, I haven't seen yet somebody with 100 suites :-).
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
Hmm, ok, let's take an example. Say there are 100 suites and suites 1 and 78 have an error. In the index page all of them will be red (because there was an error in 1) and then you will have to carefully scroll until 78 to find the other error. Do you think this is an intuitive presentation?
1/ list each suite with its result color, its own counters and on top
of them include a summary with the total counters and the color of the
overall result
2/ list each suite with the overall result, but put special markers
for those having failures/skipped; each suite is listing its own
counters and on top of them include a summary
What do you guys think about these?
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
Otherwise, the current implementation and 2. assume that the suites are dependent on each other, which is not always the case. I mean, if 1 fails and all the others are considered failed, then we could as well just not run them at all.
cheers,
bogdan.
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=93091#93091
I brought this same issue some months ago.
I agreed with Bogdan that it is confusing i.e. not intuitive.
Choice#1 is probably the best.
> 2/ list each suite with the overall result, but put
> special markers
> for those having failures/skipped; each suite is
> listing its own
> counters and on top of them include a summary
>
> What do you guys think about these?
>
> ./alex
> --
> .w( the_mindstorm )p.
> TestNG co-founder
> lipseTestNG Creator
>
> On 10/12/06, Cédric Beust ♔ <cbe...@google.com>
> wrote:
> >
> >
> > On 10/11/06, Bogdan Calmac
> <testng...@opensymphony.com> wrote:
> > >
> > > Hmm, ok, let's take an example. Say there are 100
> suites and suites 1 and
> > 78 have an error. In the index page all of them
> will be red (because there
> > was an error in 1) and then you will have to
> carefully scroll until 78 to
> > find the other error. Do you think this is an
> intuitive presentation?
> >
> > Fair point... Give us some time and Alexandru and
> I will think about this a
> > bit more...
> >
> > --
> > Cédric
> >
> > >
> >
>
> >
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=93145#93145
Oke... I will give it a shot and see if I can get it working.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
> Otherwise, the current implementation and 2. assume that the suites are dependent on each other, which is not always the case. I mean, if 1 fails and all the others are considered failed, then we could as well just not run them at all.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
Thanks guys.
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=45951&messageID=93927#93927