TestNG "haltonfailure" Ant task
BK Lau
The TestNG "haltonfailure" Ant property is defined in the TestNG doc as
"Stop the build process if a failure has occurred during the test run. No. Defaults to false ".
I got to play with it a bit and noticed that it throws an exception if at least one of the tests failed. This assumes of course that the test got to to run in the first place... BUT this might not even get to the point of actually running the test.
I have a different concern: Not of any tests failed , but if testng failed to run for some reasons:
For example, if some erroreous parameters are passed to TestNG or testng couldn't find some classes in the classpath, then the testng Ant task would failed.
However,I would like it to failed gracefully instead of failing the Ant Build script the TestNG Ant task is in.
This is to allow the Ant Build script to call another Ant target or Ant script to send out an email notification. If the Ant Build script the TestNG Ant task is in just failed, then an exception is thrown and the Build failed in the current TestNG 5.3
So, if it not too much to ask if another "haltonerror" property and its associated "errorproperty" can be implemented for this purposein next TestNG5.4 release?.
-BK-
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=50574&messageID=101719#101719
Alexandru Popescu
>
> Hi :
> The TestNG "haltonfailure" Ant property is defined in the TestNG doc as
> "Stop the build process if a failure has occurred during the test run. No. Defaults to false ".
>
> I got to play with it a bit and noticed that it throws an exception if at least one of the tests failed. This assumes of course that the test got to to run in the first place... BUT this might not even get to the point of actually running the test.
>
> I have a different concern: Not of any tests failed , but if testng failed to run for some reasons:
> For example, if some erroreous parameters are passed to TestNG or testng couldn't find some classes in the classpath, then the testng Ant task would failed.
> However,I would like it to failed gracefully instead of failing the Ant Build script the TestNG Ant task is in.
> This is to allow the Ant Build script to call another Ant target or Ant script to send out an email notification. If the Ant Build script the TestNG Ant task is in just failed, then an exception is thrown and the Build failed in the current TestNG 5.3
>
I am not quite sure I understand. haltonfailure must end the build
execution. In case you want to continue with something else even if an
failure occured use the corresponding property and trigger the email
trigger task.
<target name="test>
<testng ... failureProperty="test.failure" />
<antcall target="failure.email" />
</target>
<target name="failure.email" if="test.failure">
<!-- send email -->
</target>
> So, if it not too much to ask if another "haltonerror" property and its associated "errorproperty" can be implemented for this purposein next TestNG5.4 release?.
>
So, far we have avoided to re-introduce these 2 terms, used quite
frequently in JUnit and that almost always got me confused.
Considering the above approach I don't think this is really needed.
HTH,
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
BenLau
Alexandru Popescu wrote:
> On 11/15/06, BK Lau <testng...@opensymphony.com> wrote:
> >
> > Hi :
> > The TestNG "haltonfailure" Ant property is defined in the TestNG doc as
> > "Stop the build process if a failure has occurred during the test run. No. Defaults to false ".
> >
> > I got to play with it a bit and noticed that it throws an exception if at least one of the tests failed. This assumes of course that the test got to to run in the first place... BUT this might not even get to the point of actually running the test.
> >
> > I have a different concern: Not of any tests failed , but if testng failed to run for some reasons:
> > For example, if some erroreous parameters are passed to TestNG or testng couldn't find some classes in the classpath, then the testng Ant task would failed.
> > However,I would like it to failed gracefully instead of failing the Ant Build script the TestNG Ant task is in.
>
> > This is to allow the Ant Build script to call another Ant target or Ant script to send out an email notification. If the Ant Build script the TestNG Ant task is in just failed, then an exception is thrown and the Build failed in the current TestNG 5.3
> >
>
> I am not quite sure I understand. haltonfailure must end the build
> execution. In case you want to continue with something else even if an
> failure occured use the corresponding property and trigger the email
> trigger task.
>
> <target name="test>
> <testng ... failureProperty="test.failure" />
> <antcall target="failure.email" />
> </target>
>
> <target name="failure.email" if="test.failure">
> <!-- send email -->
> </target>
>
Alex:
Let me explian more clearly:
The above code will only work if and only if TestNG runner itself does
encountered any errors during its initialization phase.
>From my perspective ,at least with JUnit tests running under TestNG,
is that haltonfailure="true" failureProperty="test.failure"
intended purpose is to stop the tests itself, not the test
infrastructure( i.e. TestNG runner) that runs the tests.
There is a distinction here. The tests failures vs test infrastructure
failures.
Tests failures are fairly normal in regular devs cycle, as we should
expect; devs want to bail out of further running tests in order to fix
a failed test first.
In a fully automated suite with thousands of tests, with high setup
cost and time, typically, we don't want to do that. We picked thru
failed tests later on.
But test infrastructure failures are abnormal since it means the the
conditions for the TestNG runner has not been met and that we can't
even attempt to run
a test.
In short failures due to TestNG runner itself directly/indirectly vs
failures in the tests themselves are separate issues.
Hope this is clear.
Steve Loughran
>
> In short failures due to TestNG runner itself directly/indirectly vs
> failures in the tests themselves are separate issues.
> Hope this is clear.
>
If you want to catch BuildExceptions that get raised in a build,
Ant-contribs <trycatch> task can be used to catch them. If you are
feeling very leading edge, my own <functionaltest> task is available
for use
it runs activities in one threads, blocks till a probe says that
activity is live, runs your tests, calls a teardown sequence in
another thread, and has a finally cleanup. It also handles timeouts in
the test run or in teardown. Its not a test runner, just a container
for anything
<sf-functionaltest testTimeout="600" shutdownTime="10">
<application>
<sf-startdaemon-debug failonerror="false" spawn="false"/>
</application>
<probe>
<socket port="${smartfrog.daemon.port}" server="localhost"/>
</probe>
<test>
<sf-junit
includeantruntime="true"
filtertrace="${filter.trace}"
errorproperty="failure"
failureproperty="failure"
>
<classpath
refid="tests.run.classpath"/>
<sysproperty key="test.files.dir"
value="${test.src.dir}/files"/>
<sysproperty key="endpoint"
value="http://127.0.0.1:6060/alpine"/>
<syspropertyset>
<propertyref prefix="runtime"/>
</syspropertyset>
<batchtest todir="${test.data.dir}">
<fileset dir="${test.classes.dir}">
<include name="**/unit/*Test.class"/>
<include name="**/system/**/*Test.class"/>
</fileset>
</batchtest>
</sf-junit>
<fail if="failure">Junit failed</fail>
</test>
<teardown>
<sf-stopdaemon failonerror="false"/>
<sf-junitreport data="${test.data.dir}"
reports="${test.reports.dir}"
/>
</teardown>
</sf-functionaltest>
Alexandru Popescu
Nope it is not so clear. Can you show me an example pls?
BenLau
Alex: Here's an example illustrated:
TestNG runner
initialization phase | Tests running phase
|<------------------------------------>|<-----------------------------------------------------------..........->|
My sense is that haltonfailure is intended to catch test failures
during the "Tests running phase".
At my company, we asked devs to put their tests in a "test repository"
and then our framework based on
TestNG runner will pull in each testng.xml that devs defined. We have
no knowledge if those folks defined it correctly or not. For example,
did the classes that contains the tests got specified correctly?. If
so, are they in the classpath?, etc. You got the idea- we run unit
tests on massive scale; devs just conttribute their part into this big
tests repository.
We don't stop for a test failure; we just send out a report and
identify the dev who owns the test and they fixed it and resubmit their
tests and/or testng.xml again.
So as you can see, exceptions can be thrown from 2 phases above.
The exception thrown from the "Tests running phase" is less severe to
us; it is the phase "TestNG runner initilization"
that we in the test infrastructure team have to deal with.
I want to be able to differentiate between the 2.
-BK-
BenLau
Steve Loughran wrote:
> On 15/11/06, BenLau <bkla...@gmail.com> wrote:
>
> >
> > In short failures due to TestNG runner itself directly/indirectly vs
> > failures in the tests themselves are separate issues.
> > Hope this is clear.
> >
>
> If you want to catch BuildExceptions that get raised in a build,
> Ant-contribs <trycatch> task can be used to catch them. If you are
> feeling very leading edge, my own <functionaltest> task is available
> for use
Steve:
Thanks for your suggestions.
The problem with current "haltonfailures=true/false" is that it
catches both tests and non-tests related exceptions in one bucket.
So, try catch won't help here.
BenLau
In short, I need :
1) haltonfailures to control failures originated from running tests.
2) haltonerrors to control any other errors/exception not originating
from tests themselves.
So what do you think?.
Thanks,
BK
Alexandru Popescu
>
> So, Alex:
>
> In short, I need :
>
> 1) haltonfailures to control failures originated from running tests.
> 2) haltonerrors to control any other errors/exception not originating
> from tests themselves.
>
> So what do you think?.
>
Exactly what I was saying a couple of emails before:
[quote]Nope it is not so clear. Can you show me an example pls?[/quote]
Steve Loughran
>
>
> Steve Loughran wrote:
> > On 15/11/06, BenLau <bkla...@gmail.com> wrote:
> >
> > >
> > > In short failures due to TestNG runner itself directly/indirectly vs
> > > failures in the tests themselves are separate issues.
> > > Hope this is clear.
> > >
> >
> > If you want to catch BuildExceptions that get raised in a build,
> > Ant-contribs <trycatch> task can be used to catch them. If you are
> > feeling very leading edge, my own <functionaltest> task is available
> > for use
>
> Steve:
>
> Thanks for your suggestions.
> The problem with current "haltonfailures=true/false" is that it
> catches both tests and non-tests related exceptions in one bucket.
> So, try catch won't help here.
>
> >
ok. so if haltonfailure=false, any setup problem doesnt trigger a
failure. That does abandon you. If it did skip on failure you could
set haltonfailure=false, have some failure property set and bail on
that afterwards with a conditional <fail>. As it is, you are going to
have to edit the source of the ant task to meet your needs.
On that subject, can I point the task authors at antunit:
http://ant.apache.org/antlibs/antunit/
this a is a few releases from being the official unit test framework
for testing ant tasks (it should go to 1.0 at the same time as
ant1.7). If you are writing new tasks, and can test only on ant1.7+,
then it makes your test life a lot easier.
-Steve
BenLau
Have you tried TestNG with Ant 1.7?
>
> -Steve
Steve Loughran
>
> Thanks. Using AntUnit is a possibility.
> Have you tried TestNG with Ant 1.7?
> >
No, though I'd have expected the Ant team to have got complaints if
Ant1.7RC1 broke something. Certainly incompatibilities between
Ant1.6.5 and Junit 4 have raised support issues, issues that go away
with Ant1.7.
Normally I'd expect Apache gump (http://gump.apache.org) to do the
regression tests between OSS projects, its being the nightly build of
all Ant and Maven 1 based projects that it knows how to check out and
build. But i dont see any references to testng in there as either a
producer or consumer; most projects have stuck on junit3.8.x. There's
valid reasons for not moving to junit4.0, obviously, but the fact that
downstream tools like Apache Cactus and httpunit are stuck in the
junit3.8 world is a barrier to improvements in test processes.
If anyone does want to write the gump XML descriptors to get testng
building on Gump, I (or any other apache committer) can add it to the
gump codebase. Its the best way to stop OSS projects you depend on
from breaking your app, and as Kaffe and Harmony are hosting the
builds, its becoming a major functional test for the open source JVMs.
-steve
Alexandru Popescu
Steve, I am not sure why we would need this? Can you detail a bit?
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
> -steve
>
> >
>
Steve Loughran
Apache Gump is a nightly build of all OSS projects in all repositories
that it can access, apache, sourceforge, google code, whatever. IT
checks them out, it runs the builds, runs the tests and complains if
the build fails to whoever's build fails. You declare what you depend
on, so things get built in the right order and the classpath is set up
for you in advance.
If anyone regresses their code, so that log4j changes a method and
your code breaks, you catch it the day they make the change, not
months later when the thing ships. If you make a change that breaks
people downstream, they complain to you within 24 hours. Its partly a
continuous integration server, partly a regression test for the tools
and APIs of the OSS world. A bug in xerces or ant kills everything,
for example.
I build all the sourceforge-hosted examples of the forthcoming
Ant-in-Action book, and elsewhere validate the XSD and WSDL of a
grid-related specification I have to implement. When all is well, gump
is silent, but when it fails, it emails whoever you tell it to (the
group or an individual)
The log shows which things are building, which things are breaking,
and which are down with a dependency failure:
http://vmgump.apache.org/gump/public/buildLog.html
Today it is junit that wont compile, so a lot of downstream stuff is offline.
If TestNG gets on gump you get.
-a free nightly regression test against your dependencees and dependees
-easy use of your lib by other projects
-testing on kaffe and harmony JVMs.
Its not quite as reactive as a private CI server that rebuilds and
retests every 15 minutes, but because it clean builds everything, it
is a good integration test of the java world.
-steve
Alexandru Popescu
Our dependencies are minimal and fixed so I am not interested in
testing it against the latest changes.
> -easy use of your lib by other projects
I am not sure what this one means.
> -testing on kaffe and harmony JVMs.
>
Frankly speaking I am completely not interested in kafee and harmony :-).
> Its not quite as reactive as a private CI server that rebuilds and
> retests every 15 minutes, but because it clean builds everything, it
> is a good integration test of the java world.
>
Probably this would work well for daily moving project, but so far
TestNG has only a couple of developers (2 core developers and about 5
more). As all of us are busy changes are not happening on daily basis
and our internal build system has proved enough good to catch
problems.
Thanks a lot for details. I thought I have overlooked something, but
now my initial thoughts about gump were confirmed.
Steve Loughran
>
>
> > -easy use of your lib by other projects
>
> I am not sure what this one means.
It means that projects that depend on your artifact can build against
your nightly version. Otherwise someone has to stick the JAR somewhere
and create a static dependency.
> Probably this would work well for daily moving project, but so far
> TestNG has only a couple of developers (2 core developers and about 5
> more). As all of us are busy changes are not happening on daily basis
> and our internal build system has proved enough good to catch
> problems.
>
> Thanks a lot for details. I thought I have overlooked something, but
> now my initial thoughts about gump were confirmed.
Well, the main thing you'd lose by opting out of gump is you make it
harder for projects downstream to use your library. Right now none of
the OSS projects on gump appear to do this. As and when the need
arises, someone may add you to the gump whether you want to be part of
it or not. all you get to do is decide whether you want the emails or
not.
-steve
BenLau
> On 11/21/06, BenLau <bkla...@gmail.com> wrote:
> >
> > So, Alex:
> >
> > In short, I need :
> >
> > 1) haltonfailures to control failures originated from running tests.
> > 2) haltonerrors to control any other errors/exception not originating
> > from tests themselves.
> >
> > So what do you think?.
> >
>
> Exactly what I was saying a couple of emails before:
>
> [quote]Nope it is not so clear. Can you show me an example pls?[/quote]
Alex:
Here's a sample log below to emphasize what I'm talking about.
I set the haltonfailure to false. So even if the TestNG runner fails or
test itself failed, I don't know which fails
because "haltonfailure" is a "catch all" for both exception origins.
I want to be able to take different course of action depending on the
origin of the failures.
The failure below was not due to running the tests itself but rather
due to a "ClassNotFound" exception while the runner initializes.
So its not a "proper failure" due to the tests itself but due to the
runner hosting the tests.
I need to be able to separate out failures orginated from the test
proper vs other failures due to other cause.
For a developer running tests from an IDE, it might not matter, but for
an enterprise-level automation which runs thousands of tests from
different product group, it does matter to distinguish between the 2
failure mode.
That's what I mean.
-BK-
Sample log:
============================================================================================
2006-12-05 13:55:11,141[main] INFO org.testng.TestNGAntTask -
[TestNGAntTask] TESTNG PASSED
@C:\DOCUME~1\XYZ\LOCALS~1\Temp\testng14749 WHICH CONTAINS:
2006-12-05 13:55:11,156[main] INFO org.testng.TestNGAntTask -
[TestNGAntTask] -log 1 -d
G:\TestNG_5.1_Stage4\xute-0.3.0\testresults\org\junit\tests\JUnit-QuickTestWithSetup3.scen-XYZ-145
-sourcedir G:\TestNG_5.1_Stage4\xute-0.3.0 -listener
com.a.b.automation.xutex.runtime.Log4jListener -listener
com.a.b.automation.xutex.runtime.SuiteListener -suitename "Ant suite"
-testname "Ant test"
"G:\TestNG_5.1_Stage4\xute-0.3.0\tmp\suites\junit-suite-1.xml"
2006-12-05 13:55:11,188[main] DEBUG org.testng.TestNGAntTask -
Executing 'G:\TestNG_5.1_Stage4\xute-0.3.0\jdk1.4.2\jre\bin\java.exe'
with arguments:
'-Djava.library.path='
'-Dlog4j.configuration=log4j.test.xml'
'-Demma.rt.control.port=55555'
'-Demma.coverage.out.file=G:\TestNG_5.1_Stage4\xute-0.3.0\testresults\org\junit\tests\JUnit-QuickTestWithSetup3.scen-XYZ-145\coverage\result/coverage.ec'
'-Demma.coverage.out.merge=true'
'-Dtestng.show.stack.frames=true'
'-Xms32M'
'-Xmx128M'
'-ea'
'-Dxute.home=G:\TestNG_5.1_Stage4\xute-0.3.0'
'-Dxute.tai.log=xute.tai.log.xml'
'-classpath'
'G:\TestNG_5.1_Stage4\xute-0.3.0\tmp\instr\lib\junit.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\tmp\instr\lib\junit.samples.money.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\testcontents\org\junit\tests\junit.samples.money.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\apache\ant\lib\junit.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\framework\xute.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\apache\ant\lib\log4j-1.2.13.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\run;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\framework\testng-5.3-jdk14.jar'
'org.testng.TestNG'
'@C:\DOCUME~1\bklau\LOCALS~1\Temp\testng14749'
The ' characters around the executable and arguments are
not part of the command.
2006-12-05 13:55:11,953[Thread-5] WARN org.testng.TestNGAntTask
- Exception in thread "main" java.lang.NoClassDefFoundError:
com/a/b/automation/logging/LoggerDispatcher
======================================================================================================
BenLau
I have provided an example but haven't heard back from you.
So, is new "haltonerrors" atrribute OK with you for next TestNG5.5
runner?
This is meant to catch all errors originating from the runner but not
any of the errors/failures from the tests proper themselves which the
runner runs.
Thanks,
BK
Alexandru Popescu
providing the needed functionality? Then if you find it useful just
send it over and we will see if it makes sense to be incorporated in
the core.
BR,
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator