Hudson Maven 3 problem when tests fail

[João Ferreira]

Hi

I was trying the new hudson maven 3 support in my projects but I am having problems.

When a test fails during the build instead of an "unstable" build, the build stops with an error. Is this normal?

Here is the console output of the build:

2011-01-05 11:52:19.733 GMT [org.mockejb.jndi.MockContext] Unbound name TopicConnectionFactory
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec

Results :

Failed tests: 
  testCreateOrder(com.XXX.XXX.tests.services.manager.ManagerTests)

Tests run: 58, Failures: 1, Errors: 0, Skipped: 8

mojoFailed org.apache.maven.plugins:maven-surefire-plugin:2.5(default-test)
[HUDSON] Recording test results
projectFailed com.XXX.XXX:tibco_plugins:3.2-SNAPSHOT
[HUDSON] Archiving F:\hudson\jobs\trunk\workspace\XXX\java\tibco_plugins\pom.xml to F:\hudson\jobs\trunk\modules\com.XXX.XXX$tibco_plugins\builds\2011-01-05_11-47-56\archive\com.XXX.XXX\tibco_plugins\3.2-SNAPSHOT\pom.xml
sessionEnded
maven builder waiting
mavenExecutionResult exceptions not empty
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.5:test (default-test) on project tibco_plugins: There are test failures.

Please refer to F:\hudson\jobs\trunk\workspace\XXX\java\tibco_plugins\target\surefire-reports for the individual test results.
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:140)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:314)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:151)
	at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:79)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
	at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:145)
	at hudson.maven.Maven3Builder.call(Maven3Builder.java:123)
	at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
	at hudson.remoting.UserRequest.perform(UserRequest.java:114)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:270)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.

Please refer to F:\hudson\jobs\trunk\workspace\XXX\java\tibco_plugins\target\surefire-reports for the individual test results.
	at org.apache.maven.plugin.surefire.SurefirePlugin.execute(SurefirePlugin.java:629)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:107)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:195)
	... 27 more
Maven failed with error.
channel stopped
Email was triggered for: Failure
Email was triggered for: Still Failing
Trigger Failure was overridden by another trigger and will not send an email.
Sending email for trigger: Still Failing
Sending email to: XX...@ZZZZ.CC
Finished: SUCCESS

[Gabriel Petrovay]

Hi Joao,

Try using the -f... maven options when you call maven in your mvn build step:

-fae,--fail-at-end
-ff,--fail-fast
-fn,--fail-never

See the mvn --help for details

Hope this helps,
Gabriel

PS: There is a problem with some mvn plug-ins (tycho) that run tests
in other maven lifecycle phases than "test". In this case -fae option
will not work as expected. I think this is not a problem for
"surefire". I only observed it with the tycho maven plug-in (testing
eclipse plug-ins)

2011/1/5 João Ferreira <jtjefe...@gmail.com>:

--
MSc Gabriel Petrovay
Mobile: +41(0)787978034
www.28msec.com

[João Ferreira]

Hi 

Thanks for your suggestion.

I tried the -fae and indeed when there are test errors the build only fails at the end, however the build status is "failed" instead of "unstable". 

I will now try the -fn option but isn't this a bit drastic? I would like that errors could occur (compilation for example) return a "failed" build...

I understand the maven 3 support is recent, and i can rollback to the maven 2 plugin. At the moment this is the only issue I am having.

Best regards

João Ferreira

2011/1/6 Gabriel Petrovay <gabipe...@gmail.com>

[Gabriel Petrovay]

I was also fighting last days to get unstable builds as well. But I
didn'tmanage to do it. I had to diable the tests by using either:
- use @ignore for the unwanted tests (this will disable the test and
they will appear yellow in the report)
or
- use assumeTrue(false) or similar assume function (from JUnit4) for
the tests that you want to skip (they should be also marked as
skipped, yellow, in the report)

Usually the policy is that failing test should fail the build. I see
your need for "unstable". Let me know if you manage to obtain it. With
tycho (which simulates surefire but for eclipse test plugins) I
couldn't.

Thanks!

Gabriel

2011/1/6 João Ferreira <jtjefe...@gmail.com>:

[Nord, James]

Have you tried -Dmaven.test.failure.ignore=true ?

/James

Thanks!

Gabriel

>> > org.apache.maven.plugin.surefire.SurefirePlugin.execute(SurefirePlu
>> > gin.java:629)
>> > at
>> >
>> > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Defau
>> > ltBuildPluginManager.java:107)
>> > at
>> >
>> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecut

>> > or.java:195)
>> > ... 27 more
>> > Maven failed with error.
>> > channel stopped
>> > Email was triggered for: Failure
>> > Email was triggered for: Still Failing Trigger Failure was
>> > overridden by another trigger and will not send an email.
>> > Sending email for trigger: Still Failing Sending email to:
>> > XX...@ZZZZ.CC
>> > Finished: SUCCESS
>>
>>
>>
>> --
>> MSc Gabriel Petrovay
>> Mobile: +41(0)787978034
>> www.28msec.com
>
>

--
MSc Gabriel Petrovay
Mobile: +41(0)787978034
www.28msec.com

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

[Gabriel Petrovay]

Yup, but that does not make it through the tycho maven plug-in (at
least not in version 0.9.0). It appears to be ignored. That works with
surefire (tycho states that they support the surefire configuration,
but probably not completely)

I still have to try in tycho 0.10.0 and 0.11.0-SNAPSHOT, maybe
something changed.

Thanks!

[Nord, James]

Sorry missed the tycho bit - a quick scan through tycho shows there is not an option to ignore tests results
https://github.com/sonatype/sonatype-tycho/blob/master/tycho-surefire/maven-osgi-test-plugin/src/main/java/org/codehaus/tycho/osgitest/TestMojo.java

You probably want to ask this on the tycho users list. If there is no way to do it on the command line - then it will be pretty hard for Hudson to do any magic.

Thanks!

>>> > org.apache.maven.plugin.surefire.SurefirePlugin.execute(SurefirePl
>>> > u
>>> > gin.java:629)
>>> >       at
>>> >
>>> > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Defa
>>> > u
>>> > ltBuildPluginManager.java:107)
>>> >       at
>>> >
>>> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecu

>>> > t
>>> > or.java:195)
>>> >       ... 27 more
>>> > Maven failed with error.
>>> > channel stopped
>>> > Email was triggered for: Failure
>>> > Email was triggered for: Still Failing Trigger Failure was
>>> > overridden by another trigger and will not send an email.
>>> > Sending email for trigger: Still Failing Sending email to:
>>> > XX...@ZZZZ.CC
>>> > Finished: SUCCESS
>>>
>>>
>>>
>>> --
>>> MSc Gabriel Petrovay
>>> Mobile: +41(0)787978034
>>> www.28msec.com
>>
>>
>
>
>
> --
> MSc Gabriel Petrovay
> Mobile: +41(0)787978034
> www.28msec.com
>
>
> **********************************************************************

> **************** This message is confidential and intended only for

> the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.
>
> NDS Limited. Registered Office: One London Road, Staines, Middlesex,
> TW18 4EX, United Kingdom. A company registered in England and Wales.
> Registered no. 3080780. VAT no. GB 603 8808 40-00
> **********************************************************************
> ****************
>

--

[João Ferreira]

Hi

Unlike Gabriel I am not using tycho. And skipping tests is not a solution. My test is failing and, of course, needs fixing.

However with hudson maven 2 plugin, i would get a unstable build. Now with maven3 I get a failed build.

Is this by design? Or is it a bug?

Best regards

João Ferreira

2011/1/6 Nord, James <JN...@nds.com>

[Nord, James]

It would be a bug and you may(should) be able to work around it by adding -Dmaven.test.failure.ignore=true to the maven goals.

 

/James

[João Ferreira]

Hi

Thanks for the clarification. Should i file an issue?