Test failure does not cause build/stage failure in a pipeline

[Cristi Campean]

Hi,

We are trying to run in a stage some testNG tests using maven.

The setup looks like this:
 - we have the tests in git and the git repo as a material for the tests runner pipeline
 - in the pipeline we have a stage which executes: mvn clean install -DtestngGroups="sanity" in the folder with the tests
 - one of the tests is failing, however the maven build is SUCCESS (the same behavior is in Jenkins, however, Jenkins marks the build as being UNSTABLE) 
 - hence, our stage is green (successful)

Basically, we want to fail our stage when there are any test failures in the maven build.

Do you have any ideas about what should be wrong with our current setup?

Thanks,
Cristian

[srinivas upadhya]

I assume the command

mvn clean install -DtestngGroups="sanity"

is exiting with zero telling Go that process ran successfully.

Go will fail a job only if task (command spawned by Go) returns a non-zero exit code.

Can you make sure it is returning a non-zero exit code when tests fail?

Also are you uploading surefire reports (xml's) as "test artifact"? Is Go showing tests failure in test tab?

[Cristi Campean]

Yes, I'm uploading surefire reports:

Source: **/target/surefire-reports/junitreports/TEST-**
Destination: tests
Type: Test Artifact

And in the test tab I have the following message:

Although all the jobs in this stage have passed, there are some tests that have failed.

This is may be due to the test task configuration in the build script. You might want to fail the build on test failures.

[srinivas upadhya]

Which confirms that the task (command spawned by Go) is returning zero.

A quick Google gives me this: http://stackoverflow.com/questions/1375133/how-to-detect-a-build-error-from-ant-maven-via-a-bash-script

Are you using Maven 2? If yes, can you move to Maven 3?

[Cristi Campean]

I'm using maven 3.0.3.

The problem was with the surefire plugin which was not failing the build in case of test failures.

I've set the flag testFailureIgnore from the plugin to false and now it's failing when there are test failures.

Also, the test tab is filled with the tests that are failing.

Thanks for your input Srinivas!

[orlando kelly]

I'm observing the same issue running ant / selenium tests

[junit] Running com.amec.convero.uitests.documentcontrol.DocumentControlFeaturesTest
15:40:58.874     [junit] Tests run: 13, Failures: 2, Errors: 0, Time elapsed: 80.213 sec
15:40:58.874     [junit] Test com.amec.convero.uitests.documentcontrol.DocumentControlFeaturesTest FAILED
15:40:58.889 
15:40:58.889 BUILD SUCCESSFUL
15:40:58.889 Total time: 1 minute 23 seconds
15:40:58.936 [go] Current job status: passed.

I have 2 failures but GO marks the build as succesful. If I run the same ANT jobs in Hudson/Jenkins they fail correctly.

Are there other settings I should be using.

[orlando kelly]

BTW, GO passes the test results okay to say there are 2 failures, but it still passes the job as succesful 

[Ketan Padegaonkar]

Parsing the test results and a failing build are two different things. Go honours just one thing - the exit code of the provided script.

If your build script is returning a zero value on what should be a build failure, that is something that the script must fix.

Could you check the exit status of your script?

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.