retry build upon failure?

[Ed Young]

Is it possible to retry a build upon a failure? Maybe with a retry count?

[nicolas de loof]

see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin

2012/9/14 Ed Young <e...@summitbid.com>

[William Soula]

Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin

[Ed Young]

I installed the Naginator Plugin (v1.8) because the retry failed build
plugin indicated that it was deprecated in favor of the Naginatore
plugin.

It doesn't seem to be working the way I expect it to however. I
configured it to simply

Retry build after failure up to 3 times, with all other defaults, but
after a failure there is no retry. I configured it for a fixed delay
and it still did not retry.

It seems like a very simple configuration and the build is a hard
failure (test failures) so I can't tell where I'm going wrong.
Using Jenkins 1.436.

--
- Ed

[Sami Tikka]

Could you post exactly what you see in the console log of the failed build? Also, it might help if you could post your job config.xml or put in pastebin/gist so we could take a look.

-- Sami

[Mark Waite]

When Ed says the build is a"hard failure", I assumed that meant the build status was "Failed" and would be marked red.  However, in parentheses he says "(test failures)".  Typically, test failures mark the build as "unstable", not as "failed".  If the build is unstable, I would not expect a "retry on fail" to be invoked, since "unstable" is not "failed".

Can you clarify?  Was the build red (failed) or yellow (unstable)?

Mark Waite

---

From: Sami Tikka <sjt...@gmail.com>
To: "jenkins...@googlegroups.com" <jenkins...@googlegroups.com>
Sent: Sunday, September 16, 2012 11:08 AM

[Ed Young]

Sounds like we have some terminology differences. These are test
failures, which I consider a failed builds. I've experimented with the
configuration in many different ways and it never results in a retry,
even when I set it to retry after "unstable" builds (test failure I
guess)

If the build passes all tests, it's a pass and "blue". If it fails any
tests at all it's "red" which I call a "hard failure", but others may
not, I reckon. In any case, our policy is that if it fails with test
failures, it's a failed build.

So I set it to retry upon "unstable" builds and it still does not retry.

These are grails builds, and we're using the spock test framework.

Here's the relevant sections of the build config:
<com.chikli.hudson.plugin.naginator.NaginatorPublisher>
<regexpForRerun></regexpForRerun>
<rerunIfUnstable>false</rerunIfUnstable> <-- original setting
<checkRegexp>false</checkRegexp>
<delay class="com.chikli.hudson.plugin.naginator.FixedDelay">
<delay>1</delay>
</delay>
<maxSchedule>3</maxSchedule>
</com.chikli.hudson.plugin.naginator.NaginatorPublisher>

--------

<com.chikli.hudson.plugin.naginator.NaginatorPublisher>
<regexpForRerun></regexpForRerun>
<rerunIfUnstable>true</rerunIfUnstable> <---- retry on unstable
<checkRegexp>false</checkRegexp>
<maxSchedule>2</maxSchedule>
</com.chikli.hudson.plugin.naginator.NaginatorPublisher>

---------

In no cases does it retry.

Here is the end of an example build that fails due to test failures.

...
Running 342 spock tests... 188 of 342
| Completed 342 spock tests, 3 failed in 865846ms
| Compiling 1 source files
| Compiling 1 source files.
| Compiling 1 source files..
| Tests FAILED - view reports in
/opt/comcast/data/jenkins/jobs/chimps-geb-grails-lb-test/workspace/target/test-reports
Build step 'Build With Grails' marked build as failure

Still didn't retry even after specifying retry after an "unstable" build.

--
- Ed

[Mark Waite]

I think you've found a bug.  At least, my initial experiments seem to confirm the same results you reported.

I had never experimented with the Naginator plugin before.  I installed it into a Jenkins 1.466.2 installation and created a new free style project with a single build step, a shell with the one line command "exit 1" and a single post build action, retry failed build.

I configured the retry post build action to rebuild up to two times.

The first time that job ran, the jenkins log file contained the following report:

INFO: Failure-Test #1 main build action completed: FAILURE

Sep 16, 2012 9:56:54 PM hudson.model.Executor run

SEVERE: Executor threw an exception

java.lang.NullPointerException

        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)

        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)

        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)

        at hudson.model.Run.run(Run.java:1488)

        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)

        at hudson.model.ResourceController.execute(ResourceController.java:88)

        at hudson.model.Executor.run(Executor.java:239)

The job did not retry.

Can you submit a bug against the Naginator plugin?

Thanks,

Mark Waite

---

From: Ed Young <e...@summitbid.com>
To: jenkins...@googlegroups.com
Sent: Sunday, September 16, 2012 8:56 PM

[Mark Waite]

I further simplified my configuration by removing Jenkins and its configuration files completely from a test computer, then I installed Jenkins 1.466.2, installed the Naginator plugin, and confirmed it was the only plugin installed.  I created a new Jenkins job with a shell command "exit 1" and set it to retry up to twice on failed builds.

The /var/log/jenkins.log file again included the following stack trace:

Sep 16, 2012 10:15:51 PM hudson.model.Run run

INFO: Naginator-Retry #1 main build action completed: FAILURE

Sep 16, 2012 10:15:51 PM hudson.model.Executor run

SEVERE: Executor threw an exception

java.lang.NullPointerException

        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)

        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)

        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)

        at hudson.model.Run.run(Run.java:1488)

        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)

        at hudson.model.ResourceController.execute(ResourceController.java:88)

        at hudson.model.Executor.run(Executor.java:239)

Looking at the source code, it appears the plugin wants a delay value.  I had not set a delay value, since it was optional in the user interface.  When I set the delay value to a fixed 2 second delay, then the job retried after first failure (and no null pointer exception in the log file).  I was able to confirm that it would retry up to 4 times after a failed build.

You might consider setting a delay value to see if that helps your case like it helped mine.

You might also consider watching the log file to see if there are any further hints in what you're doing.

I'll file the bug report with the results of my investigation.

Mark Waite

---

From: Mark Waite <mark...@yahoo.com>
To: "jenkins...@googlegroups.com" <jenkins...@googlegroups.com>
Sent: Sunday, September 16, 2012 10:09 PM

[Ed Young]

done:
https://issues.jenkins-ci.org/browse/JENKINS-15184

I'll do some more investigating, as well.

Thanks,

Ed

--
- Ed

[Mark Waite]

I see that you've already submitted Jenkins-15184.  Can you investigate if your failure mode is the same failure mode as the one I describe below?  

If it is the same, then you can attach this information to the bug report (and you can use it as a work around).

If it is not the same, then I can submit a bug report for this problem in addition to your problem.

Thanks,

Mark Waite

---

From: Mark Waite <mark...@yahoo.com>
To: "jenkins...@googlegroups.com" <jenkins...@googlegroups.com>

Sent: Sunday, September 16, 2012 10:26 PM

[Ed Young]

I can, but not till tomorrow. Thanks for looking into it with me. -Ed

--
- Ed