Official definition of the terms stable, unstable, failure, successful, unsuccesful
Christoph Kutzinski
to my knowledge there's no official definition of these terms. I found this page https://wiki.jenkins-ci.org/display/JENKINS/Terminology where it's tried to explain them, but I find some of the explanations not satisfying. E.g. "Successful build: A build is successful when the compilation reported no errors." There are other reasons, why a build wouldn't be successful.
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Result.java has some better - still not perfect - explanations, but these are AFAIK at no plcae user visible.
Shouldn't we have some clear definitions of these terms?
cheers
Christoph
PS: I've never liked that the term 'successful' means stable OR unstable. IMO unstable builds are not successful, but I fear that for backwards compatibility we have to leave it that way
PPS: On a side node, I see some problems with Result and some of the comparative methods in it. E.g. why is a NOT_BUILT build 'worse' than a FAILURE build?
How can we sensefully 'combine' a NOT_BUILT result with any other result?
Christoph Kutzinski
Currently, it's IMO not clear what the correct behaviour of Jenkins should be in the described case.
Martin B.
some impovement = more terms explained in more detail.
My last mail wrt. to Terminology on the users list
(http://jenkins.361315.n4.nabble.com/Terminology-blocked-vs-stuck-vs-buildable-td3698604.html)
didn't exactly get any response.
Maybe I should try on IRC or jenkins-dev next time I need something
explained :-)
br,
Martin
Sami Tikka
There are 2 main categories and some subcategories:
Category A: Job has no test reports configured:
A1: If the last build command executed by Jenkins exits with failure
(that is, non-zero) exit code, the build is considered to be failed.
Permalinks lastBuild, lastUnsuccessfulBuild and lastFailedBuild will
move forward.
A2: If the last build command exits with success (that is, zero) exit
code, the build is successful. Permalinks lastBuild,
lastSuccessfulBuild and lastStableBuild will all move forward.
Category B: Job has one or more test reports configured:
B1: If the last build command executed by Jenkins exits with failure
(that is, non-zero) exit code, the build is considered to be failed.
(Identical to A1). Permalinks lastBuild, lastUnsuccessulBuild and
lastFailedBuild will move forward.
B2: If the last build command exits with success (that is, zero) exit
code but test reports have some failed tests, the build is unstable.
Permalinks lastBuild, lastSuccessfulBuild and lastUnsuccessfulBuild
will move forward.
B3: If the last build command exits with success and all test have
passed, the build is stable. Permalinks lastBuild, lastSuccessfulBuild
and lastStableBuild will move forward.
OK, maybe the behavior of the lastUnsuccessfulBuild isn't very intuitive. :)
-- Sami
2011/8/13 Christoph Kutzinski <ku...@gmx.de>:
Steve M. Robbins
> Category B: Job has one or more test reports configured:
>
> B1: If the last build command executed by Jenkins exits with failure
> (that is, non-zero) exit code, the build is considered to be failed.
> (Identical to A1). Permalinks lastBuild, lastUnsuccessulBuild and
> lastFailedBuild will move forward.
>
> B2: If the last build command exits with success (that is, zero) exit
> code but test reports have some failed tests, the build is unstable.
> Permalinks lastBuild, lastSuccessfulBuild and lastUnsuccessfulBuild
> will move forward.
Coincidentally, I've been lately wondering how to make "B2"
fail as in "B1". Is there a way to do that? Specifically,
we run FxCop on C# code and we'd like to fail a build where there
are more than N fxcop warnings.
Thanks,
-Steve
Christoph Kutzinski
> A1: If the last build command executed by Jenkins exits with failure
> (that is, non-zero) exit code, the build is considered to be failed.
Does it really need to be the last build command? Can't it be that some intermediate (or concurrent) step failed and the build must still be considered failed?
What if the build itself succeeded, but some Jenkins internal reporters had an error? Does that mean the build should be stable, unstable or failed?
On the other hand, can't it be that all build steps succeeded (i.e. 0 exit code), but still the build should be considered as failed? E.g. some code analysis showed that there are severe bugs in the code?
> B2: If the last build command exits with success (that is, zero) exit
> code but test reports have some failed tests, the build is unstable.
As you describe it, a build can only become unstable, if it has tests. I don't think that's correct. See the code analysis example from above. If there are some not-so-severe bugs detected in the code analyis the build could be marked as unstable.
What if the build succeeded, it also ran some tests, but Jenkins was unable to evaluate the test results (see JENKINS-7676), should the build be unstable or failed?
For the reasons above, I think I would like to have a 'business value' (can't find a better term for that, currently) definition of these terms. I.e. not which concrete steps can lead to a failed or unstable builds, but instead: what does this mean for the artifacts created by this build?
I could imagine something like this:
- Failed: artifacts created by the build couldn't/shouldn't be used by any further steps in the value chain. E.g. shouldn't be deployed onto a test system for manual testing
- Unstable: Artifacts *could* be used in further steps in the value chain - but it's usually not recommended to do so, because there are some known defects in them
- Stable: everything fine with the build. Artifacts can be used without constraints in further steps.
(Of course this assume that there are some artifacts created by the build, but I cannot imagine, currently, how to express it differently)
cheers
Christoph
-------- Original-Nachricht --------
> Datum: Sun, 14 Aug 2011 21:58:44 +0300
> Von: Sami Tikka <sjt...@gmail.com>
> An: jenkin...@googlegroups.com
> Betreff: Re: Official definition of the terms stable, unstable, failure, successful, unsuccesful
David Kennedy
[...] On the other hand, can't it be that all build steps succeeded (i.e. 0 exit code), but still the build should be considered as failed? E.g. some code analysis showed that there are severe bugs in the code?
Good discussion. We're actually deciding internally what to do with not-quite-clear-cut cases like this - some developers get unhappy when Jenkins fails a build for reasons unclear to them at commit time. For example, we cannot replicate historical behaviour on the command line, but Jenkins might fail if regression test times go up, or if there are more static analysis warnings than before (while allowing existing warnings to pass).
I think it's very valid for builds to fail this way, often due to plugins, but I accept the argument that if you can't replicate the fail on your commit target, then it's an unfair call on a developer's work*. This is probably the only reason I can see for an intermediate Unstable state - but as I'd block downstream builds, it's still effectively Failed in terms of automated workflows.
(*However, I think getting defensive about any tool is always a perspective issue, and I'm happy to have infrequent unexpected fails PROVIDED the team get value from whatever failed)
[...] For the reasons above, I think I would like to have a 'business value' (can't find a better term for that, currently) definition of these terms
- Failed: artifacts created by the build couldn't/shouldn't be used by any further steps in the value chain. E.g. shouldn't be deployed onto a test system for manual testing
- Unstable: Artifacts *could* be used in further steps in the value chain - but it's usually not recommended to do so, because there are some known defects in them
- Stable: everything fine with the build. Artifacts can be used without constraints in further steps.
Excellent suggestion; that's nailed how I think of the states. Admittedly, I have a philosophical problem with Unstable, and regard it simply as Failed - as a human has to make a judgement call on the artifacts anyway, cf Junit - but as discussed above, there are edge cases to consider. However, this third state does complicate the links discussed upthread (lastUnsuccessful etc)
Summary - +1 for documenting /something/ as a guide, and acknowledge that there is some ambiguity in the terms. I like Christoph's proposed guidelines for how to describe each state in terms of the builds utility instead of focusing on completion/failure of steps alone.
Stephen Connolly
Remember unstable is more of a marker for Jenkins itself, to allow
further builds to continue using those artifacts, or to allow QA to
run tests in a different area.
with TDD, you might have a few test cases failing for a while until
you get the feature implemented, that should not block downstream
consumers, but it should not be a full release...
with Blind Panic Driven Development, you (hope to) have fixed the
issue and broken some tests in the course of so doing, now you want
the final artifact where you (or maybe somebody else) can actually
test the fix before you spend more time fixing the test cases that you
broke.
Unstable helps in these cases... though I don't like them ;-)
Martin B.
> ....
>
> For the reasons above, I think I would like to have a 'business value' (can't find a better term for that, currently) definition of these terms. I.e. not which concrete steps can lead to a failed or unstable builds, but instead: what does this mean for the artifacts created by this build?
> I could imagine something like this:
>
> - Failed: artifacts created by the build couldn't/shouldn't be used by any further steps in the value chain. E.g. shouldn't be deployed onto a test system for manual testing
>
> - Unstable: Artifacts *could* be used in further steps in the value chain - but it's usually not recommended to do so, because there are some known defects in them
>
> - Stable: everything fine with the build. Artifacts can be used without constraints in further steps.
>
>
> (Of course this assume that there are some artifacts created by the build, but I cannot imagine, currently, how to express it differently)
>
+1 - I think these could go straight to the Terminology page :-)
What might be a nice twist would be for the Wiki to directly link to the
actual Java Code that sets these statuses. That way, if you need the
details, you can always look it up in the code. (Also means less
maintenance for the docs, should the code change.)
cheers,
Martin
Christoph Kutzinski
https://issues.jenkins-ci.org/browse/JENKINS-10763
Maybe I'll be taking it to the next governance board meeting on IRC, too.
Stephen, regarding your additions regarding Unstable builds:
I don't agree completely that this is more a Jenkins internal state, only.
I see however, that this state is handled different in different
organisations and therefore makes it more difficult to describe the
terms sharp.
In my company it's e.g. so that Unstable means basically the same as
Failed: no further actions happen on the artifacts, builds must be fixed
ASAP.
In more test driven companies it may however be normal that builds are
unstable for a longer period of time
We should remember that when defining the terms, e.g. making them only
'recommendations' o.s.l.t.
Dean Yu
> For the reasons above, I think I would like to have a 'business value' (can't
> find a better term for that, currently) definition of these terms. I.e. not
> which concrete steps can lead to a failed or unstable builds, but instead:
> what does this mean for the artifacts created by this build?
> I could imagine something like this:
I liked these overall. One minor point...
>
> - Failed: artifacts created by the build couldn't/shouldn't be used by any
> further steps in the value chain. E.g. shouldn't be deployed onto a test
> system for manual testing
>
Wouldn't a failed status likely indicate no artifacts were produced at all?
-- Dean
Christoph Kutzinski
-------- Original-Nachricht --------
> Datum: Thu, 18 Aug 2011 15:15:42 -0700
> Von: Dean Yu <d...@yahoo-inc.com>
> An: "jenkin...@googlegroups.com" <jenkin...@googlegroups.com>, Christoph Kutzinski <ku...@gmx.de>
> Betreff: Re: Official definition of the terms stable, unstable, failure, successful, unsuccesful
> On 8/15/11 2:50 AM , "Christoph Kutzinski" <ku...@gmx.de> wrote:
Yes, good point. In most cases yes, but the are also scenarios where artifacts were already created, but later analysis steps (e.g. FindBugs) mark the build as failure.
But nevertheless we should adapt the description to account for the common case.
Christoph
J T Halliley
may wish to fail a build if
code coverage per method is < 70%
or if
crappiness > 3%
or other such measures.
Thoughts?
Tom
Sami Tikka
>> A1: If the last build command executed by Jenkins exits with failure
>> (that is, non-zero) exit code, the build is considered to be failed.
>
> Does it really need to be the last build command? Can't it be that some intermediate (or concurrent) step failed and the build must still be considered failed?
I do not know about any "need", I was just describing how Jenkins
works at the moment.
Actually, I should clarify: I was describing how Jenkins "Free style"
jobs work. I am now familiar with how Maven jobs work.
Using free style jobs it is possible to write your own build step and
if you want to, you can ignore command failures and check command exit
codes yourself and decide which ones are relevant.
> What if the build itself succeeded, but some Jenkins internal reporters had an error? Does that mean the build should be stable, unstable or failed?
Internal reporters? Do you mean e.g. a situation where a test report
.xml file has invalid XML and Jenkins fails to parse it? Good
question. I haven't seen that very often and do not remember how
Jenkins behaves. I hope it marks the build as failed.
I myself think of "red" builds as really bad things: someone committed
code so broken it did not compile or my test infrastructure is so
broken it failed to run any tests. => Things to fix immediately.
"Yellow" builds are bad but slightly better: at least it compiles and
some tests could be run. But the tests need to be fixed ASAP.
"Blue" builds are good.
> On the other hand, can't it be that all build steps succeeded (i.e. 0 exit code), but still the build should be considered as failed? E.g. some code analysis showed that there are severe bugs in the code?
Sure, some plugins can be used to alter the behavior of Jenkins.
>> B2: If the last build command exits with success (that is, zero) exit
>> code but test reports have some failed tests, the build is unstable.
>
> As you describe it, a build can only become unstable, if it has tests. I don't think that's correct. See the code analysis example from above. If there are some not-so-severe bugs detected in the code analyis the build could be marked as unstable.
I was describing the behavior of Jenkins free style jobs without any
plugins that modify build result. I agree there exists analysis
plugins that can make a build unstable without test results.
> For the reasons above, I think I would like to have a 'business value' (can't find a better term for that, currently) definition of these terms. I.e. not which concrete steps can lead to a failed or unstable builds, but instead: what does this mean for the artifacts created by this build?
IMHO Jenkins only provides the technology.
Each organization/team/person should decide how the build result maps
to their business values based on their build commands and
interpretations.
One organization could decide unstable build is good enough for
internal deployment. Another organization may think unstable is
equivalent to failed. It depends.
-- Sami
Sami Tikka
> Actually, I should clarify: I was describing how Jenkins "Free style"
> jobs work. I am now familiar with how Maven jobs work.
I am not familiar with how Maven jobs work.
(Damn you, OS X Lion autocorrect)
-- Sami