Cucumber binary exit status with failing tests
Jon Druse
From the Cucumber wiki:
"The cucumber executable will exit with an exit status different from 0 if one or more scenarios are failing..."
My question is, why is this a good idea? I cannot think of a scenario which this makes sense. The exit status should reflect the outcome of running the binary successfully or not, not the actual product of the program. The fact that there are failing scenarios is irrelevant. I'm sure that this decision had some thought put into it, I'm just trying to figure that out. Thanks!
happy cuking.
aslak hellesoy
A non-0 exit status allows parent processes (CI servers, build scripts
etc) to detect that there were test failures.
All command-line test tools work this way.
Aslak
> happy cuking.
>
> --
> You received this message because you are subscribed to the Google Groups "Cukes" group.
> To post to this group, send email to cu...@googlegroups.com.
> To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cukes?hl=en.
>
>
Jon Druse
aslak hellesoy
> Don't CI servers detect failures by parsing test results?
No they don't. They look at child processes' exit status. There are
more things than tests that can go wrong in a build:
* A compiler (gcc, javac etc.) can fail to compile sources.
* Tests can fail
* Code analysis can fail
* Deployment can fail
There are way too many different tools that do each of these things
that a CI server can understand their output to detect failure.
There might be some CI servers that look at tool output in addition to
the exit status, but it's the exit status that is the most reliable
way to communicate success/failure.
Aslak
David Chelimsky
> On Tue, May 24, 2011 at 12:30 AM, Jon Druse <jon....@gmail.com> wrote:
>> Don't CI servers detect failures by parsing test results?
>
> No they don't. They look at child processes' exit status. There are
> more things than tests that can go wrong in a build:
>
> * A compiler (gcc, javac etc.) can fail to compile sources.
> * Tests can fail
> * Code analysis can fail
> * Deployment can fail
>
> There are way too many different tools that do each of these things
> that a CI server can understand their output to detect failure.
> There might be some CI servers that look at tool output in addition to
> the exit status, but it's the exit status that is the most reliable
> way to communicate success/failure.
>
> Aslak
John - Aslak's experience aligns with mine: command line test tools indicate failure with a non-zero exit status.
I'm curious - what trouble is this causing you? Maybe there is a different way we can help you solve it.
David
Mike Sassak
> On May 23, 2011, at 6:39 PM, aslak hellesoy wrote:
>
>> On Tue, May 24, 2011 at 12:30 AM, Jon Druse <jon....@gmail.com> wrote:
>>> Don't CI servers detect failures by parsing test results?
>>
>> No they don't. They look at child processes' exit status. There are
>> more things than tests that can go wrong in a build:
>>
>> * A compiler (gcc, javac etc.) can fail to compile sources.
>> * Tests can fail
>> * Code analysis can fail
>> * Deployment can fail
>>
>> There are way too many different tools that do each of these things
>> that a CI server can understand their output to detect failure.
>> There might be some CI servers that look at tool output in addition to
>> the exit status, but it's the exit status that is the most reliable
>> way to communicate success/failure.
>>
>> Aslak
>
> John - Aslak's experience aligns with mine: command line test tools indicate failure with a non-zero exit status.
>
Not just test tools--virtually every Unix-ish command-line tool does
this. (Try echoing the value of $? after running ls without arguments
and with a filename that doesn't exist.) The meaning of numbers
greater than 0 are generally undefined outside of "an error occurred",
though some conventions do exist. There's nothing stopping Cucumber
from being more careful about its exit status, though. E.g. 1 could
mean the binary exploded, and 2 that a scenario failed.
John Lonergan
In hudson we often see support for this ...
<failTheBuild>false</failTheBuild>
as a purely tech observation , dunno if this is supported here.
But yep what are you trying to achieve?
On May 24, 3:30 am, Mike Sassak <msas...@gmail.com> wrote:
> > On May 23, 2011, at 6:39 PM, aslak hellesoy wrote:
>
> >>>>> Hey all,
>
> >>>>> From the Cucumber wiki:
>
> >>>>> "The cucumber executable will exit with an exit status different from 0 if one or more scenarios are failing..."
>
> >>>>> My question is, why is this a good idea? I cannot think of a scenario which this makes sense. The exit status should reflect the outcome of running the binary successfully or not, not the actual product of the program. The fact that there are failing scenarios is irrelevant. I'm sure that this decision had some thought put into it, I'm just trying to figure that out. Thanks!
>
> >>>> A non-0 exit status allows parent processes (CI servers, build scripts
> >>>> etc) to detect that there were test failures.
> >>>> All command-line test tools work this way.
>
> >>>> Aslak
>
> >>>>> happy cuking.
>
> >>>>> --
> >>>>> You received this message because you are subscribed to the Google Groups "Cukes" group.
> >>>>> To post to this group, send email to cu...@googlegroups.com.
> >>>>> To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
> >>>> --
> >>>> You received this message because you are subscribed to the Google Groups "Cukes" group.
> >>>> To post to this group, send email to cu...@googlegroups.com.
> >>>> To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
> >>> --
> >>> You received this message because you are subscribed to the Google Groups "Cukes" group.
> >>> To post to this group, send email to cu...@googlegroups.com.
> >>> To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
> >> --
> >> You received this message because you are subscribed to the Google Groups "Cukes" group.
> >> To post to this group, send email to cu...@googlegroups.com.
> >> To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
Jon Druse
David Chelimsky
I'm migrating our builds from Jenkins to Bamboo. I have separate jobs for running tests and parsing test results. The problem is, because cucumber exits with non 0 when tests are failing. As a result, Bamboo assumes (as would I) that non 0 exit codes mean that cucumber encountered an execution error and immediately stops the plan and displays "No failed tests found, a possible compilation error occurred."It's cucumber's job to find failing tests, and should therefore exit with 0. Now if your tests have syntax errors or missing gems, then it would be appropriate to exit with 1. In the example of running ls on a file that didn't exist, I would expect an exit status of 1. You tried to list a file that didn't exist, ls couldn't do it's job. Maybe I'm crazy but IMHO the exit status should reflect the execution of a program, not the result.
Here's a thought. What if I wrote a program called, find_errors. If it found errors, what should it's exit status be? 0 or 1?
Matt Wynne
On 24 May 2011, at 17:10, Jon Druse wrote:
> I have separate jobs for running tests and parsing test results.
That's your problem, and as David has said I think you'd struggle to get that to work with any test framework.
Either you need to configure Bamboo to run the second job regardless of whether the first job failed, or you need to make the test result parsing (whatever that means) part of the first job. Is there a Cucumber formatter for Bamboo that could use? Does Bamboo understand the standard JUnit XML test result format which Cucumber supports?
Failing that, just use a little bash-fu to ignore Cucmber's exit status in your first job:
cucumber features/fail.feature || :
Hope that helps.
cheers,
Matt
--
Freelance programmer & coach
Founder, http://relishapp.com
+44(0)7974430184 | http://twitter.com/mattwynne
aslak hellesoy
> I'm migrating our builds from Jenkins to Bamboo. I have separate jobs for
> running tests and parsing test results. The problem is, because cucumber
> exits with non 0 when tests are failing. As a result, Bamboo assumes (as
> would I) that non 0 exit codes mean that cucumber encountered an execution
> error and immediately stops the plan
If there were failing cucumber scenarios, why would you want to
continue the build?
> and displays "No failed tests found, a
> possible compilation error occurred."
I have no idea what logic Bamboo has to jump to that kind of
conclusion. Does Bamboo somehow know that when running cucumber, it's
running some kind of testing tool, and therefore looks for test
results (on disk) after it finishes? If that's the case, just make
Cucumber spit out results to disk using the junit formatter, and tell
bamboo where to find them. That should make Bamboo fail the build with
a proper error message - something like "There were 3 test failures."
> It's cucumber's job to find failing tests, and should therefore exit with 0.
It's Cucumber's job to *execute* tests, and leave commonly understood
artifacts on disk that any other tool can examine after execution,
whether the process exited with 0 or not. One such commonly understood
artifact is the output from the junit formatter.
You keep saying that Cucumber should always exit with 0. You're
advocating behaviour that goes against every other imaginable testing
tool or command line tool written in the spirit of UNIX.
From http://en.wikipedia.org/wiki/Exit_status:
"The specific set of codes returned is unique to the program that sets
it. Typically it indicates success or failure."
If we did what you say - exit with 0 when there are failing scenarios,
then there is no way to detect test failure from the exit status. Now
we have to *parse* cucumber's output to figure it out. If cucumber
decides to change the output format, all tools must be changed. We
have traded a standard, simple solution for a non-standard,
complicated and brittle one. Great!
> Now if your tests have syntax errors or missing gems, then it would be
> appropriate to exit with 1.
Yes, that will exit with a non-zero code.
> In the example of running ls on a file that
> didn't exist, I would expect an exit status of 1.
That too
> You tried to list a file
> that didn't exist, ls couldn't do it's job. Maybe I'm crazy but IMHO the
> exit status should reflect the execution of a program, not the result.
I don't think you're crazy, just wrong.
> Here's a thought. What if I wrote a program called, find_errors. If it
> found errors, what should it's exit status be? 0 or 1?
That's a contrived example, but I guess if you consider finding errors
a *good thing*, finding errors should return 0. If you consider
finding errors a *bad thing*, finding errors should return 1. Most
people consider finding errors a *bad thing*, and design their
programs that way.
To me it seems you're arguing that Cucumber should break exit status
conventions just because you don't know how to configure your CI tool,
or possibly because your CI tool is poorly implemented.
Have a try at telling Bamboo where Cucumber's junit XML files are -
that should solve your problem.
Aslak
Andrew Premdas
I'm migrating our builds from Jenkins to Bamboo. I have separate jobs for running tests and parsing test results. The problem is, because cucumber exits with non 0 when tests are failing. As a result, Bamboo assumes (as would I) that non 0 exit codes mean that cucumber encountered an execution error and immediately stops the plan and displays "No failed tests found, a possible compilation error occurred."It's cucumber's job to find failing tests, and should therefore exit with 0. Now if your tests have syntax errors or missing gems, then it would be appropriate to exit with 1. In the example of running ls on a file that didn't exist, I would expect an exit status of 1. You tried to list a file that didn't exist, ls couldn't do it's job. Maybe I'm crazy but IMHO the exit status should reflect the execution of a program, not the result.Here's a thought. What if I wrote a program called, find_errors. If it found errors, what should it's exit status be? 0 or 1?
--
George Dinwiddie
On 5/24/11 12:10 PM, Jon Druse wrote:
> but IMHO the exit status should reflect the execution of a program, not
> the result.
Run 'man test' on a Unix-ish system. You'll see that your assumption
that non-zero means the program crashed has never been the norm.
--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------