MXUnit ANT script & Jenkins: tests fail but build succeeds... ?

[Adam Tuttle]

Hi all,

Hopefully someone here can help clear things up. I'm using the MXUnit ANT task to run my tests via Jenkins and even though some tests are failing, the ANT script seems to indicate back to Jenkins that everything's fine, so the build is marked as successful...

http://fusiongrokker.com:8080/job/Taffy/161/console

Can anyone tell me what I'm doing wrong?

Thanks,

Adam

[Jim Priest]

Maybe this?

http://wiki.mxunit.org/display/default/Continuous+Integration+--+Failing+builds+for+failing+tests

Jim

[Adam Tuttle]

Perfect, thanks!

http://fusiongrokker.com:8080/job/Taffy/163/console

Adam

Jim Priest

March 26, 2013 9:24 AM

Maybe this?

http://wiki.mxunit.org/display/default/Continuous+Integration+--+Failing+builds+for+failing+tests

Jim

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mxunit+un...@googlegroups.com.
To post to this group, send email to mxu...@googlegroups.com.
Visit this group at http://groups.google.com/group/mxunit?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Adam Tuttle

March 26, 2013 9:06 AM

Hi all,

Hopefully someone here can help clear things up. I'm using the MXUnit ANT task to run my tests via Jenkins and even though some tests are failing, the ANT script seems to indicate back to Jenkins that everything's fine, so the build is marked as successful...

http://fusiongrokker.com:8080/job/Taffy/161/console

Can anyone tell me what I'm doing wrong?

Thanks,

Adam

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mxunit+un...@googlegroups.com.
To post to this group, send email to mxu...@googlegroups.com.
Visit this group at http://groups.google.com/group/mxunit?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Gavin Baumanis]

I feel obliged to ask why isn't there a Chuck Norris plugin included here?

Surely everyone knows that Chuck Norris makes EVERYTHING better, don't they?

Gavin.

[Harry Klein]

Chuck Norris can write multi-threaded applications with a single thread.

 

Cheers,

Harry

[Marc Esher]

Jim, thanks a lot for following up on this so quickly!

[Adam Tuttle]

Hey all,

Is there any way to get the ANT output to include any more detail of failed/erroring tests? I know I can view the xml or html output, but I keep the whole build & output in a non-web-accessible directory for security reasons, so doing so requires me to RDS onto the build server.

It's not the end of the world if the answer is no, I'm just wondering if maybe it's a yes? :)

Adam

Marc Esher

March 27, 2013 7:52 AM

Jim, thanks a lot for following up on this so quickly!

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mxunit+un...@googlegroups.com.
To post to this group, send email to mxu...@googlegroups.com.
Visit this group at http://groups.google.com/group/mxunit?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Harry Klein

March 27, 2013 5:57 AM

Gavin Baumanis

March 26, 2013 8:10 PM

I feel obliged to ask why isn't there a Chuck Norris plugin included here?

Surely everyone knows that Chuck Norris makes EVERYTHING better, don't they?

Gavin.

On Wednesday, March 27, 2013 3:52:24 AM UTC+11, Adam Tuttle wrote:

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mxunit+un...@googlegroups.com.
To post to this group, send email to mxu...@googlegroups.com.
Visit this group at http://groups.google.com/group/mxunit?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Adam Tuttle

March 26, 2013 12:52 PM

[Marc Esher]

Adam, do you still need this?

[Adam Tuttle]

It's a nice to have, nothing urgent, but it's yet to be satisfied. :)

Adam

Marc Esher

April 12, 2013 5:47 AM

Adam, do you still need this?

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mxunit+un...@googlegroups.com.
To post to this group, send email to mxu...@googlegroups.com.
Visit this group at http://groups.google.com/group/mxunit?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Adam Tuttle

April 11, 2013 1:31 PM

[denstar]

On 4/15/13 10:32 AM, Adam Tuttle wrote:
> It's a nice to have, nothing urgent, but it's yet to be satisfied. :)

Anything specific?

I've been digging into the Ant task for some other stuff... we're
looking at a mini release for the non-breaking changes, and a bigger
release for the potentially breaking changes (might switch to the latest
version of httpclient while I'm at it for that one).

So far I've only been working with the Ant task itself, and there's only
so much exposed there-- what you're after (I'm assuming maybe a list of
what tests failed, etc.?) will probably entail updating the
HttpAntRunner.cfc file, at the least...

I'm also playing with the idea of optionally using JSON instead of a
webservice for the runner, as it seems like certain errors are
un-catchable when using a webservice... but that's still in the "I
wonder how feasible that is" category. =]

Anyways, now's a good time to holler if there's something you'd like to
see relating to the ant task. No promises, but it certainly won't hurt.

:DeN

--
A good man can make you feel sexy, strong and able to take on the
world...Ohh sorry that's wine.... wine does that... - unknown

[Adam Tuttle]

At a minimum, a list of which tests are failing would be great. Next thing I'd be looking for is the stack trace. And finally, if possible, the debug output -- though this would be less usable in the ANT/Jenkins Console output because it's a CFDump.

Adam

[denstar]

On 4/16/13 11:41 AM, Adam Tuttle wrote:
> At a minimum, a list of which tests are failing would be great. Next thing
> I'd be looking for is the stack trace. And finally, if possible, the debug
> output -- though this would be less usable in the ANT/Jenkins Console
> output because it's a CFDump.

We can't do too much with the results, as they're communicated via a
cookie I think, and there's not much space in cookies. (We're talking
about the results from the ant task here, vs. the actual test results
XML, which is streamed directly to the client and not parsed by the ant
task itself, due to memory concerns when running hundreds/thousands of
tests.)

Even a list of failing tests could quickly fill up what space is
available (request headers are limited to ~8kb in general), unless we
switch to a different method of communication... (not an easy tweak =]).

It's surely easier, currently, to run your own summary on the test
results XML files, than to try to get it out of the task itself.

That's one thing that is very nice about the junit plugin for Jenkins,
as it provides all you're after. I've sent you a link to how that
looks, BTW.

Also, I'm like some kind of savANT (ha!), and can share some of the
tricks I've come up with-- for example, when there's an error with a
test run, my script loops through all the logs and outputs what is in
them (last X lines) to the build log, so you can see what is going on
from the build results vs. looking into log files manually-- and I have
oodles of stuff like that, because I'm criminally lazy. ;]

:Denny

--
"HAVE PATIENCE" is the favourite word of LAZY PEOPLE
- unknown

[Jim Priest]

Share please! :)

On Tue, Apr 16, 2013 at 8:43 PM, denstar <vallia...@gmail.com> wrote:

Also, I'm like some kind of savANT (ha!), and can share some of the
tricks I've come up with-- for example, when there's an error with a
test run, my script loops through all the logs and outputs what is in
them (last X lines) to the build log, so you can see what is going on
from the build results vs. looking into log files manually-- and I have
oodles of stuff like that, because I'm criminally lazy. ;]

:Denny

--
"HAVE PATIENCE" is the favourite word of LAZY PEOPLE
- unknown

[Marc Esher]

Adam, we have two types of output in play here: the very simple summary output that is returned by the ant task via a couple of properties, and the junit XML. Are you saying that you want more information in the simple properties, or in the XML?

Or, perhaps put a better way: where do you want to see more detail? 

Thanks,

Marc

On Thu, Apr 11, 2013 at 1:31 PM, Adam Tuttle <ad...@fusiongrokker.com> wrote:

[Adam Tuttle]

I want to see more detail in Jenkins. The link Denny sent me that shows off the junit plugin seems promising... I just haven't had time to dig into it much yet.

Marc Esher

April 17, 2013 5:34 AM

Adam, we have two types of output in play here: the very simple summary output that is returned by the ant task via a couple of properties, and the junit XML. Are you saying that you want more information in the simple properties, or in the XML?

Or, perhaps put a better way: where do you want to see more detail? 

Thanks,

Marc

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mxunit+un...@googlegroups.com.
To post to this group, send email to mxu...@googlegroups.com.
Visit this group at http://groups.google.com/group/mxunit?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Adam Tuttle

April 11, 2013 1:31 PM

[denstar]

On 4/16/13 7:23 PM, Jim Priest wrote:
> Share please! :)

You asked for it! :)

This is the macrodef which does a server run:

https://github.com/denuno/cfdistro/blob/master/server.xml#L115

It's abstracted, so it will work with whatever server you're using
(jetty,jboss,etc.), and the "stuff" element basically passes through
whatever ant stuff you want to run while the server is running.

As is evident, the "stuff" is wrapped in a try/catch. Stuff could be
your mxunit call, like so for example:

https://github.com/denuno/cfdistro/blob/master/testing/mxunit.xml#L53

If the tests fail, for instance, that will be caught in the server-run
catch block, and the server-log-tail macrodef will be called to output x
lines from the logs in the logs directory.

Finally, the server is shut down, and a fail fires if there was err:

https://github.com/denuno/cfdistro/blob/master/server.xml#L129

As you can see, conceptually it's pretty straightforward. There are
more gory details which I've abstracted away here, like killing runaway
server processes (feel free to see (or ask) how that is tackled) and the
various ways of starting servers, but this should demonstrate the power
(and familiarity!) of Ant. Macrodefs are like custom tags, and a suite
of them makes even complex builds easy.

The try/catch and some other bits come from ant-contrib, which is swell.

Another nice one is XMLTask, which can easily be used to output a list
of failed tests, replete with stacktrace, etc.. I'll add an example of
how to do that in the mxunit file I linked to above.

:den

--
Going into the unknown is how you expand what is known.