beforeTests() raises exception where setup() simply raises an MXUnit error

[Adam Cameron]

G'day:
We're running MXUnit 2.1.3.

When an error arises in setup(), the test is simply marked as an "error" in the ensuing output (this is running via a browser), but the test round keeps going and runs the rest of the tests. When there's an error in beforeTests(), the exception is bubbled back, and the test round exits.

Is there any reason for this difference in behaviour in what is two very similar situations?

I note this old comment - https://groups.google.com/d/msg/mxunit/I4wr4DpoeuI/knEkbG7PZk8J - which might have something to do with it.

This is very annoying from the perspective of running through all our (2000+) tests, because if one breaks the testing stops. I can understand how the tests in the specific CFC should possibly be "skipped" (being reported back as an error or failure), but completely exiting seems counter to the way the rest of MXUnit runs?

Is there any way of configuring beforeTests() to not raise an exception if it has a problem? (short of hacking the code I mean, which is what I'll do if needs must...)

Cheers.

--
Adam

[Marc Esher]

As for the current behavior, setUp/tearDown happens at the method level, and so there's a clear "line of reporting" between a failure in setup/teardown and a test method. In other words, if a test method fails due to a failure in setUp, it's simple to assign that failure at the test method level b/c it all happens at the same try/catch block.

However, with before/after Tests(), that happens *before* methods are iterated and consequently when a failure occurs, the error could be attached to the testcase itself but it'd like -- currently, at any rate -- not surface because in all interfaces (web, ANT) the test method is the most specific item... testcase's have no data themselves except for a name and a collection of methods.

Thinking about how to address the behavior you describe -- which I agree is not ideal -- what if we'd wrap beforeTests() in try/catch and, if a failure occurs:

1) save that exception

2) iterate over all methods

3) fail all of those methods immediately using the error from beforeTests()

This way, we can both clearly identify the cause of the error (which would be duplicated amongst all methods in the testcase) and correctly surface the impact of that beforeTests() error.... if there are 10 methods in the testcase, then you'd see 10 test errors and not a single "dummy" error.

Thoughts?

Thanks for bringing this up, Adam.

Marc

--
Adam

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mxunit/-/hEdxfvGQ944J.
To post to this group, send email to mxu...@googlegroups.com.
To unsubscribe from this group, send email to mxunit+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mxunit?hl=en.

[Adam Cameron]

Hi Marc
Yeah... not this time around, but the last time I was caught out by this and wondered about hacking a fix together I came to the same conclusions you did, but didn't quite get to the point of contriving a good way of "telling" the test methods to just report an error.

But that's the way I'd do it, yes.

Thanks for getting back to me so quickly on this, mate.

--
Adam

[Marc Esher]

Thanks Adam.

Right now, I can't commit to a fix any time soon. Are you -- or anyone else on the list -- interested in submitting a fix and pull request?

I can tell you that the offending code is in TestSuiteRunner.cfc, in the run() method.

Marc

--
You received this message because you are subscribed to the Google Groups "mxunit" group.

To view this discussion on the web visit https://groups.google.com/d/msg/mxunit/-/bSn07L0GGoIJ.

[Adam Cameron]

Hi Marc
Sure, I've "fixed" it. Just need to decide that I like how I've fixed it.

Also need to do all the git shenanigans so as to be able to get it to you. Which is going to require me learning how to do more with git than just commit to my own "for safe keeping" repo.

But will get it sorted tonight.

Cheers mate.

--
Adam

[Marc Esher]

Sounds good, Adam. If you need any help on the git stuff, let me know. 

It should be:

1)  committing your changes locally

2) pushing to your github fork of mxunit

3) submitting a pull request to mxunit. 

Github will then perform the mechanics of notifying me, performing any diffs, etc so that we can pull it into the main repo.

Marc

--
You received this message because you are subscribed to the Google Groups "mxunit" group.

To view this discussion on the web visit https://groups.google.com/d/msg/mxunit/-/nk_Z6srNyjUJ.

[Adam Cameron]

Will do. This has not dropped off my radar, I've just been flat out with work this week (and forgot about it on the w/end). It also occurs to me that as well as fixing the code, there's probably some unit tests somewhere I need to augment, too..? ;-)

--
Adam

[Marc Esher]

On Tue, Jan 22, 2013 at 5:35 AM, Adam Cameron <adamcamero...@gmail.com> wrote:

Will do. This has not dropped off my radar, I've just been flat out with work this week (and forgot about it on the w/end).

I know the feeling!

 

It also occurs to me that as well as fixing the code, there's probably some unit tests somewhere I need to augment, too..? ;-)

Well.... this is definitely an area where coverage is... weak. We do have a single test in TestCaseTest.cfc for beforeTests(), but it is useless.

I'm thinking that for this new behavior, the most appropriate place would be a new test in TestSuiteTest.cfc which runs a test against a new fixture TestCase (put in the "fixture" directory). That testcase would just have a handful of empty methods and a beforeTests() which throws an error. 

The test would use TestSuite to run that fixture test, get the results, and confirm it had two test failures.

Thanks!

Marc

 

To view this discussion on the web visit https://groups.google.com/d/msg/mxunit/-/IHFTR4Fus5kJ.