Test Suites?
Sean Corfield
In cfcUnit - which is how all my tests are written so far - I can
write a test suite object (that extends org.cfcunit.Object) and define
a suite() method that creates and populates a TestSuite object and
returns it.
You can then specify that object to the runner, just like a regular
TestCase and the runner figures out how to get the suite and run it
(as opposed to just running the object as a test case). This allows
you to write test cases and then have an AllTests.cfc that acts as a
suite to run all the test cases. Then you can have hierarchical test
suite objects that pull in the sub test suites etc.
According to the MXUnit docs, you can only use test suites by writing
a .cfm page that creates, populates and runs the test suite directly.
Is this another case of me missing some MXUnit functionality or is the
way I have all my test cases and test suites just not possible with
MXUnit?
Sean
billy
We're still mulling these questions over and will get back you with a
firm answer soon. Thanks for your patience.
bill
billy
"reasoning" (and I use this term very loosely) about it, I'm sure this
would have been put to bed last night. But it's difficult to actually
_be_ multi-threaded. Anyway ...
Sean, again, you are not mistaken. MXUnit's TestSuite does not support
adding of other TestSuites. This has been on my radar since early on,
and as was stated in the JUnit 3.8 Cook's Tour, "We want to support
suites of suites of suites of tests." But, we have not implemented
that ... yet. So, (1) I can see that we will consider refactoring of
TestSuite in order to support this, and (2) the html runner we will
build will then determine if it is to run a TestSuite, a TestCase, or
a directory of tests. For a TestSuite I can see something like this:
<!--- Path for this: myapp.tests.MyTestSuite --->
<cfcomponent extends="mxunit.framework.TestSuite">
<cfscript>
suite1 =
createObject("component","mxunit.framework.TestSuite").TestSuite();
suite1.addAll("myapp.tests.testCase");
suite2 =
createObject("component","mxunit.framework.TestSuite").TestSuite();
suite2.addAll("myapp.tests.otherTestCase");
suite3 =
createObject("component","mxunit.framework.TestSuite").TestSuite();
suite2.addAll("myapp.tests.sub.someOtherTestCase");
this.add(suite1);
this.add(suite2);
</cfscript>
</cfcomponent>
Now, we should be able to call the runner something like:
http://localhost:8500/mxunit/runner/myapp.tests.MyTestSuite
This is a nice challenge and will provide some good refactoring
practice. MXUnit is doing a lot of things, not the least of which is
work that pays the bills ;-), so, this likely will not be in the next
build, but I hope we can have it in the next major release... As I get
closer, I'll post a release schedule for this.
In the interim, the Eclipse Plugin is _way_ sweet for loading
directories of tests and selectively targeting the ones of importance
to run. IMO, this saves you the time of actually having to write a
TestSuite. But, you still may want a way to persist these tests ...
Thanks again for your comments!
best,
Sean Corfield
> So, (1) I can see that we will consider refactoring of
> TestSuite in order to support this, and (2) the html runner we will
> build will then determine if it is to run a TestSuite, a TestCase, or
> a directory of tests. For a TestSuite I can see something like this:
Cool.
> suite1 =
> createObject("component","mxunit.framework.TestSuite").TestSuite();
> suite1.addAll("myapp.tests.testCase");
...
> this.add(suite1);
Would this support suites containing suites? That's really what is
needed to organize tests properly (and something cfcUnit has always
provided).
> Now, we should be able to call the runner something like:
> http://localhost:8500/mxunit/runner/myapp.tests.MyTestSuite
Don't forget about folks with more complex setups that already use
IIS/Apache and URL rewriting. It would be nice to support this:
http://localhost/mxunit/runner/?test=myapp.tests.MyTestSuite
and also:
http://localhost/mxunit/runner/
which should display a simple form with a text box to enter the test
path (and it can just do a GET instead of POST so it degrades to the
former case).
> In the interim, the Eclipse Plugin is _way_ sweet for loading
> directories of tests and selectively targeting the ones of importance
> to run.
Agreed. I'm more interested in the ant task and use that to automate
things. The fact that it can recursively scan directory trees is very,
very nice.
I just prefer to have a single test suite that programmatically
"includes" all my other test suites (which in turn "include" all the
specific test cases).
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood
Marc Esher
doesn't. Nonetheless, my stance on test suites in CF -- based on a
long history of using test suites in junit -- has been to ignore them
entirely, if not hold them in outright contempt.
For me, I think they're too much work. I write a *lot* of tests. I
code pretty fast (too fast probably). But when I'm in the groove, the
last thing I want to think about is adding my new test to a suite.
Because there are going to be times when I flat out forget. and then I
have my automated tests running, and things are peachy, and little do
I know that my new test(s) aren't in the automated suite because I
forgot to add them using suite.add(MyNewTest.cfc) or whatever. This
always annoyed me about JUnit. Sure, eclipse provides the "rebuild
suite" stuff; but I'd rather not have to worry about it at all.
That's why, for me any way, the DirectoryTestSuite is how I work. I
prefer to let my directory structure denote suites. If I have a
directory of tests, and subdirs of tests underneath that, well then I
must've organized them that way for a reason. So I just stick a
run.cfm file in the top-level of my test directory, copy the contents
of the DirectoryTestSuiteSample.cfm (or "ScheduledRun.cfm"), and
that's the end of that. Automatic test suite that I can create once
and never have to worry about.
Now, I'm one of those "bad testers" who more half the time will skip
the mock/stub route and go straight to the "integration" test. I know
that's how I test and so I have a directory structure I use to reflect
that style. But, the problem is that integration tests are slow. So
recently I've taken to a structure like so:
--tests
run.cfm
--Integration
--SomeSubDir
SomeTest.cfc
SomeOtherTest.cfc
--OtherSubDir
BlahTest.cfc
--Unit
--SomeSubDir
SomeTest.cfc
SomeOtherTest.cfc
--OtherSubDir
BlahTest.cfc
And so let's say I want my unit tests to run once an hour and my
integration tests to run once a day (or whatever). My scheduled task
for the Integration tests passes a URL param, like
"excludes=Integration", and then the run.cfm file recognizes it,
passes the excludes param to the directorytestsuite object, and then
all the integration tests are ignored. for the integration tests, i
pass "excludes=Unit" or just run the unit tests along with them.
That's a reduced example, but that's the idea anyway.
In my mind, "test suites are dead... long live test suites".
Now, quite likely a lot of people will not like this approach. The
"excludes" approach works great for me, mostly because I don't have
any need at all to "cherrypick" which tests get run at any given time.
I use excludes primarily to exclude entire directory trees such as in
the case described above. But I'll take "run unless excluded" over
"ignore unless included", which appears to be how testsuites operate,
any day. And you can use "excludes" to ignore both entire directories
and specific files. But no, the ANT wildcard syntax isn't supported.
it's pretty rudimentary.
Now, this excludes approach is just the beginning, I'd say. One thing
I really, really like is how TestNG uses the notion of "groups"
("slow", "fast", blah blah blah). I was thinking for a long time about
the notion of "contexts", which is that you might want to indicate,
via attributes on a test method, which contexts you want to run a test
in. but when I saw TestNG, I thought "yeah, that's the way to do it...
not this 'context' stuff". But that's for another discussion.
Thanks for the stuff to think about, Sean.
Marc
M Steele
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
Marc Esher
A) rad.
B) big time. Will it require CF8 or will it work on 6.1+?
Marc
M Steele
billy
>Would this support suites containing suites? That's really what is
>needed to organize tests properly (and something cfcUnit has always
>provided).
A GoF:Composite flavor, might work.
best,
bill
Sean Corfield
> For me, I think they're too much work. I write a *lot* of tests. I
> code pretty fast (too fast probably). But when I'm in the groove, the
> last thing I want to think about is adding my new test to a suite.
My TestSuites almost always add TestCases by wildcard from a
subdirectory. This gives me the functionality of your
DirectoryTestSuite but in the same homogeneous form as a TestCase.
The way MXUnit currently works is that there are TestCases and there
are CFML files that invoke tests which is really kind of ugly...
Marc Esher
Thanks for your thoughts. to your point, yes, that's correct. I'm not
going to speak for bill (the framework is his baby), but as a user of
the framework, i appreciate the no-nonsense approach to running
batches of tests. I suppose someday I'll mature enough as a programmer
to care about the distinction between running tests in a cfc vs
running them in a cfm page; right now, I don't. With the way I use
unit tests, which is "There they are, go run em", I do not see the
benefit. I'm not speaking for you or anyone else... just me. But if
you're willing to teach, I'm willing to learn.
Marc
Sean Corfield
> you're willing to teach, I'm willing to learn.
The main benefit is that it allows you to easily reorganize your
tests, moving them into different directory structures as the suites
grow and grow, without having to change how they are run.
Typically I have a tests directory for the entire project and it
contains a test suite that pulls in test suites for subsections of the
project. Each of those subsections contains a test suite that pulls in
test cases for that section. This allows me to easily run the entire
suite or just the tests for a specific subsection. Since test suite
CFCs and test case CFCs behave interchangeably from cfcUnit's point of
view, I can easily move tests around and just change the suite CFC
that pulls in those tests. I can also swap a single test case out and
replace it with a suite if that section grows beyond a single test
case.
Being able to run a single test or a single suite of tests (or the
entire suite for a section or the entire suite for a project) all
using the exact same approach (run a CFC) is very convenient when you
have a large project and you like to keep your tests organized. It
also means you can keep you primary test runner out of the webroot
(since it is a CFC, not a web accessible CFML file).
Hope that clarifies some?