Best practice for testing content of CFM file or cfinclude

[Shannon]

I found an example of how to test the content of a CFM file or
cfinclude in the article "Test Driven Development with Coldfusion Part
II: Building Unit Tests for CFUnit" found here
http://www.fusionauthority.com/techniques/4568-test-driven-development-with-coldfusion-part-ii.htm,
written by Robert Blackburn.

His technique amounts to running the page in the browser and saving
the content as a text file, then comparing the template to this file.
Over time if something changes the page, you fix either the code or
regenerate the text file and rerun the test.

I want to do something similar to test the content of my Coldfusion
pages using MXUnit. CFUnit has an assertOutput() method. I might
write my own if no one else has one, but this is what I whipped up and
it seems to work:

<cfcomponent extends="mxunit.framework.TestCase" output="false">
<cffunction name="testTemplate" access="public" output="false"
returntype="void">
<cfsaveContent variable="expected">
<cfinclude template="index_Content.txt">
</cfsaveContent>
<cfsaveContent variable="actual">
<cfinclude template="/stuff/index.cfm">
</cfsaveContent>

<cfset assertEquals(expected, actual)>
<cfreturn />
</cffunction>
</cfcomponent>

My question: Does any one have any better suggestions on how you do
this? What do you use if you test your rendered output?

I just started using MXUnit yesterday, but I noticed all the test
samples are geared toward testing CFCs only. I'm in a shop that is
still a bit oldschool and does a lot of stuff via CFM, cfinclude, and
custom tags. So, I've got to try to find something that works on
existing code as I'm trying to get them to adopt continuous
integration.

I don't want to get into a discussion about why I am doing this, so
please, only those who have tried this or have an idea on how you
could do this should respond.

Thanks in advance for your help!

[Marc Esher]

I've tried to test output, and what's worked best for me has been to not explicitly do assertEquals() type stuff. it's been way too hard to maintain for me with large chunks of content.

Instead, I've opted for more of an assertOutput() or even a "findNoCase()" approach, or else a custom assertion sort of like "containsChunks()" where I just test for the existence of certain strings. for example, if I know that my code should contain a div with a message in it based on certain criteria, i'll test for the existence of that div in the html (with a regex) and possibly test some of the contents of the message.

I actually wrote a fairly sizable app a while back using httpunit and junit together to do just this. I didn't use MXUnit because I needed to carry on a "Web Conversation", not just test single cfm pages at a time. I wanted to simulate what a user would actually do in our site. This was definitely "integration" testing, and other, better tools now exist for doing this stuff... Selenium being one. I'm actually wondering if you'd be better off looking into a tool like that for what you're currently trying to do. 

I love MXUnit, but its strength is testing functions either in cfcs or cfm pages that contain functions). Bill's been working on integrating WebDriver into MXUnit to offer more powerful integration-testing features, but it's not there yet, and if you're trying to sell these techniques in a shop that heretofore hasn't been quick to adopt such things, then I think it's a mistake to go in with a low-level unit testing framework and expect people to accept it graciously.

This isn't to say that a unit testing framework doesn't have its place. In your case, if you've got libraries of custom functions lying around, then they're perfect candidates to get started with unit testing. I'd strongly advise NOT to start writing tests for your existing functions right away; rather, when functions get modified or added, write tests for them first. I am still amazed at how much time unit testing saves me. It's counter-intuitive (more typing = less time?), but it's true, for me at any rate. 

To sum: can you use mxunit for cfincluding your cfm pages... absolutely. Will it scale, or even be sustainable long term? I'm not so sure. Better tools currently exist.

I'd also like to say one other thing, and that is to your last point about your shop being old school. I imagine it's very frustrating, and I also bet that most of us have been there, or are still there. People change slowly. Don't feel like you have to change your shop's culture overnight... it's a road to pain and haters and it probably won't get you what you want unless you have at least one  manager pushing or rooting for you. If you're a loner, trying to make incremental change, start slow. Instead of doing things in .cfm files, start using .cfc files. don't worry about "being OO" if that worries your team. Just make your CFCs big old bags of functions (to start). If you can build up some early wins (hey, look, these things are easy to use and easy to test), well, that's a start.

Good question, Shannon. Good luck!

Marc

[Mark Mandel]

I tend to use Selenium for UI testing.

I find MXUnit and Selenium a wonderful combination together.

Mark

--
E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

[bill shelton]

Hi Shannon,

I wouldn't be the first to say this, but testing generated HTML is a
pain, especially volatile content. Still, there's a need. Your
approach would certainly work, but may be hard to maintain down the
road. You may want something more fine grained like assertXPath()
which addresses testing HTML (and XML). Check out the docs here and
scroll down: http://wiki.mxunit.org/display/default/Built-In+Assertions

Basically, it's this:
assertXPath(String xpath, any data, [String text], [String message])

where 'data' is any string or URL (http:// or file://). So, there
should be enough flexibility there to test HTML, if it's reasonably
well formed. Others on the list have much more experience with testing
custom tags, and I hope they might chime in, too.

Another_ completely_ different approach I've been discussing with
someone at work, is using some kind of a checksum. Now, this is may
seem odd, but bear with me. Essentially, a checksum could be computed
from certain critical portions of the page and then embedded in an
easy to find HTML comment. As you are rendering a page, you could
collect various values that you deem important and once collected
successfully, hash them and embed that in the page. In your test, you
take the generated content and locate the checksum.

Example (assuming database is in a stable state):

<cfoutput query="q">
<cfset variables.collected += q.id />
<div id="#q.id#">#q.name#</div>
</cfoutput>
...somewhere else on the page, like the bottom...
<cfset variables.checksum = hash(variables.collected) />
<cfoutput>
<!--@checksum=#variables.checksum#-->
</cfoutput>

Then in your test, you could use a regular expression to extract the
@checksum value and compare that to your actual value. The idea is
that the checksum would only be generated if the page successfully
rendered. It wouldn't get the whole page or every element, of course,
but might be a decent barometer.

best,
bill

On Jul 22, 6:12 pm, Shannon <smho...@gmail.com> wrote:
> I found an example of how to test the content of a CFM file or
> cfinclude in the article "Test Driven Development with Coldfusion Part

> II: Building Unit Tests for CFUnit" found herehttp://www.fusionauthority.com/techniques/4568-test-driven-developmen...,

[Bill Rawlinson]

I've done a little custom tag testing and I'll send some example code
to the list tomorrow

Sent from my iPhone

[Shannon]

Marc,

Thanks for all the suggestions. Shop has bought into Continuous
Integration and we are taking baby steps. :)

I've seen assertOutput in CFUnit, but not in MXUnit. Does this
exist? Or would I have to create my own?

[Shannon]

On Jul 23, 7:10 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> I tend to use Selenium for UI testing.
> I find MXUnit and Selenium a wonderful combination together.

Mark,

Do you only test the CFCs in MxUnit and everything else in Selenium
then? Or what do you task to MxUnit vs Selenium?

Thanks,
Shannon

[Marc Esher]

I've not used cfunit in a very long time, but isn't assertOutput pretty much the same thing as assertTrue(FindNoCase(...))  ?

[Shannon]

Bill,

Hmnn... I'll give a try to the assertXPath(). Checksum is
interesting, but seems a bit complicated and most of the code already
exists. I think we'll be writing Selenium tests for what is in
existence and look at how to unit tests for bugs we find or new code.

[Mark Mandel]

Pretty much.

All my model testing I do in mxUnit.

All my UI testing I do with Selenium. To a degree this means that I am doing my Controller testing with Selenium as well, but I find that works well for me, as essentially my Controller code is determining my UI logic flow.

When I use Selenium I tend to script a lot of it, so it is not as fragile as it can be - i.e. pull a value from a top of this datagrid, use that to edit this, concatenate this value, generate this random number etc.

Mark