Code coverage anyone?

[Raffaele Castagno]

Do you have notice of any code coverage tool for coldfusion, at least
experimental?
I've read some info around, but is mainly old stuff, so maybe there's
something new in the air...

Raffaele, hating the lack of tools for coldfusion...

[bill shelton]

Raffaele,

We were pursuing this effort pretty strongly about a year ago. We
spoke with some of the engineers with the engine providers about
interfacing with the various CF runtimes in order to instrument byte
code. However, as much as we pushed, we didn't get a whole lot of
cooperation or interest. It would be great to get a little help and
guidance from the engine providers to implement a code coverage tool.
From an engineering perspective, it would be an interesting effort,
and we would welcome any assistance, however little. I'd be happy to
speak with you or anyone else about this. It's something I'm
personally very interested in.

bill

On Sep 24, 5:46 am, Raffaele Castagno <raffaele.casta...@gmail.com>
wrote:

[Raffaele Castagno]

Some background.
I work for a company which uses ColdFusion for some of our top
products. I've not worked so much on CF (about 1 year in total) but
I've been involved in Java projects.

Now I'm called to work again on CF, and decided to give a broader look
at the situation of our projects, and I don't like what I've seen.

We score about 4 on the "Joel test". We don't have automatic tests, no
one-step builds, no best-practice enforcing, etc.

We have five main products, two of those have 50 customizations for
different customers, totalling about 11k cfc/cfm files, and the mess
is growing on.

There's an awful lot of duplicate code, and code that should be
theoretically equal which is not.


Now I'm doing my homework on my spare time, doing some research about
how to deal with legacy code in general, trying to figure out how to
create a believable plan of action, a plan I could propose to
management.

To achieve, and avoid my proposal to be simply trashed out, I have to
propose a solution requiring minimum effort.
That means that I need to "automate all moving parts", but also that I
need a way to measure improvements.

And here we come back to Code Coverage, among the others, since imho
is a necessary metric to measure effectiveness of test cases, and to
keep people focused on the task (ie. not only "get that test green"
but also "point to 100% coverage", to avoid "cheating").

I don't think I can do much more than being a user of such a tool, and
don't have the knowledge necessary. But if my proposal will find a
good answer in my company, I will for sure need such a tool, and will
absolutely give an hand.

About insight and guidance, there are few options:
- CF runtime providers: adobe, railo, bluedragon
- Companies relying heavily on CF and with an inclination to
innovation (Alagad)
- developers of frameworks/plugins based or dedicated to coldfusion
and testing (ColdBox, mach, cfcunit, cfunit, etc).

That, plus someone willing to lead the effort, would be enough man
power, I suppose.

My 0.02€

--
Raffaele

[Daryl Banttari]

I'd be interested. We know that CF at least compiles line numbers
into code, though i'm guessing that it'll be difficult or impossible
to do branch detection. (For those that haven't worked with coverage
tools, a compound IF statement [eg if(condition1 or condition2)]
wouldn't be fully "covered" unless you test all four permutations of
condition1|2 true|false.)

I wonder if something like Cobertura
(http://cobertura.sourceforge.net/) could be used if we pre-compile
classes? Perhaps they'd let us contribute.

I'll see if I can find some free cycles to play with this.

--Daryl

[Ilyas Patel]

I spent a lot of time looking into tools for code coverage a few
months and I think it was on Peter Bell's blog mentioning it could be
a project for the people that make Fusion Reactor.

The company I work for also has a lot of legacy code and duplication
which we hope to phase out someday. We recently embarked on a 3 month
project plan to push Continuous Integration. We are hoping the unit
tests and automated GUI tests (QTP) will provide enough confidence to
make these changes with confidence. We have completely stopped all
project development and spent time creating automated regression
tests, and setting up all our environments so the datasets fit the
criteria the test team have setup. I know most companies would be
reluctant to stop product development but its been such a huge step in
the right direction.

Code coverage would be the icing on the cake. One of the products I
looked at was Clover by Atlassian - a tool similar would be perfect.

Ilyas

[Raffaele Castagno]

Can you please tell me something about your effort?
I mean, how many people is involved? How many lines of code are you
refactoring?

Can I ask you the name of your company, and if there is a blog or
something where I can follow your progress?

Thanks a lot, and good luck!

Raffaele

[Raffaele Castagno]

On Sep 24, 3:18 pm, Daryl Banttari <dbantt...@gmail.com> wrote:

> I wonder if something like Cobertura
> (http://cobertura.sourceforge.net/) could be used if we pre-compile
> classes?  Perhaps they'd let us contribute.

I think Groovy/Grails folks may give some great opinion on this, since
there's support for Cobertura. Groovy/Grails is IMHO not so different
from ColdFusion, despite being far more powerful and integrated with
Java.

The only problem I can see is the missing of a 1:1 relationship
between cfc/cfm and generated classes, but is probably something that
can be arranged someway.

Cheers

Raffaele

[Raffaele Castagno]

I found this:
http://www.henke.ws/post.cfm/Adobe-MAX-2008-Top-Ten-List---Announcements

"6. Jason Delmore, the Product Manager for ColdFusion, hints
ColdFusion will leverage Java code coverage tools for developers
(maybe Cobertura or EMMA) in the next release. The ColdFusion coverage
reports will be produced in a number of ways and even an Eclipse
plugin to get directly to the code mentioned in the report."

But it's old stuff...

Regards

--
Raffaele


On Sep 24, 11:46 am, Raffaele Castagno <raffaele.casta...@gmail.com>
wrote:

[Raffaele Castagno]

On Sep 24, 3:18 pm, Daryl Banttari <dbantt...@gmail.com> wrote:

> I'd be interested.  We know that CF at least compiles line numbers
> into code, though i'm guessing that it'll be difficult or impossible
> to do branch detection.  (For those that haven't worked with coverage
> tools, a compound IF statement [eg if(condition1 or condition2)]
> wouldn't be fully "covered" unless you test all four permutations of
> condition1|2 true|false.)
>
> I wonder if something like Cobertura
> (http://cobertura.sourceforge.net/) could be used if we pre-compile
> classes?  Perhaps they'd let us contribute.
>
> I'll see if I can find some free cycles to play with this.

I've made a quick experiment with cobertura.
1) Create small CFC with a simple method, created small CFM page to
call the CFC.
2) Enabled "Save class files" in the administrator
3) Instrumented generated classes with cobertura-instrument
4) added cobertura.jar to the classpath, and required parameters to
the launch config
5) restarted CF8

Now, something is happening for sure, because without instrumentation
the page loads normally, but with instrumentation it loads forever and
ever and ever.

Let me know if you found something interesting.

Raffaele

[Marc Esher]

Raffaele,
  Mike's post was entirely in jest.

Marc

[Mike Rankin]

I have to throw an unsolicited opinion in here, more for my own piece
of mind than as any sort of contribution to the conversation. So here
goes:

Unless you're building heart pacemaker software or controlling the O2
supply on the first manned mission to Mars, chasing code coverage is a
big waste of time. You can have a tool run down 100% of your code
branches with every possible permutation of a poorly designed app that
doesn't do what the customer wanted and all you've done is prove that
you have built a poorly designed app that doesn't do what the customer
wants and took twice as long to find out.

If you really just want to know that you have a test, any test, for
each method in your app, you can probably do that with cfc metadata
and some naming standards. If you're really after code coverage where
all of your decisions are exercised, good luck.

Sorry for the rant, but code coverage just seems like extreme over-
engineering otherwise known as "tilting at windmills".

On Sep 25, 7:36 am, Raffaele Castagno <raffaele.casta...@gmail.com>
wrote:

[Raffaele Castagno]

On Sep 25, 3:49 pm, Mike Rankin <anodynep...@gmail.com> wrote:
> I have to throw an unsolicited opinion in here, more for my own piece
> of mind than as any sort of contribution to the conversation.  So here
> goes:

Every consideration is welcome!

> Unless you're building heart pacemaker software or controlling the O2
> supply on the first manned mission to Mars, chasing code coverage is a
> big waste of time. You can have a tool run down 100% of your code
> branches with every possible permutation of a poorly designed app that
> doesn't do what the customer wanted and all you've done is prove that
> you have built a poorly designed app that doesn't do what the customer
> wants and took twice as long to find out.

I've never used Code Coverage tools, really, and I don't consider them
an unique measure of code quality. I think of code coverage tools as a
measure of what has been done, and what has to be done.

> If you really just want to know that you have a test, any test, for
> each method in your app, you can probably do that with cfc metadata
> and some naming standards. If you're really after code coverage where
> all of your decisions are exercised, good luck.

Nope, I simply consider CodeCoverage as a step for my "plan" for the
refactoring of our legacy code.
- identify a piece of code needing refactoring
- cover it with test
- when satisfying level of coverage is reached, refactor

Code Coverage would be useful for the "satisfying level of coverage"
part.
It's obvious that coverage alone cannot be an unique measure of code
quality.

> Sorry for the rant, but code coverage just seems like extreme over-
> engineering otherwise known as "tilting at windmills".

I see it simply as an additional information, useful to have better
understanding of what's going on.

Raffaele

[Raffaele Castagno]

On Sep 25, 3:34 pm, Marc Esher <marc.es...@gmail.com> wrote:
> Raffaele,
>   Mike's post was entirely in jest.

uhoh...I read it superficially...

Raffaele

[bill shelton]

I think like anything, code coverage can be used in such a way as to
be misleading. From my inexperienced view point, I see it as a quick
barometer and let's you know if you simply missed writing a "basic"
test for something. To perform "True Coverage" is impractical, if not
impossible. And by True Coverage, I mean exercising every line in your
production code with every possible permutation of input data. This
Cartesian product of input data would take hours, days, months, or
even years for even the most basic of tests. The one concept that
makes sense to me is writing tests that apply "Coverage Criteria" to
your code. This drives you to think about your tests a bit more deeply
than just to write the Sunny Day path through a series of
conditionals. In more practical terms, for me, this means articulating
what the valid range of input data should be and testing both within
that range and beyond it. Still, I would love to have code coverage to
alert me, to say "Hey, bonehead. You forgot to exercise this chunk of
code!"

Of course, you're welcome to say f*** it and just ship the thing and
be all cool and bad ass and shit ;-)
http://www.joelonsoftware.com/items/2009/09/23.html


bill

On Sep 25, 11:03 am, Raffaele Castagno <raffaele.casta...@gmail.com>
wrote:

[Raffaele Castagno]

On Sep 26, 10:37 am, bill shelton <vir...@gmail.com> wrote:

> Of course, you're welcome to say f*** it and just ship the thing and
> be all cool and bad ass and shit ;-)http://www.joelonsoftware.com/items/2009/09/23.html

Agree with what you said before, but I do not agree much with the
story of the "duct tape programmer", because that mentality is the
exact CAUSE of the mess I'm dealing with now (i.e. "just ship it, and
who cares if in a few months we will come up with an unmaintainable
mess").

I'll go on reading and applying "Working effectively with legacy
code", it seems more reasonable to me. I'll probably read also "Coders
at work", but I really dislike the "duct tape methodology".

Raffaele

[Sean Corfield]

On Sat, Sep 26, 2009 at 4:28 AM, Raffaele Castagno
<raffaele...@gmail.com> wrote:
> Agree with what you said before, but I do not agree much with the
> story of the "duct tape programmer", because that mentality is the
> exact CAUSE of the mess I'm dealing with now (i.e. "just ship it, and
> who cares if in a few months we will come up with an unmaintainable
> mess").

There's a VERY important caveat right at the end of Joel's piece:

"Duct tape programmers have to have a lot of talent to pull off this shtick."

That's pretty much true of anyone who - based on lots of experience
and lots of innate talent - knows when to cut corners. This is one of
the big problems with a lot of what Joel writes: he's not talking
about the millions of average-to-good programmers who make the world
go around, he's only talking about the select handful of really,
really talented guys who enable the leaps-and-bounds advances in our
field.

Scott Meyers and I were talking years back about types of programmers
and he said something I'll never forget: in his opinion, there are
three types of programmers: "public" programmers who write
applications, "protected" programmers who write libraries and help
"public" programmers, and "private" programmers who write operating
systems and compilers and can mentor the "protected" and "public"
programmers. And those "private" programmers are born, not made -
their talent is innate: they have to program because it's in their
blood. They started young, they generally know a *lot* of different
programming languages and they write code for fun. Scott said the
ideal situation would be for every company to have a "private"
programmer (or two), several "protected" programmers and a good, solid
team of "public" programmers (but in reality there are too few
"private" programmers and they often congregate at "cool" companies).
Scott Meyers wrote the Effective C++ series (seminal books on C++ best
practices).

A lot of people I've talked to over the years don't like Scott's
opinion because they want to believe all programmers have the
potential to become excellent. My experience doesn't support that
belief, I'm afraid :(

There's also a flip side to this - and very important advice it is too
in my opinion - know what you don't know and have the strength to stay
within your comfort zone when there's a lot of risk involved with a
project. In other words, don't try to learn a lot of new stuff on a
big, important project! Learn it little by little on smaller, less
critical projects. Walk before you can run.

In the CF community, a lot of people have unrealistic expectations
about how quickly they can master - or even pick up - a lot of the
"cool" techniques that are bandied around. Not everyone can be a
"private" programmer. Not everyone picks stuff up quickly. And don't
be discouraged because "this stuff is hard" - for 'stuff', substitute
pretty much any of the techniques touted by a small, vocal group of
programmers worldwide are advocating, whether that's Agile processes,
OO, Functional Programming, Concurrent Programming... All these things
are hard to master and you almost certainly won't get to any level of
competence on your own (unless you're born a gifted programmer and
even then you need to work with teams to find out what does and what
does not work in group dynamics).

Hmm, that turned into a bit of a philosophical rant... blame it on
being the first email I wrote after falling out of bed on a Saturday
morning!
--
Sean A Corfield -- (904) 302-SEAN
Railo Technologies US -- http://getrailo.com/
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[bill shelton]

Sean

You are more than welcome to fall out of bed and rant here - that was
some good stuff! Now, if we can just find some love to build a code
coverage tool the world would be better place :-)

bill

On Sep 26, 3:14 pm, Sean Corfield <seancorfi...@gmail.com> wrote:
> On Sat, Sep 26, 2009 at 4:28 AM, Raffaele Castagno
>

> Railo Technologies US --http://getrailo.com/
> An Architect's View --http://corfield.org/

[Raffaele Castagno]

On Sep 26, 9:14 pm, Sean Corfield <seancorfi...@gmail.com> wrote:
> On Sat, Sep 26, 2009 at 4:28 AM, Raffaele Castagno
>

> <raffaele.casta...@gmail.com> wrote:
> > Agree with what you said before, but I do not agree much with the
> > story of the "duct tape programmer", because that mentality is the
> > exact CAUSE of the mess I'm dealing with now (i.e. "just ship it, and
> > who cares if in a few months we will come up with an unmaintainable
> > mess").
>
> There's a VERY important caveat right at the end of Joel's piece:
>
> "Duct tape programmers have to have a lot of talent to pull off this shtick."

I thought the same thing minutes after posting that: I don't think I'm
smart enough to be a decent "duct tape programmer", and probably
nobody in my company can. Probably, we are some kind of "patchwork
programmers" :)

> A lot of people I've talked to over the years don't like Scott's
> opinion because they want to believe all programmers have the
> potential to become excellent. My experience doesn't support that
> belief, I'm afraid :(

That's not my case. Excellence is not for everyone, including myself.

> There's also a flip side to this - and very important advice it is too
> in my opinion - know what you don't know and have the strength to stay
> within your comfort zone when there's a lot of risk involved with a
> project. In other words, don't try to learn a lot of new stuff on a
> big, important project! Learn it little by little on smaller, less
> critical projects. Walk before you can run.

I think it's enough to say that I believe in the "Teach Yourself
Programming in Ten Years" way of life:
http://norvig.com/21-days.html

> In the CF community, a lot of people have unrealistic expectations
> about how quickly they can master - or even pick up - a lot of the
> "cool" techniques that are bandied around.

That's the big curse of "low entry level" languages: everyone can
start writing stuff with them without knowing what are they doing.
The problem is that nobody cares, here...
So they are keeping writing lines of codes without caring for quality.

Regards

Raffaele

[bill shelton]

Regarding excellence, I think that's proportional to desire, energy,
plan, and execution. I think you need a good combination of _all_ that
in order to develop excellence. Of course, there are the true naturals
who were compiling Linux kernels between classes in the hallways of
high school.

Cedric Beust, TestNG, posted his response last night to the Spolsky's
duct tape paradigm. Cedric also cites Mike Brunt's post -
http://beust.com/weblog/archives/000519.html

judge for yourself ...

peace,
bill

On Sep 27, 5:26 pm, Raffaele Castagno <raffaele.casta...@gmail.com>
wrote:

[Ilyas Patel]

Apologise for the late response Raffaele.

There are about 16 of us in the team including developers, testers,
project management, DBAs and IT support. We were split into 4 teams:

- Test team
- Environments team
- Configuration Management
- Software Lifecycle

Test team were responsible for creating a regression pack and then set
out to create automated tests. This is still an ongoing effort but we
should hopefully be in a position soon where the test team can
concentrate on any manual tests which can't be automated and being
able to spend time planning before a sprint.

We use to be in a position we're all our environments were slightly
different and the data didn't reflect the sort of data that was in
production. The test team would have to also spend time creating new
data for every release. Now, all our environments are identical and we
have a master copy of the database so we can replace any data that has
changed during a release cycle with a known dataset. The environments
we have are also identical in terms of hardware and software.

The configuration management team were further broken down into more
specific tasks – change the code base so it is more configurable,
sorting out a process for source control, look into areas for
automated testing and an aim to introduce unit testing to the entire
team. This team was also responsible for creating ANT tasks so we can
automate deployment to any environment, restart CF services, re-
initialising ColdSpring, the list goes on and on 

The project managers were responsible for fine tuning the release
process.


Hopefully once the project is finished we will be in a position to
remove redundant code, legacy code, run the automated tests and be
confident nothing has broken.

It’s a bit difficult to say how many lines of code need refactoring
but we currently maintain two versions of the same site. The newer
version of the site has a better user interface and is a lot more
reusable as components have specific tasks. We are aiming to phase out
the old code base.

I think we’re lucky here that everyone is committed to learning new
techniques and improving their programming skills. The management/
business has supported the project above even though it has meant
product development has stopped. They have been sold on the long term
benefits of such a project. Maybe you need to try the same.



On Sep 24, 2:50 pm, Raffaele Castagno <raffaele.casta...@gmail.com>
wrote:

[Raffaele Castagno]

On Sep 28, 6:44 pm, Ilyas Patel <ily...@gmail.com> wrote:
> Apologise for the late response Raffaele.
>
> There are about 16 of us in the team including developers, testers,
> project management, DBAs and IT support. We were split into 4 teams:

Oh god, I need an army!

Anyway, it's about what I was thinking about:
1) automate
2) cover with test
3) refactor with confidence

Thanks for wharing! Do you have any kind of blog with gotchas? You
must have tons of interesting stuff to write! :D

I think I'll have an hard time with my folks...

Thanks!

Raffaele

[Ilyas Patel]

No blog yet - I'm still quite a newbie to all this myself but it will
certainly be one of my goals for the near future.
.

On Sep 28, 7:12 pm, Raffaele Castagno <raffaele.casta...@gmail.com>
wrote: