Unit Profiling?

[Randy]

I'm starting to get more into trying to automate processes for my code
and make it a part of my workflow and I had a question about
profiling.

I've seen people blog about profiling with ColdFusion profiler and
server monitor, but I was curious if there was a way to code
ColdFusion components similar to the way that we do unit tests?

Is there a way to not only profile a block of code, but also store
those so that you can see a historical perspective of how much things
have slowed down or sped up from different revisions of the codebase?

Thanks,

Randy

[Marc Esher]

randy, i think this is one of the things you'd really want to not
roll your own on, but instead go the continuous integration route.
i've been investigating hudson and it's got good trending with respect
to time. check out blog.mxunit.org for my (currently 3-part) series on
hudson continuous integration server, ColdFusion, and MXUnit.

marc

[Randy]

Yeah, I would not be that excited to write my own complete profiling
app.

I perused the Hudson posts before I posted, but I didn't see anything
in there about actual profiling of the code. Does it tell how long it
took to run each unit test? But that's not really a good way to
profile some pieces of code is it?

I guess that you could setup a profile directory and write unit tests
that just do profiling inside the functions.

A lot of it is ignorance... I don't really know what there is that
could be used for this. If there was an application that already
handled the introspection of logs from profiling then at most we would
just have to build something that would run the functions to profile
and log it in the format the other application uses. Similar to how
MXUnit logs in a JUnit format... but I really don't want to create
anything extra if there is something that does it already.

Randy

On Jun 5, 3:53 pm, Marc Esher <marc.es...@gmail.com> wrote:
> randy,  i think this is one of the things you'd really want to not
> roll your own on, but instead go the continuous integration route.
> i've been investigating hudson and it's got good trending with respect
> to time. check out blog.mxunit.org for my (currently 3-part) series on
> hudson continuous integration server, ColdFusion, and MXUnit.
>
> marc
>

[Marc Esher]

Randy,
Do you want to see the global view of trending for all your tests
over time? you can easily get that from the main hudson screen. Though
I see now that I didn't put any screenshots up about that. I have a
post in the works -- probably the one i should've started with -- on
"why the hell would you even need this stuff?", which will cover my
own motivation for investigating hudson, and in fact it started with
time trending!

but i'm not so much concerned about 'does test 1 take longer today
than it took yesterday'. I'm concerned about "are all my tests
progressively taking longer to run", which would indicate some
problem. then i could presumably drill down into the tests to get at
that information. yes, you get the total time tests take when you dig
into the test results.

i'm not sure what other stuff is out there yet for getting
more/better/easier-to-read information with respect to time trending
in Hudson, but I'll definitely be digging into that b/c it's important
to me. I do know that continuous integration servers generally have a
mechanism for causing builds to fail when build times spike, but i
haven't implemented that in hudson yet so I don't know how it's done.

I'm curious: are you working alone, or on a team? I think for me,
this would influence whether I went down the CI route or whether I
just rolled my own test result app.

marc

[Randy]

Very good questions and points.

To start, I'll describe the position that I am in. Right now I am
starting work on a new project and trying to start off on the right
foot and get all the automation and build processes setup from the get-
go. As far as a team goes, I am working with a few people on the
project but most of the time it will be me doing most of the
programming. I would really like to have a CI server setup, atm I
don't know if that is going to be financially feasible to have running
all the time, can you setup hudson locally or be setup on an EC2
instance and just started and stopped when I want to use it? I don't
know how well hudson would do with being started and stopped.

I would be interested in the general unit testing times to look for
trends, but for some of the core code it would be nice to see it when
functions are run thousands of times, but I don't really know if that
would be worth the effort, or if I should just rely on the overall
tests.

So, really I would like to have a CI server for when things get
larger, then it will just be part of the process, and not have to be
rigged later on to a larger project.

Randy

[Marc Esher]

here's my quickie answer on hudson then i gotta split for a while:
hudson has extremely low overhead. You could literally install tomcat
on a box, drop hudson into it, schedule tests, and the only blip you'd
get would be whenever it ran your tests. I have it running on my
laptop all the time and I never notice it. At work, I'm about to
install it on one of our old "we're gonna pitch this box if noone
wants to use it" machines and I'm fully confident that it'll be fine.

in my case -- and it sounds like yours -- it's definitely not this
monolithic monster thing that's fetching sources from a thousand
different subversion repositories around the globe, compiling for an
hour, and running millions of tests. Rather, it's just an app that's
gonna make http calls to your tests and record the results. You could
probably run it on your desktop, and firefox will take up more memory
than hudson would most of the time.

Definitely don't be afraid of it in terms of server-hoggishness. It's no pig.

[marc....@gmail.com]

Hey Randy,

I updated the blog post here http://blog.mxunit.org/2009/06/continuous-integration-with-hudson-cf.html to show what the time trend graph looks like. I suspect that once you get it set up and running steady, the graph will even out and then be more meaningful as a tool because if things are going wrong, it'll creep up.

marc