Best way to track/monitor durring load testing
Brook
This question is fusionreactor and analytics related. I am planning
some load testing and I want to know what I should be watching for
durring load testing and if there are any settings (recording logs
etc) that I should enable to make sure I get the most out of the load
testing and am able to go back through the test result periods and
find bottlenecks etc.
Any pointers?
David Stockton
Thanks for your great question.
The first step would be to make sure you've got your FR+FA environment
setup and FR is pushing data to FA for analysis. Make sure this is
working before you do your load testing.
In FusionReactor, you should ensure that the JDBC wrapping process is
completed so that FR can track DB metrics. Here's some helpful links
regarding that process:
http://www.fusion-reactor.com/fr/helpdocs/jdbc_driver_wrapper_userguide.pdf
http://www.fusion-reactor.com/fr/helpdocs/jdbc_tutorial.pdf
http://www.fusion-reactor.com/community/developers/autodev_detailed.cfm?article=FRS-234
That last link is a little CFM tool to help automate the wrapping
process.
Once you're collecting JDBC metrics, make sure they're being logged to
disk ("JDBC -> JDBC Settings" page, then enable the "JDBC Logging (Log
Files) -> JDBC Logging" setting if not already done so).
Also, for the best coverage, also install the FusionReactor extensions
for ColdFusion (FREC) plugin: http://www.fusion-reactor.com/fr/plugins/frec.cfm
Once you've installed the FREC software and then deployed it to your
CF instance, make sure you see the ColdFusion plugin in the list of
active plugins within FusionReactor ( it looks like this:
http://docs.intergral.com/download/attachments/9666568/plugins1.png?version=1&modificationDate=1311241798733
; documentation available here:
http://docs.intergral.com/display/FR40/Installing+FREC+into+FusionReactor+Instances
).
The FREC plugin captures CF specific information (template cache size,
query cache size, etc) that could be useful to you during your
analysis.
Finally, whilst all parts of FusionReactor are designed from the group-
up to be very high performance and low overhead it's important to
understand and consider what your application and tests are doing -
and what you're logging. For example, a request running 1,000,000 DB
queries is going to need some fast I/O to log that information to disk
- so make sure the log file location is on fast disks (usually not a
problem on servers but just trying to be as transparent as possible).
With this level of configuration you should be able to perform any
sort of analysis required as the data will all be present - from real-
world, production experience I can honestly say you'll find
FusionAnalytics makes your job a LOT easier in a load-test/analysis
scenario - we do this process for customers and having worked "the old
fashioned way" charting up metrics in Excel or other external tools
having access to everything you need through a unified interface with
the reports and filtering you'll need all built in could easily save
you weeks of time on a large project.
As a foot-note I'd also say that if you find yourself needing even
more information you can enable full request/response logging - this
essentially tracks the entire HTTP "conversation" for every request.
Obviously that's a LOT of information and in my experience rarely
needed (especially for load testing) but I wanted to mention it as it
may be something you're interested in.
If you need any further assistance we offer professional consulting
services under our http://www.cfconsultant.com/ brand - "consulting
from the makers of FusionReactor".
Good luck with your testing. Best regards,
David Stockton
Fusion Support Team
Darren Pywell
Customers often follow a Continuous Server Investigation (CSI) policy.
Typically the way that they do this is to look the top items from the
areas where they want to improve (e.g. performance, database
interaction, CPU load etc) and then aim to cut these down. By doing
this on a continuous basis (during development, testing and
production) they keep the applications and the servers really fit.
This stops the "cracks" forming that lead to bigger structural/
architectural issues. The process really works and we know of some big
customers that adopt this approach and are having a lot of success
with it:
So lets say you're looking to improve page performance overall. A good
metric to look at in this case is the Slowest Requests on Average:
Request
URL
Exec Time Request Count
http://int0046.bbn.intergral.com:18080/apps/ConsumejustFunction.cfm
20s 590ms 120
http://int0046.bbn.intergral.com:18080/apps/justFunction.cfc
20s 379ms 136
http://int0046.bbn.intergral.com:18080/apps/tellTime.cfc
15s 820ms 32
http://int0046.bbn.intergral.com:18080/apps/humanresources/Preview.cfm
986ms 6,434
http://int0046.bbn.intergral.com:18080/apps/humanresources/LongQuery.cfm
985ms 5,335
From my list you can easily see that we have 3 requests that really
don't look so great and are taking 15-20 times a long as the next
"highest group". Tuning these requests can bring a lot of
improvement... but also notice that the Request count is very low on
those request so they don't run often. That doesn't mean they aren't
worth tuning but it might mean the effort of tuning could be better
spent somewhere else. In this case take a look at the Request -> Long
Request Execution Time in Total
Request
URL
Page Views Avg Exec Time Max Exec Time Min Exec Time Total Exec Time
http://int0046.bbn.intergral.com:18080/apps/humanresources/Preview.cfm
6,434 986ms 10s 32ms 78ms 1h 45m 44s
936ms
http://int0046.bbn.intergral.com:18080/apps/humanresources/LongQuery.cfm
5,335 985ms 1m 0s 0ms 78ms 1h 27m 37s 708ms
http://int0046.bbn.intergral.com:18080/apps/modelgluesamples/lhp/index.cfm
33,778 114ms 1m 0s 0ms 15ms 1h 4m 33s 289ms
http://int0046.bbn.intergral.com:18080/apps/justFunction.cfc
136 20s 379ms 1m 0s 0ms 20s 0ms 46m 11s
605ms
http://int0046.bbn.intergral.com:18080/apps/ConsumejustFunction.cfm
120 20s 590ms 1m 0s 0ms 20s 15ms 41m 10s
810ms
http://int0046.bbn.intergral.com:18080/apps/humanresources/ShortQuery.cfm
4,256 579ms 6s 766ms 46ms 41m 6s 174ms
http://int0046.bbn.intergral.com:18080/apps/humanresources/SearchResults.cfm
15,955 63ms 1m 0s 0ms 0ms 16m 55s
639ms
Now things almost look that other way up. In fact the quicker requests
that get run more often spend most time executing in total. Just
shaving off a small amount of time on these request can improve the
capacity on the server a lot. It's worth then also taking a look at
the Slowest DB requests for example to correlate if JDBC is a root
cause of the performance usage. It really is that simple to find and
improve stuff using FusionAnalytics; just pick the way that you want
to improve and then take the reports that give you insight.
I'd also really recommend you to take a look at the Request
Performance Bell-Curve. This gives you an overall image of what
response times look like on your app/server. If you have a few really
slow running requests, I'd suggest that you change the request
duration (third filter tab on the bottom) to read say 10 seconds
maximum so that the very slow requests are filtered out. The bell
curve is showing you what performance looks like across the whole
application/server. If you draw a line down from the peak of the curve
you'll see what the average request performance looks like im
milliseconds where it crosses the axis. You're looking to get the peak
to be as far left and as steep as possible. The steeper the curve, the
tighter the request performance is, the flatter the curve the more
variance you have (sounded a little like Joda that!).
Hope that helps,
Darren
Brook Davies
We have multiple load balanced servers, 3 actually, and I am wondering if it makes sense to do some of the load testing on a single server. Wouldn't this make it easier to measure performance and track issues?
Brook
--
You received this message because you are subscribed to the Google Groups "FusionAnalytics" group.
To post to this group, send email to fusiona...@googlegroups.com.
To unsubscribe from this group, send email to fusionanalyti...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fusionanalytics?hl=en.
Charlie Arehart
Brook, that may make it easier, but it may not make it as accurate as testing against multiple load balanced servers, if that’s indeed what will run in prod.
2 other things about load testing in general:
a) creating accurate load tests is hard work, and even if you do guess at traffic patterns and setup sophisticated ratios of calling one URL or another, you may miss out on load you’d never consider: spiders and bots. I’ve seen sites where 80% of traffic each day was from spiders and bots. If you don’t account for simulating them in your load testing, you may either fail to adequately mimic your current prod load (if the site is in prod already), or you may underestimate your likely load when you put a new server into production.
b) related to the above, and yet separate: be sure you tell the load testing tool to honor and return cookies that are sent back and forth from CF (especially cfid/cftoken if you enable sessionmanagement or clientmanagement, and/or jsessionid if using j2ee sessions in CF). If you don’t do that, then each page request from the load testing tool will seem to CF to be a new session and client, which could seriously negatively affect your testing. It would make your load testing tool work like a spider or a bot (because they, too, do not typically bother to present client/session cookies to CF).
/charlie
Brook Davies
I was reading a performance tuning article on Adobe and saw the use of GCviewer to track garbage collections. Does FR/FA have something similar or would using this tool while loadtesting also be a good idea?
Brook
--
Charlie Arehart
My sincere opinion would be that there’s no reason to worry about garbage collections at all. All that really matters generally (with respect to them) is whether you get any outofmemory errors (typically shown in the [cf]\runtime\logs or [jrun]\logs on multiple instances).
I know mine is a minority opinion in the community, but this comes from my focus on troubleshooting CF server problems daily for hundreds of customers. I just wonder if some people over-estimate the significance of watching GCs because they otherwise lacked much better metrics. Now you have FA and FR—use the force, Luke. :-)
/charlie
From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of Brook Davies
Sent: Wednesday, February 15, 2012 8:00 PM
To: fusiona...@googlegroups.com
Subject: Re: [fusionanalytics] Re: Best way to track/monitor durring load testing
Thanks Charlie, I will keep that in mind.
Brook Davies
I have 3 CF Pro servers on Windows 2008 Web Server x64 with 8 gigs of ram each and 2.8 dual core processors. My app has a lot of applicaton scoped CFCs and uses client variable storage (db). I would say the young generation would be the biggest as for each request a lot of big objects are created in the local scope (of the application scoped cfcs) and in the request scope.
So would I be wise to do something like:
-Xmx:6024
-Xms:3024
-Xmn:2024
Should I set the min size to the same as the max? I've always done that at when allocating 1024mb but never worked at these numbers before. Should I increase the SurvivorRatio?
I'm still working on the FA install :) Had to create some gold VM's first, and sysprep and all that other stuff is becoming a bit of pain today. Not as much as building servers from scratch mind you :)
Brook
--
Charlie Arehart
Brook, I would say you’re engaging in premature optimization, and perhaps have been reading too many blogs and java technotes. :-) Is your head swimming after doing so? Sure, there are lots of opinions out there, and lots of “suggested settings”.
But surely many of them will have also said that the best answers come from load testing, and the whole point of doing them is so that you can make a run, review the results, tweak a setting (if need be), and run the test again. It needs to be an iterative process.
No one can tell you the “right numbers” to use. It’s SO entirely dependent on your app, its design, other integration points it may have (and how they may affect performance), as well as the configuration of CF, the JVM, the web server, the DB server, the network, and so on. (And I would say that any guessing at use of memory within the generations, as well as what heap sizes you need, is also not something you can guess from the outside. Run the tests, watch for errors, use FR and FA to help with understanding what’s going on, and use all that info to help drive tweaks by meaningful diagnostics.)
And you don’t have to go it alone. There are several folks who offer their services to help do troubleshooting (whether of live servers or of load testing results). See a list of them I keep in a category of my cf411.com site: www.cf411.com/cfconsult
Hope that’s helpful.
/charlie
From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of Brook Davies
Sent: Wednesday, February 15, 2012 10:33 PM
To: fusiona...@googlegroups.com
Subject: Re: [fusionanalytics] Re: Best way to track/monitor durring load testing
lol, thank you Obie. I will try. I'm excited and nervous about my load testing and watching FA. Maybe this is not the right forum, but whats swirling through my head right now is: