Enabling profiler agent (jprofiler) continually in production app?

[Kevin Burton]

I'm curious what you guys think about this.

I'm considering enabling jprofiler on all or a few select machines in our cluster.

The profiling isn't enabled until you connect.. but of course I'm sure this is not entirely free.  

-agentpath:/usr/share/jprofiler/bin/linux-x64/libjprofilerti.so=port=8849,nowait

Also, I think the agent rewrites classes to add the instrumentation, correct?  So once I run a profile, there's a constant time overhead.

But more importantly, I'm curious about stability.

Does anyone run a production setup like this?

[Jason Koch]

Hi Kevin

I have definitely had Yourkit profiler running in Production without significant issues, for 1-2 weeks at a time, for moderate volume "web service" style services (where SLAs are measured in seconds). As usual, I'd suggest testing in your own environment.

A couple of things to note

- It is cheap but not free, as the agent is usually collecting some basic data such as heap utilisation metrics. 

- Today's APM platforms (eg CA APM, new relic) are often deployed to application servers with very little overhead, and these effectively perform the same thing. If you are looking for these sorts of metrics, an APM tool may be worth considering.

Of course, if you're considering a profiler instead of an APM tool, it's because you want the deep dive capability. This means knowing a few things:

- Yes class instrumentation occurs. Check in test environment that your PermGen or equivalent sizing is sufficient as this can hit you (I have seen this repeatedly on Tomcat hosted apps with default 64MB). For Yourkit this can be disabled, however you limit the functionality available if you choose to connect and want to profile.

- If you load the attach agent at runtime then starting some profiling mean many (all?) classes need a rewrite - this can take a LONG time (10+ seconds). For Yourkit if you load ahead of time the cost to start is significantly reduced but it should mean a runtime impact.

- Connecting to the profiler is ~free. Getting some basic info such as GC history is ~free.

- Performing anything more advanced will definitely impact your performance (such as cpu sampling, memory allocation profiling).

- A full memory dump will take some time and pause your VM :).

For stability, the key one is your permgen sizing and being careful about which operations you use, however as mentioned I've not done this for more than a couple of weeks.

Thanks

Jason

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Booth]

I think this can sometimes be more a culture question than a technical one- how much fear/cojones

is presnet in your working environment. It's also easier to feel OK about this if you are talking about 

two out of twelve boxes than one of three boxes. In the web world I've routinely used New Relic with

only positive results. In the financial apps world I've occasionally used yourkit in production (in sampling 

mode) without undue impact.

If I were doing it today I'd probably opt to use Oracle 1.8.0_45 with java flight recorder, because (1) it supports

asynch profiling, avoiding the bias inherent in yourkit/jprofiler/visual vm, and because it exposes 

so much fine grained configurability. My subjective impression is that it's lower impact than yourkit/jprofiler, 

but I'd like to validate that empirically.

Peter