Using oprofile to profile Java applications via JVMTI

[Kevin Burton]

So this is interesting.  It looks like oprofile has supported profiling java applications for a LONG time now.  Since 2008 apparently.

I haven't heard of ANYONE doing this though...

http://oprofile.sourceforge.net/doc/applications.html

here's an example of the output:

http://oprofile.sourceforge.net/doc/getting-jit-reports.html

It seems like it's just using JVMTI for symbol resolution.  

Has anyone done this?  

Since oprofile claims a 2-3% system overhead, this might solve a lot of the need for a continuous java profiler.

[Todd Lipcon]

I've used this a bunch for profiling Hadoop and HBase. Very useful to see the post-JIT runtime in a way that is actually truthful, and has the bonus that it can expose other hardware counters like cache misses, etc.

It's a pain to use and I've found that trying to do call-graph profiling with it crashes, but I still prefer it over any Java profilers for accuracy.

Apparently Twitter has a fork of OpenJDK that enables you to use "perf" on java processes with JIT symbols and source correlation, but I don't think it has been open sourced as of yet. I generally think perf is better than oprofile these days, and I use it for all my native code profiling.

-Todd

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Kevin Burton]

oh.. .well hopefully twitter OSSes that because that would rock.  that would essentially be a stable continuous profiler.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Björn Töpel]

I've completely switched from oprofile to perf.

Porting the oprofile JVMTI plugin to work with perf is pretty straight
forward. IBM's Anton Blanchard has a JVMTI perf-hack for OpenJDK
available at http://ozlabs.org/~anton/junkcode/perf_jvmti.c (mentioned
in his Linux.conf.au 2013 talk:
http://mirror.linux.org.au/linux.conf.au/2013/ogv/How_to_make_almost_anything_go_faster.ogv).

I haven't tried Anton's hack (running my own), but I'm using perf to
profile our OpenJDK+JNI code all the time!


2013/9/27 Kevin Burton <burto...@gmail.com>:

>>> email to mechanical-symp...@googlegroups.com.

>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>

> --
> You received this message because you are subscribed to the Google Groups
> "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mechanical-symp...@googlegroups.com.

[Kevin Burton]

Ah... Nice..  Any idea what performance overhead you're hitting with perf here?

Could you publish your version/hack too?   That would rock.

My goal is to make our stack completely transparent by pushing a continuous profiler implementation in production ... 

There's lots of visibility and bugs I'm sure we're missing.

KEvin

>>> email to mechanical-sympathy+unsub...@googlegroups.com.

>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mechanical-sympathy+unsub...@googlegroups.com.

[Björn Töpel]

Unfortunately, I'm not allowed to open-source my hack. :-( I haven't
looked into the overhead of perf; My guess is that it's on par with
oprofile.

2013/9/27 Kevin Burton <burto...@gmail.com>:

>> >>> email to mechanical-symp...@googlegroups.com.

>> >>> For more options, visit https://groups.google.com/groups/opt_out.
>> >>
>> >>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mechanical-sympathy" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to mechanical-symp...@googlegroups.com.

>> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mechanical-symp...@googlegroups.com.

[Kevin Burton]

Darn..  So I played with the perf version you posted.

I couldn't get it to work but I didn't spend a ton of time with it.

It might not work with JDK7... It appends symbols to a file as they are loaded.

It definitely doesn't work with 'perf top' which is a bit disappointing.

The perf tools are also not as advanced as I would like, but maybe I just need to wrap my head around this problem.

>> >>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> >>> For more options, visit https://groups.google.com/groups/opt_out.
>> >>
>> >>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mechanical-sympathy" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to mechanical-sympathy+unsub...@googlegroups.com.

>> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mechanical-sympathy+unsub...@googlegroups.com.

[pron]

Does anyone have any idea how the new JDK 7u40's flight-recorder (+ Java Mission Control) works? It seems to record a lot of information with a very low overhead, and the results seem quite good.

[Kevin Burton]

It's using AsyncGetCallTrace I believe:

http://jeremymanson.blogspot.com/2007/05/profiling-with-jvmtijvmpi-sigprof-and.html 

It's ONLY in Oracle's JDK, correct?

I think we need a solution for this in the OSS community.  

Google released some code but it needs some cleanup.  There's also a JVMTI implementation for perf ... and now apparently Twitter has OpenJDK integrated with perf.

Lots of people are duplicating effort here...  

[pron]

> It's ONLY in Oracle's JDK, correct?

Not only is it only in Oracle's JDK, it's designated as a "commercial feature", which means that a JDK user is licensed to freely use flight recorder and JMC only in development. Using it in production requires a commercial license. It's quite impressive, though. I've used it a lot in the past few weeks, and it seems to give better results than VisualVM's sampling profiler, with a lot more information.  

[Simone Bordet]

Hi,

On Sun, Sep 29, 2013 at 2:38 PM, Kevin Burton <burto...@gmail.com> wrote:
> It's using AsyncGetCallTrace I believe:
>
> http://jeremymanson.blogspot.com/2007/05/profiling-with-jvmtijvmpi-sigprof-and.html
>
> It's ONLY in Oracle's JDK, correct?

That's correct, so source code is not available.

I spoke to Marcus Hirt at a lunch, and I asked exactly if JFR uses
AsyncGetCallTrace() and he said that no, they are using some other
internal tricks which he did not detailed, probably coming from the
JRockit days.
I would not be surprised if new calls have been added on 7u40 because
of this work, ported from JRockit to HotSpot.

I hope I was able to explain my question clearly to Marcus, and I
think he got it, but for 100% certainty it's probably best to get an
email confirmation from him directly.

--
Simone Bordet
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless. Victoria Livschitz

[Kevin Burton]

Ah.. fascinating... Can one just strace the JVM with it running?  The methods are in C ... so you could figure out what features were added to OpenJDK to add these.

In the mean time, my plan is to just have one of our worker nodes which MUCH lower load than production servers.  Basically adjust their weight, then enable profiling on them ...

Still not happy with that scenario. 

Seems like we need a decent OSS profiler for the OpenJDK.

Kevin

[Tobias Lindaaker]

From talking (at JavaOne this past week) to the engineers that built both Flight Recorder, and those who build AsyncGetCallTrace, it seems like Flight Recorder isn't using AsyncGetCallTrace as an API, but it is using the same internal JVM functionality as AsyncGetCallTrace uses.

I haven't read up on JDK source code enough to have my world view include the code for Flight Recorder. But this is my understanding based on the conversations I've had over the past week.

The metrics that it actually collects are supposedly defined in the OpenJDK source, but the actual Flight Recorder (the thing that actually records the events) is Oracle proprietary code, included only in their commercial build. The profiling part of it, since it is a sampling profiler, is also completely in Oracle proprietary code, with the internal structures and APIs that it uses of course being present in OpenJDK as well (and as far as I understand being the same as those used by AsyncGetCallTrace).

I too had a look at the perf-hack that Björn posted, and it looks like a sensible low overhead approach. Essentially what it does is inject itself into the JVM and reacting to the events that get fired when the JVM loads (and unloads) JIT compiled code, emitting the symbolic names and code offsets into a file that can be used to match the offsets that perf would find when profiling to symbolic names. This sounds like a sensible approach in theory, but I don't have enough experience with perf to know how to make it use such a symbol file, and I haven't had time to dig into that topic yet. If anyone has that insight, please share. A link to a page with good documentation about such a feature would be sufficient. Otherwise I'll dig into it when I get back to working on profiling (late october / early november).

-Tobias

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Kevin Burton]

The whole Oracle proprietary and commercial JDK is not going to end well.  The incentives aren't aligned here.  What if Google, Twitter, and Facebook want to push for an open profiling infrastructure in OpenJDK... but this impacts millions of dollars of Oracle sales?  What are they going to do then?

I looked at the JVMTI perf hack. I didn't get it working because I didn't have enough time.  It looks like it essentially writes data to a output file and I assume you have to load that along with your report so that it can resolve symbols.

'perf top' didn't work.

It would be ideal to figure out a way to just make it transparently work with the JDK.  Maybe have it use some shared memory segments to share the data with perf so that it can easily resolve what's happening.

[Kirk Pepperdine]

AFAIK, Misson Control will be OSS'ed. The API's for extension may (or may not) be pub'ed by Marcus Hirt. They will not be supported in 8 which is why he's a wee bit leery about pub'ing them. By 9 MC should be OSS'ed and the plugin support should be formalized. The current implementation is in Eclipse RCP due to the SWT components they use. I'm guessing based on little birdie music that the final target will be VisualVM. That will make adding your own extensions fairly trivial. Currently there is *nothing* in Eclipse to support monitoring/profiling that comes even close to what is available in VVM. That is about all I can say for the moment.

-- Kirk

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[pron]

> I'm guessing based on little birdie music that the final target will be VisualVM.

Oh good. As cool as JMC is – it's butt ugly.

[pron]

On Monday, September 30, 2013 1:01:00 AM UTC+3, Kevin Burton wrote:

The whole Oracle proprietary and commercial JDK is not going to end well.  The incentives aren't aligned here.  What if Google, Twitter, and Facebook want to push for an open profiling infrastructure in OpenJDK... but this impacts millions of dollars of Oracle sales?  What are they going to do then?

 Nah, I don't think it's that bad. Oracle isn't making millions off of JFR. It's just an extra bonus for those who need support. By the time something like this gets into OpenJDK, Oracle will have another cool feature in the proprietary JDK. So I think of it simply as getting early access to future features when you buy a support subscription. It actually seems like a good business model that benefits everyone: if it works then it will incentivize Oracle into developing more advanced features which will eventually be open-sourced.

[Kevin Burton]

Is this your hypothesis or has this happened?  

[pron]

Call it wishful thinking :)

[liang xie]

Seems need to modify kernel as well besides OpenJDK, i heart that taobao did the similar stuff like Twitter's, see the kernel patch if curious:)
https://github.com/alibaba/taobao-kernel/blob/master/patches.taobao/perf-jit-symbol-annotation-220.patch

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Jin Mingjian]

Great to see Xie Liang to mention our Chinese developer's open source contribution:) One guy of Taobao presents some works in this field in this year. Although the patch is in kernel tree, but it should be just for the perf tool itself. So, it is highly possible to be pulled out as standalone one. I remember they say that it is well for "perf top" although sometimes crashes for "perf record". So, if we have interests in this, we can push more in this:) 

Jin

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.