Scripting Java Mission Control..

668 views
Skip to first unread message

Kevin Burton

unread,
May 3, 2016, 6:19:34 PM5/3/16
to mechanical-sympathy
I accidentally stumbled upon this by running jmc -help... (see below).

I'd LOVE a way to capture profiles and have regular reports of our releases so I can instantly see a performance regression.

THIS seems to hint that it's possible.  



General
==========================
call              Executes the commands in a file.
exit              Exits the application.
help              Shows help.
print             Prints a message.
printproperties   Prints all system properties.
setproperty       Sets a system property.
showversion       Prints version information
version           Prints version information and exits
wait              Waits for a user defined number of seconds.

Scripting
==========================
click             Clicks on a widget, e.g. a button or tab.
closeeditor       Closes the current active editor.
closewelcome      Closes the welcome screen.
open              Opens up a file.
select            Selects an item, e.g in a ComboBox
selectgroup       Selects a tab group in an editor.
selecttab         Selects a tab in an editor.
startconsole      Starts the JMX Console.

Daniel Mitterdorfer

unread,
May 4, 2016, 1:15:28 AM5/4/16
to mechanica...@googlegroups.com
Hi Kevin,

nice, I didn't know that. We also have the option to take flight recordings in our benchmarks so I also thought of capturing a few key metrics from JFR and putting it into our benchmark metrics store. However, I want to parse the JFR file rather than using the UI (though I haven't started playing around yet, so no practical experience). I think both approaches are somewhat fragile but I'd still lean towards using the (unsupported) parsers [1]. Have you considered that already?

Daniel


--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Isuru Perera

unread,
May 4, 2016, 1:42:09 AM5/4/16
to mechanica...@googlegroups.com
Hi Daniel,

I used the JFR parser to generate a flame graph compatible input [1]. I find it easy to view the call tree in a flame graph.

[1] https://github.com/chrishantha/jfr-flame-graph
--

ki...@kodewerk.com

unread,
May 4, 2016, 4:25:44 AM5/4/16
to mechanica...@googlegroups.com
Hi Kevin,

You can already run JFR as an direct attach agent and you can use both configuration and external controls to control how and what it will capture. The JFR report is a binary file that can be read currently only by JMC. However it is possible to have other things read from it as well. While I love these features in JFR, I find the corresponding views in JMC noisy, resulting in confused views that are full of jargon that doesn’t really map very well with the language developers use. For example, the allocations view shows TLAB allocations however it is completely missing an allocation frequency view. In a recent survey of attendees of my performance tuning talk, only a couple of people actually knew what a TLAB was. Now, one might argue that it is the role of an execution profiler to pick up on frequency events. However, execution profiler views on allocations are some what less then ideal. In my experience developers are less likely to pick up on an allocation issue with an execution profiler then they are when looking at the problem through the lens of an allocation profiler.

Further more, in my testing of JMC/JFR, it failed to identify the top bottlenecks in an application I use as part of my performance tuning workshop. In each successive round of tuning the top bottleneck was either buried in the noise floor, buried in a view that was either difficult to understand or difficult to find, or completely missed.

I think the JFR has great potential and I know that 9 will bring in some big changes in how it all works. So, I have some hope… My fear is that unless Oracle is taking feedback from people who are really tuning applications, we’ll only see more of the same.

Regards,
Kirk
signature.asc

Richard Warburton

unread,
May 4, 2016, 4:59:03 AM5/4/16
to mechanica...@googlegroups.com
Hi,

You can already run JFR as an direct attach agent and you can use both configuration and external controls to control how and what it will capture. The JFR report is a binary file that can be read currently only by JMC. However it is possible to have other things read from it as well. While I love these features in JFR, I find the corresponding views in JMC noisy, resulting in confused views that are full of jargon that doesn’t really map very well with the language developers use. For example, the allocations view shows TLAB allocations however it is completely missing an allocation frequency view. In a recent survey of attendees of my performance tuning talk, only a couple of people actually knew what a TLAB was. Now, one might argue that it is the role of an execution profiler to pick up on frequency events. However, execution profiler views on allocations are some what less then ideal. In my experience developers are less likely to pick up on an allocation issue with an execution profiler then they are when looking at the problem through the lens of an allocation profiler.

Worth noting that the allocations view in JMC does have a total allocated size and an average size. So you can work out the number of allocations! I agree its less than ideal.

I'm not sure what your complaint is around allocation vs execution though. JMC does have a memory allocation profiler, and you can capture stack traces with it to see the root cause of allocation problems. Not only that but many memory profilers bytecode weave code that causes more allocations, as Nitsan pointed out on his blog.

Further more, in my testing of JMC/JFR, it failed to identify the top bottlenecks in an application I use as part of my performance tuning workshop. In each successive round of tuning the top bottleneck was either buried in the noise floor, buried in a view that was either difficult to understand or difficult to find, or completely missed.

I appreciate you may not want to talk about the problems in your workshop publicly, but can you describe the failure to find the bottlenecks in more detail?

regards,

  Richard Warburton

ki...@kodewerk.com

unread,
May 4, 2016, 5:11:52 AM5/4/16
to mechanica...@googlegroups.com
On May 4, 2016, at 10:59 AM, Richard Warburton <richard....@gmail.com> wrote:

Hi,

You can already run JFR as an direct attach agent and you can use both configuration and external controls to control how and what it will capture. The JFR report is a binary file that can be read currently only by JMC. However it is possible to have other things read from it as well. While I love these features in JFR, I find the corresponding views in JMC noisy, resulting in confused views that are full of jargon that doesn’t really map very well with the language developers use. For example, the allocations view shows TLAB allocations however it is completely missing an allocation frequency view. In a recent survey of attendees of my performance tuning talk, only a couple of people actually knew what a TLAB was. Now, one might argue that it is the role of an execution profiler to pick up on frequency events. However, execution profiler views on allocations are some what less then ideal. In my experience developers are less likely to pick up on an allocation issue with an execution profiler then they are when looking at the problem through the lens of an allocation profiler.

Worth noting that the allocations view in JMC does have a total allocated size and an average size. So you can work out the number of allocations! I agree its less than ideal.

I'm not sure what your complaint is around allocation vs execution though.

Not really sure how to best describe it. It’s more of an observation based on years of coming into projects where people have not been able to find the root cause because they were using the wrong lens (profiler with the right view).

JMC does have a memory allocation profiler, and you can capture stack traces with it to see the root cause of allocation problems.

right but only by volume, not by frequency.

Not only that but many memory profilers bytecode weave code that causes more allocations, as Nitsan pointed out on his blog.

True, bytecode weaving will result in higher allocation rates… and it disturbs escape analysis which will affect how the allocators will decide how to allocate and so on.. however, it’s still been a very effective way of finding allocation hotspots…


Further more, in my testing of JMC/JFR, it failed to identify the top bottlenecks in an application I use as part of my performance tuning workshop. In each successive round of tuning the top bottleneck was either buried in the noise floor, buried in a view that was either difficult to understand or difficult to find, or completely missed.

I appreciate you may not want to talk about the problems in your workshop publicly, but can you describe the failure to find the bottlenecks in more detail?

It missed on hot locks…

— Kirk

signature.asc

Richard Warburton

unread,
May 4, 2016, 7:03:24 AM5/4/16
to mechanica...@googlegroups.com
Hi,

You can already run JFR as an direct attach agent and you can use both configuration and external controls to control how and what it will capture. The JFR report is a binary file that can be read currently only by JMC. However it is possible to have other things read from it as well. While I love these features in JFR, I find the corresponding views in JMC noisy, resulting in confused views that are full of jargon that doesn’t really map very well with the language developers use. For example, the allocations view shows TLAB allocations however it is completely missing an allocation frequency view. In a recent survey of attendees of my performance tuning talk, only a couple of people actually knew what a TLAB was. Now, one might argue that it is the role of an execution profiler to pick up on frequency events. However, execution profiler views on allocations are some what less then ideal. In my experience developers are less likely to pick up on an allocation issue with an execution profiler then they are when looking at the problem through the lens of an allocation profiler.

Worth noting that the allocations view in JMC does have a total allocated size and an average size. So you can work out the number of allocations! I agree its less than ideal.

I'm not sure what your complaint is around allocation vs execution though.

Not really sure how to best describe it. It’s more of an observation based on years of coming into projects where people have not been able to find the root cause because they were using the wrong lens (profiler with the right view).

You think people be using memory profiling more and execution profiling less, right? Isn't that an educational problem rather than a tooling problem. Ie, as long as JMC ships with a memory profiler its doing its job. I agree on the missing frequency view though.

Not only that but many memory profilers bytecode weave code that causes more allocations, as Nitsan pointed out on his blog.

True, bytecode weaving will result in higher allocation rates… and it disturbs escape analysis which will affect how the allocators will decide how to allocate and so on.. however, it’s still been a very effective way of finding allocation hotspots…

Sure.

Further more, in my testing of JMC/JFR, it failed to identify the top bottlenecks in an application I use as part of my performance tuning workshop. In each successive round of tuning the top bottleneck was either buried in the noise floor, buried in a view that was either difficult to understand or difficult to find, or completely missed.

I appreciate you may not want to talk about the problems in your workshop publicly, but can you describe the failure to find the bottlenecks in more detail?

It missed on hot locks…

Were the locks JVM intrinsic monitors or Java 5 locks? I ask because a weakness in sigprof based profilers is their inability to sample sleeping code which is what you get with Java 5 locks. If its intrinsic monitors then I don't know what's going on.  I probably don't think that a sigprof based profiler is a good tool for finding lock based problems in the same way that its a good for finding execution problems.

ki...@kodewerk.com

unread,
May 4, 2016, 7:26:11 AM5/4/16
to mechanica...@googlegroups.com
Hi Richard,


On May 4, 2016, at 1:03 PM, Richard Warburton <richard....@gmail.com> wrote:

Hi,

You can already run JFR as an direct attach agent and you can use both configuration and external controls to control how and what it will capture. The JFR report is a binary file that can be read currently only by JMC. However it is possible to have other things read from it as well. While I love these features in JFR, I find the corresponding views in JMC noisy, resulting in confused views that are full of jargon that doesn’t really map very well with the language developers use. For example, the allocations view shows TLAB allocations however it is completely missing an allocation frequency view. In a recent survey of attendees of my performance tuning talk, only a couple of people actually knew what a TLAB was. Now, one might argue that it is the role of an execution profiler to pick up on frequency events. However, execution profiler views on allocations are some what less then ideal. In my experience developers are less likely to pick up on an allocation issue with an execution profiler then they are when looking at the problem through the lens of an allocation profiler.

Worth noting that the allocations view in JMC does have a total allocated size and an average size. So you can work out the number of allocations! I agree its less than ideal.

I'm not sure what your complaint is around allocation vs execution though.

Not really sure how to best describe it. It’s more of an observation based on years of coming into projects where people have not been able to find the root cause because they were using the wrong lens (profiler with the right view).

You think people be using memory profiling more and execution profiling less, right? Isn't that an educational problem rather than a tooling problem. Ie, as long as JMC ships with a memory profiler its doing its job.

I agree it is an educational problem. However I don’t believe it’s a good practice to throw an industrial load of underwear against the wall and then have to pick through it all to sort out why it stuck…. This is how developers get tricked by profilers that are “lying" to them. A targeted use of profilers is a much more productive approach. That is not to say that I’ll not speculatively profile something. One just has to recognize when you do speculative profiling you’ll most often get speculative answers which may or may not be worth acting on.


It missed on hot locks…

Were the locks JVM intrinsic monitors or Java 5 locks? I ask because a weakness in sigprof based profilers is their inability to sample sleeping code which is what you get with Java 5 locks. If its intrinsic monitors then I don't know what's going on.  I probably don't think that a sigprof based profiler is a good tool for finding lock based problems in the same way that its a good for finding execution problems.

indeed, indeed, indeed… and yes the lock is JVM intrinsic….

Regards,
Kirk

signature.asc

Nitsan Wakart

unread,
May 6, 2016, 5:02:48 PM5/6/16
to mechanica...@googlegroups.com
We sort of lost the thread of the OP and moved on to general observations on JMC. 2c worth for the discussion:
- IMO, at present time JMC is the best allocation profiler, if only because it does not cause allocation and kill escape analysis. Is the useful info on display "in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”? yes it is. If you got a better tool for this job, let me know.
- CPU profiling will only show lock related CPU usage. If the problem you are facing is not "where cycles are spent?" but "why is no work getting done?" then this is not the tool you should be using.

- JMC has a 'Threads' tab, under which you can find Java locks info under 'Contention' and object monitor info (and park/sleep) under 'Latencies'. There's also regular collection of full thread dumps, so if you got a long profile you'll get a few of those lovelies in there. Again, perhaps buried where you might not want it, but there.
I would agree that JMC collects a ton of info, and expects the user to sift through it looking for answers.
I think it would be good if the format or the event sources were open to tooling (maybe they are and I just missed it?) so that people could contribute/hack and the best solution might emerge.
I also think the first page in the JMC view could be replaced with an intelligent selection of relevant findings e.g. you could comfortably fit on one page the top 10 hot methods, top 10 allocation hotspots, top 10 contended locks etc. This will be allot more insightful than the 2 pretty, but ultimately uninformative dials that take up most of the space on that page.

Marcus Hirt

unread,
May 6, 2016, 5:51:16 PM5/6/16
to mechanical-sympathy
Hi Kevin!

We love any and all feedback on JMC. :) So, this is what I can say right now:

1. The APIs for producing custom events and for controlling the Flight Recorder will be supported in JDK 9.
2. The parser in the JDK and the parser in JMC _may_ become supported in JDK 9. If not in JDK 9, then hopefully in an update to JDK 9. We may want more feedback before locking them down.
3. With the new APIs exposed around the JMC parser, what you want to achieve will be greatly simplified.
4. I will do a talk at JavaOne which will provide you with a sneak peak of these features.
5. If your company own WLS or Java SE Advanced, there is an EA program you can join to start playing with JMC 6 and JFR in JDK 9 today. 

Please note that whilst nothing would make me happier than handing out betas to everyone who would want them, the rules for doing betas are not Oracle's fault. Oracle must to play by the US SEC rules regarding revenue recognition. That said, I think there may be a limited number of spots for Java Champions. Ping me, and I will see what can be done.

Kind regards,
Marcus

Marcus Hirt

unread,
May 6, 2016, 5:51:55 PM5/6/16
to mechanical-sympathy, nit...@yahoo.com

Wojciech Kudla

unread,
May 6, 2016, 6:09:27 PM5/6/16
to mechanica...@googlegroups.com
There are alternatives to some of what JMC does. Some really valuable metrics lurk in perf counters (available through JVMstat api/jcmd/jstat). Zero-overhead, easy to collect and parse. We actually keep many of the counters in a database and visualize per jvm to observe synchronization pressure.

sun.rt.safepointSyncTime
sun.rt.safepointTime
sun.rt.safepoints

sun.rt._sync_Notifications
sun.rt._sync_Parks
sun.rt._sync_ContendedLockAttempts
sun.rt._sync_Deflations
sun.rt._sync_EmptyNotifications
sun.rt._sync_FailedSpins
sun.rt._sync_FutileWakeups
sun.rt._sync_Inflations

And many others. 

Marcus Hirt

unread,
May 6, 2016, 6:15:11 PM5/6/16
to mechanical-sympathy, nit...@yahoo.com
One point that I feel is often lost is that the main focus for JFR is to be a great option for production time profiling and diagnostics. We know it is, as our own support keep solving customer production cases using it. That said, I think it is a great testament to the value of the data produced, and the low observer effect associated with capturing that data, that the popularity of JFR is on the rise among developers.

We know that a major drawback for newcomers is that we capture quite a lot of data and that we display it in a rather direct fashion. It takes some time, and sometimes quite deep knowledge of the runtime, to put the information to good use. The origin of this tool was really as a tool for JVM developers to gather information on how the JVM was being utilized by real world applications. We realize that much of that DNA has survived. At the same time, we don't wish to do too many sweeping gestures that may hide information important to solve a problem. As I said, I hope that JMC 6 will be a big step in the right direction for both newcomers and experts. Newcomers will be able to put the tooling to use with very little information about HotSpot internals, and experts will need to spend less effort to find the interesting information, whilst still be able to go deep if necessary.

And finally - for those of you who don't enjoy the dials - In JMC 5.x, just bring up the design view and remove them (in JMC 6 there is no need ;) ). If you have friends you want to share your new UIs with, just export the pages you've changed as a plug-in from within JMC. Also, if you lack Max, Min, Stddev and other columns in the tables, again, just bring up the design view and add them.  

Kind regards,
Marcus

Marcus Hirt

unread,
May 6, 2016, 6:19:31 PM5/6/16
to mechanical-sympathy
And here is an example on how you can build a dynamic MBean to expose them all in the platform MBean server:

:)

Kind regards,
Marcus

Marcus Hirt

unread,
May 6, 2016, 6:35:07 PM5/6/16
to mechanical-sympathy
Hi Kirk,

It would be great if you could share that application. It would be great to verify that the issue is detected properly in JMC 6! And if it isn't, to make sure that it is! :)

Kind regards,
Marcus

ki...@kodewerk.com

unread,
May 7, 2016, 3:44:30 AM5/7/16
to mechanica...@googlegroups.com

> On May 6, 2016, at 11:02 PM, 'Nitsan Wakart' via mechanical-sympathy <mechanica...@googlegroups.com> wrote:
>
> We sort of lost the thread of the OP and moved on to general observations on JMC. 2c worth for the discussion:
> - IMO, at present time JMC is the best allocation profiler, if only because it does not cause allocation and kill escape analysis.

I’m going to go out on a limb here and say that I’m not sure that killing escape analysis is that big an issue. It’s not to say that I’m happy that existing tooling does break it. What I am saying is that IME, most of the memory profilers i’ve used have still been helpful in identifying allocation hotspots that have lead to decent results.

> Is the useful info on display "in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”? yes it is. If you got a better tool for this job, let me know.
> - CPU profiling will only show lock related CPU usage. If the problem you are facing is not "where cycles are spent?" but "why is no work getting done?" then this is not the tool you should be using.

Since the vast majority of my tuning efforts are exactly about “why is no work getting done” I really don’t do a lot of CPU profiling. My workflows simply rarely lead me that direction.

>
> - JMC has a 'Threads' tab, under which you can find Java locks info under 'Contention' and object monitor info (and park/sleep) under 'Latencies'. There's also regular collection of full thread dumps, so if you got a long profile you'll get a few of those lovelies in there. Again, perhaps buried where you might not want it, but there.

Which is why I was surprised when it didn’t show up in any of the views.

> I would agree that JMC collects a ton of info, and expects the user to sift through it looking for answers.
> I think it would be good if the format or the event sources were open to tooling (maybe they are and I just missed it?) so that people could contribute/hack and the best solution might emerge.

One of the best features is that you can create your own event sources. You can monitor JMX Beans and have JFR react to them. There are a ton of great features in JFR.

Kind regards,
Kirk

signature.asc

ki...@kodewerk.com

unread,
May 7, 2016, 3:48:34 AM5/7/16
to mechanica...@googlegroups.com
Hi Markus,


I’ve tested this out along with the custom event feature.. it works very well and one that I’ve suggest to customers that are able to use JFR take advantage of.

Regards,
Kirk

signature.asc

ki...@kodewerk.com

unread,
May 7, 2016, 3:55:05 AM5/7/16
to mechanica...@googlegroups.com
Hi Markus,

I was able to make the problem show up but only after adjusting the sampling rates for that particular aspect of the tooling. However, the adjustment in the sampling rate did heavily impact the performance of the application. So, it took a long time for the problem to burn it’s way out of the startup anomalies. Unfortunately the application in question isn’t one that I can freely pass along. However I do have some test applications that I’m going to try to bake the problem into. If I can reproduce the problem I’ll pass it along to you.

Regards,
Kirk


signature.asc

Nitsan Wakart

unread,
May 7, 2016, 11:47:03 AM5/7/16
to mechanica...@googlegroups.com
"most of the memory profilers I’ve used have still been helpful in identifying allocation hotspots that have lead to decent results."

I'd agree with that comment on average, but I've had my time wasted by garbage created by the allocation profiler. I can say it happened to me several times, but perhaps I'm just unlucky.

"Which is why I was surprised when it didn’t show up in any of the views."

A blind spot to JMC (and all AsyncGetCallTrace based profilers) is pathologically failed samples (in small numbers they don't matter). In Honest Profiler you can get a histogram of failures and their types(it's not as clear as it should be, I'm getting to it), in JMC they simply disappear as far as I can tell. It could be that you were hitting a piece of code on monitors which is not walkable/parsable to AGCT. In particular I know that some of the JVM runtime stubs can cause a sample failure. You can validate this theory by profiling with Honest Profiler and looking at the failed samples breakdown.

"One of the best features is that you can create your own event sources. You can monitor JMX Beans and have JFR react to them. There are a ton of great features in JFR."
I knew about adding you own events. I've learnt from Marcus's comment that you can create your own reports, which is also cool and should enable the contribution of a better default page. These are powerful features indeed. What I would like to see is the file format becoming open, and the event sources in the JVM becoming an official JVM API available on all JVMs. This is perhaps a pipe dream, but I think it would enable the Java community to create the tools it needs.

Marcus Hirt

unread,
May 12, 2016, 3:19:11 PM5/12/16
to mechanical-sympathy
Hi Kirk,

If you do come up with a producer that you can share, it would be immensely helpful!

Kind regards,
Marcus
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

ki...@kodewerk.com

unread,
May 15, 2016, 9:16:34 AM5/15/16
to mechanica...@googlegroups.com
Hi Marcus,

I’m going to spend some time on it this week. I’ve incorporated some JMC/JFR into my performance tuning materials and I will give a JMR/JFR quickie at J on the Beach so I’d very much like this all to work with any arbitrary problem I might throw at it.

— Kirk

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
signature.asc
Reply all
Reply to author
Forward
0 new messages