How expensive JMX calls are?

[Vero K.]

We decided to use Telegraf together with Jolokia JMX plugin (https://github.com/influxdb/telegraf/tree/master/plugins/jolokia) to collect metrics from our app: uptime, heap, business jmx methods. So we will expose 15-20 which will be triggered every 10-15 seconds. How expensive these calls can be? Our app is used for trading and I don't want to put too much pressure on jvm. 

Other solution is to explicitly publish stat to the Telegraf from the background thread running inside app - but I don't like this solution.

thanks

[Michael Barker]

The obvious answer here is to benchmark and and profile the calls.

From memory JMX calls through the MBeanServerConnection (https://docs.oracle.com/javase/8/docs/api/javax/management/MBeanServerConnection.html) or via a JMX proxy use reflection and each call generates a small amount of garbage.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Thompson]

If you want an efficient means of monitoring an application that does not generate garbage or cost much to record then have a look at the system counters in Aeron and the AeronStat application for reading them.

https://github.com/real-logic/Aeron/blob/master/aeron-driver/src/main/java/uk/co/real_logic/aeron/driver/SystemCounters.java

https://github.com/real-logic/Aeron/blob/master/aeron-samples/src/main/java/uk/co/real_logic/aeron/samples/AeronStat.java

--

[Vero K.]

thanks, but what about jmx performance consumption. if I will trigger 15-20 methods every 10-20 sec, will it put a pressure on running jvm app? when jmx instrument classes, is it during profiling or always? I just want to use uptime, heap etc. stat and also some business logic methods

thanks.

[Martin Thompson]

That frequency of calls should have a negligible impact on performance.

[Gil Tene]

From a CPU point of view: JMX call will be served by separate threads, and the work involved will be pretty trivial (e.g. much smaller than e,g. executing a single vmstat or netstat shell command). Since it will not be happening in your regular Java threads and critical path, the only load-based concern will be having available empty cores to execute this stuff without stealing cores away from critical work. If you are in the latency sensitive or low latency game, you probably already have plenty of empty cores (or configure core affinity or isolation) to handle such "background" work without needing to interrupt the critical path (things like periodic VM thread operations, or the various GC operations that are concurrent, or some daemon process wanting to do something, or performing "top", vmstat, etc.). Servicing JMX monitoring requests just adds one more thing to a long list of background stuff that needs such cores anyway.

From an allocation point of view: Unless you are already successful in achieving zero-allocation *everywhere* else, the impact of the allocation involved in servicing JMX calls will be inconsequential: Exactly how much each such JMX call will allocate will depend on what you are doing exactly, but it will probably be a drop in the bucket compared with allocation that results from other work you are doing. Those allocations will also tend to all die nicely in the newgen. So you are looking at a tiny (probably immeasurable) increase in newgen frequency. 

Bottom line: for most applications, including latency sensitive and low latency applications, JMX monitoring is not an issue. Zero-allocation (or the 1MB/hr type of things that may last days between newness) are probably the only exception.

[Benedict Elliott Smith]

Depending on your JVM, GC settings, and if you open a new JMX connection for each call, this could cause either more frequent full GC or unexpected OOM.

http://mail.openjdk.java.net/pipermail/jmx-dev/2014-February/000585.html

https://issues.apache.org/jira/browse/CASSANDRA-6541

https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent

[Vero K.]

thanks, everyone. also good to know that jmx has class unloading problem, we use jdk8_65 so we will add CMSClassUnloadingEnabled  and it should solve the problem.

[ki...@kodewerk.com]

On Jan 3, 2016, at 6:43 PM, Benedict Elliott Smith <b.ellio...@gmail.com> wrote:

Depending on your JVM, GC settings, and if you open a new JMX connection for each call, this could cause either more frequent full GC or unexpected OOM.

The Full GC is only called once per hour if there has been no GC activity in the last hour. You can dial that back but then you risk issues with recovering reference object of which it was finalization that was the biggest concern for the authors of RMI when the wrote the DGC as that is often the way things such as file descriptors recovered. Using Jolokia instead of RMI should for the JMXConnector will offer relief for some of that pressure but Jolokia is not a drop in replacement for a JMXConnector… unfortunately. It is mostly compliant on the server side but missed completely on the client side of the wire. Too bad IMHO because JMX would be much more useful if it weren’t using RMI at the core of the JMXConnector.

Regards,

Kirk

[Benedict Elliott Smith]

No, it is dependent on the number of JMX calls, as they fill up permgen/metaspace.

[Vero K.]

I'm now really confused. Looks like nagios uses jmx for monitoring, just interesting how they do it. Anyway, for me stability is more important than performance. 

What is the deal with metaspace. If we will use CMSClassUnloadingEnabled with JDK 8_u65 and trigger 15-20 jmx methods with jolokia every 12 sec. will we be safe or OOM error still be a problem?

On Monday, January 4, 2016 at 12:38:58 PM UTC+2, Benedict Elliott Smith wrote:

No, it is dependent on the number of JMX calls, as they fill up permgen/metaspace.

On 4 January 2016 at 08:08, ki...@kodewerk.com <ki...@kodewerk.com> wrote:

On Jan 3, 2016, at 6:43 PM, Benedict Elliott Smith <b.ellio...@gmail.com> wrote:

Depending on your JVM, GC settings, and if you open a new JMX connection for each call, this could cause either more frequent full GC or unexpected OOM.

The Full GC is only called once per hour if there has been no GC activity in the last hour. You can dial that back but then you risk issues with recovering reference object of which it was finalization that was the biggest concern for the authors of RMI when the wrote the DGC as that is often the way things such as file descriptors recovered. Using Jolokia instead of RMI should for the JMXConnector will offer relief for some of that pressure but Jolokia is not a drop in replacement for a JMXConnector… unfortunately. It is mostly compliant on the server side but missed completely on the client side of the wire. Too bad IMHO because JMX would be much more useful if it weren’t using RMI at the core of the JMXConnector.

Regards,

Kirk

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Grzegorz Molenda]

CMSClassUnloadingEnabled is by default enabled since JDK 8, therefore you don't have to set it explicitly. This allows CMS to do some unloading work during re-mark phase, which makes the phase a bit longer, but the impact as previously stated should be neglectable for the number of calls you provided. This helps you to avoid long lasting FullScan, triggered by perm/metaspace filling. I wouldn't expect OOM errors in this case.    

There should be no problem with arbitrarly triggered FullScans by RMI as you are using Jolokia.  

Thanks, 

Grzegorz

2016-01-04 15:29 GMT+01:00 Vero K. <vero....@gmail.com>:

I'm now really confused. Looks like nagios uses jmx for monitoring, just interesting how they do it. Anyway, for me stability is more important than performance. 

What is the deal with metaspace. If we will use CMSClassUnloadingEnabled with JDK 8_u65 and trigger 15-20 jmx methods with jolokia every 12 sec. will we be safe or OOM error still be a problem?

On Monday, January 4, 2016 at 12:38:58 PM UTC+2, Benedict Elliott Smith wrote:

No, it is dependent on the number of JMX calls, as they fill up permgen/metaspace.

On 4 January 2016 at 08:08, ki...@kodewerk.com <ki...@kodewerk.com> wrote:

On Jan 3, 2016, at 6:43 PM, Benedict Elliott Smith <b.ellio...@gmail.com> wrote:

Depending on your JVM, GC settings, and if you open a new JMX connection for each call, this could cause either more frequent full GC or unexpected OOM.

The Full GC is only called once per hour if there has been no GC activity in the last hour. You can dial that back but then you risk issues with recovering reference object of which it was finalization that was the biggest concern for the authors of RMI when the wrote the DGC as that is often the way things such as file descriptors recovered. Using Jolokia instead of RMI should for the JMXConnector will offer relief for some of that pressure but Jolokia is not a drop in replacement for a JMXConnector… unfortunately. It is mostly compliant on the server side but missed completely on the client side of the wire. Too bad IMHO because JMX would be much more useful if it weren’t using RMI at the core of the JMXConnector.

Regards,

Kirk

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Vero K.]

sounds good, I guess jolokia which we are going to use as a jvm agent won't be creating jmx connection everytime, it just keeps it, right?

On Monday, January 4, 2016 at 6:04:45 PM UTC+2, Grzegorz Molenda wrote:

CMSClassUnloadingEnabled is by default enabled since JDK 8, therefore you don't have to set it explicitly. This allows CMS to do some unloading work during re-mark phase, which makes the phase a bit longer, but the impact as previously stated should be neglectable for the number of calls you provided. This helps you to avoid long lasting FullScan, triggered by perm/metaspace filling. I wouldn't expect OOM errors in this case.    

There should be no problem with arbitrarly triggered FullScans by RMI as you are using Jolokia.  

Thanks, 

Grzegorz

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[ki...@kodewerk.com]

No, it is dependent on the number of JMX calls, as they fill up permgen/metaspace.

AFAIK, the only thing being loaded into permgen are the generated stubs. It would take a lot of calls to fill this space. Additionally, filling  permgen or metaspace would automatically trigger a collection that would also collect those spaces. This implies that there is no need for RMI to make a call to System.gc(). However, the arguments that I’ve heard were used to defend the call to System.gc() was the need to free resources such as file descriptors via finalization. If this is incorrect, there would be no way to defend this behavior.

Regards,

Kirk

[Kevin Burton]

The "classic" JMX APIs are really expensive.  Especially the remote method stub stuff.

Jolokia is pretty decent as I *think* it is fast on the backend so you're really just dealing with REST at that point.  

[ki...@kodewerk.com]

On Jan 12, 2016, at 9:44 PM, Kevin Burton <burto...@gmail.com> wrote:

The "classic" JMX APIs are really expensive.  Especially the remote method stub stuff.

Jolokia is pretty decent as I *think* it is fast on the backend so you're really just dealing with REST at that point.  

It’s Rest but unfortunately it’s not JMXConnector compliant… It works reasonably well though.

— Kirk