Is it safe to grab metrics via JMX in production?

[Nikolay Artamonov]

Hi!

Documentation for Dropwizard metrics contains interesting warning about using JMX: "We don’t recommend that you try to gather metrics from your production environment. JMX’s RPC API is fragile and bonkers. For development purposes and browsing, though, it can be very useful."

I wonder is it applicable to jmx_exporter?

Thanks!

[Brian Brazil]

JMX is not the best of protocols, but it's also the only way to monitor many Java services. 

--

Brian Brazil

www.robustperception.io

[Harald Koch]

On Thu, Oct 17, 2019, at 07:58, Nikolay Artamonov wrote:

try to gather metrics from your production environment. JMX’s RPC API is fragile and bonkers. For development purposes and browsing, though, it can be very useful."

I wonder is it applicable to jmx_exporter?

The JMX exporter queries all of the published management beans and then filters the results using the expressions provided in the configuration.

We had an application with a management bean that would run a database query to get information. Polling this application from Prometheus severely affected performance. We now publish metrics from that application using the java client.

We have another application (ActiveMQ) with similar issues; querying every single MBean in a busy broker takes 10s of seconds, and severely degrades the performance of the broker. In this case, we wrote a custom exporter that only queries the MBeans and attributes that we care about.

In short, I haven't had good experiences using it. Others may have different experiences to share...

--
Harald

[Nikolay Artamonov]

The JMX exporter queries all of the published management beans and then filters the results using the expressions provided in the configuration.

Hmm, at first it looks strange. I thought otherwise. May be we should open issue?

[Cameron Kerr]

It's like trying to monitor network devices by walking an entire SNMP tree; you're going to end up grabbing much more information than you really need and cause side-effects in order to get it.

Eg. One common side-effect you may not realise is that you'll end up locking data in order to get a consistent read; or you'll end up causing a database query.

You should be sure to use the whitelist and blacklist features; browse the available mbean objects using jconsole or similar to find just the bits you will be interested in, and then query only those.

JMX for a production system ought to be fine; instrumentation is for the benefit of production environments after all; its also what a lot of the middleware management consoles use to expose the internals to administrator. Just don't go scraping everything; information doesn't come for free, but if you pick and choose it can come cheaply.

Cheers,

Cameron

[Harald Koch]

On Thu, Jun 4, 2020, at 08:10, Cameron Kerr wrote:

You should be sure to use the whitelist and blacklist features; browse the available mbean objects using jconsole or similar to find just the bits you will be interested in, and then query only those.

But that's my point - the JMX exporter queries all of your mbeans and only then filters the results using the lists. So you'll still get the problems you mention; locks, db queries, and so on.

(Also this is an 8-month old thread :)

--

Harald

[Cameron Kerr]

On Friday, 5 June 2020 05:20:55 UTC+12, Harald Koch wrote:

On Thu, Jun 4, 2020, at 08:10, Cameron Kerr wrote:

You should be sure to use the whitelist and blacklist features; browse the available mbean objects using jconsole or similar to find just the bits you will be interested in, and then query only those.

But that's my point - the JMX exporter queries all of your mbeans and only then filters the results using the lists. So you'll still get the problems you mention; locks, db queries, and so on.

True, the default configuration is unhelpful (harmful even) in that it grabs everything, and the provided examples should do a better job of showing something more production-ready. The whitelistObjectNames and blacklistObjectNames are the bits that limit what is actually grabbed, and then the rules kick in to affect the transformation into metrics for prometheus.

Even then, the white/blacklistObjectNames only apply to the MBean object itself, and not to individual Attributes, which raise its own, similar issues. (eg. if I want to monitor a thread pool I might just be interested in the current usage, rather than the status of every thread).

 

(Also this is an 8-month old thread :)

I realised; my reply for largely for the benefit of people who might find this in a search, as this is not yet well documented.

Cheers,

Cameron

[Harald Koch]

On Thu, Jun 4, 2020, at 20:19, Cameron Kerr wrote:

True, the default configuration is unhelpful (harmful even) in that it grabs everything, and the provided examples should do a better job of showing something more production-ready. The whitelistObjectNames and blacklistObjectNames are the bits that limit what is actually grabbed, and then the rules kick in to affect the transformation into metrics for prometheus.

Even then, the white/blacklistObjectNames only apply to the MBean object itself, and not to individual Attributes, which raise its own, similar issues. (eg. if I want to monitor a thread pool I might just be interested in the current usage, rather than the status of every thread).

Ah - thanks for clearing up my misunderstanding. I thought I had experimented with the whitelist/blacklist and not just the filters, but my recollection is certainly faulty.  I'll need to experiment with this again!

--

Harald

[Brian Brazil]

On Fri, 5 Jun 2020 at 01:19, Cameron Kerr <cameron...@gmail.com> wrote:

On Friday, 5 June 2020 05:20:55 UTC+12, Harald Koch wrote:

On Thu, Jun 4, 2020, at 08:10, Cameron Kerr wrote:

You should be sure to use the whitelist and blacklist features; browse the available mbean objects using jconsole or similar to find just the bits you will be interested in, and then query only those.

But that's my point - the JMX exporter queries all of your mbeans and only then filters the results using the lists. So you'll still get the problems you mention; locks, db queries, and so on.

True, the default configuration is unhelpful (harmful even) in that it grabs everything, and the provided examples should do a better job of showing something more production-ready. The whitelistObjectNames and blacklistObjectNames are the bits that limit what is actually grabbed, and then the rules kick in to affect the transformation into metrics for prometheus.

PRs to improve the example configs are welcome. The usual challenge is that the expensive mBeans are usually also the ones with the key metrics.

Brian

 

Even then, the white/blacklistObjectNames only apply to the MBean object itself, and not to individual Attributes, which raise its own, similar issues. (eg. if I want to monitor a thread pool I might just be interested in the current usage, rather than the status of every thread).

 

(Also this is an 8-month old thread :)

I realised; my reply for largely for the benefit of people who might find this in a search, as this is not yet well documented.

Cheers,

Cameron

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/25c13f1f-b96d-4a4f-9cb6-80aaafa984b5o%40googlegroups.com.

--

Brian Brazil

www.robustperception.io