Non-deterministic metric key generation with JMX

[marko]

I'm having trouble with what seems to result from non-deterministic metric key generation occurring when using JMX export.

Prior to upgrading our JVM version I'm receiving two metrics (ParNew and ConcurrentMarkSweep) under the following key:

java_lang_GarbageCollector_CollectionCount

Whereas after switching to a newer JVM version Prometheus generates two distinct metrics:

java_lang_G1_Young_Generation_CollectionCount

java_lang_G1_Old_Generation_CollectionCount

Any ideas what's causing this and how can I make key generation deterministic?

Below is an extract of what jmx_exporter outputs when I do a manual scraping.

It looks like JMX bean property order affects metric key naming.

jmx_exporter % grep CollectionCount scrape1.txt

java.lang{type=GarbageCollector, name=ParNew}[]CollectionCount: 10659

java.lang{type=GarbageCollector, name=ConcurrentMarkSweep}[]CollectionCount: 24

In Prometheus this results in java_lang_GarbageCollector_CollectionCount

% grep CollectionCount scrape2.txt

java.lang{name=G1 Young Generation, type=GarbageCollector}[]CollectionCount: 811

java.lang{name=G1 Old Generation, type=GarbageCollector}[]CollectionCount: 0

[Brian Brazil]

On Thu, 26 Mar 2020 at 12:03, marko <marko....@gmail.com> wrote:

I'm having trouble with what seems to result from non-deterministic metric key generation occurring when using JMX export.

Prior to upgrading our JVM version I'm receiving two metrics (ParNew and ConcurrentMarkSweep) under the following key:

java_lang_GarbageCollector_CollectionCount

Whereas after switching to a newer JVM version Prometheus generates two distinct metrics:

java_lang_G1_Young_Generation_CollectionCount

java_lang_G1_Old_Generation_CollectionCount

Any ideas what's causing this and how can I make key generation deterministic?

Below is an extract of what jmx_exporter outputs when I do a manual scraping.

It looks like JMX bean property order affects metric key naming.

It does affect the default naming, however you shouldn't be relying on the jmx exporter for any java.lang metrics - the agent will provide them itself.

Brian

 

jmx_exporter % grep CollectionCount scrape1.txt

java.lang{type=GarbageCollector, name=ParNew}[]CollectionCount: 10659

java.lang{type=GarbageCollector, name=ConcurrentMarkSweep}[]CollectionCount: 24

In Prometheus this results in java_lang_GarbageCollector_CollectionCount

% grep CollectionCount scrape2.txt

java.lang{name=G1 Young Generation, type=GarbageCollector}[]CollectionCount: 811

java.lang{name=G1 Old Generation, type=GarbageCollector}[]CollectionCount: 0

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/34204c97-94af-46da-a7e7-d2df326e91da%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[marko]

We're specifying the Prometheus Java agent on JVM process startup as follows:

-javaagent:/foo/jmx_prometheus_javaagent-0.12.0.jar=1234:/foo/prometheus.conf

with basically just an empty config.

Being new to Prometheus I'm not sure I fully understand what you mean.

Should this setup already get us the java.lang metrics even if I'd blacklist java.lang metrics in prometheus.conf as follows?

blacklistObjectNames: ["java.lang:"]

marko

[Brian Brazil]

They'll be under process_ and jvm_

 

--

Brian Brazil

www.robustperception.io

[marko]

Got it - thanks!

So, I'll just update my queries as follows:

java_lang_GarbageCollector_CollectionCount ==> jvm_gc_collection_seconds_count

java_lang_GarbageCollector_CollectionTime (in millis) ==> jvm_gc_collection_seconds_sum (in seconds)

If it's not recommended to use java.lang metrics, should I blacklist them from getting exported? Is there any benefit from doing that?

blacklistObjectNames: ["java.lang:*"]

marko

[Brian Brazil]

You'll save a small bit of resources to blacklist them, though usually it'd be more than your rules never matched them in the first places. 

--

Brian Brazil

www.robustperception.io