Jmx Exporter output beans which are not matched in the rules

[briche...@gmail.com]

Hi,

I'm trying to configure the Jmx Exporter to export Kafka beans with a very simple output format (no labels, all flat).

The problem I encounter is that I cannot manage to remove some lines in the output:

jvm_memory_bytes_used{area="heap",} 2.1546968E7
jvm_memory_bytes_used{area="nonheap",} 2.1384864E7
jvm_memory_bytes_committed{area="heap",} 1.25304832E8
jvm_memory_bytes_committed{area="nonheap",} 3.5909632E7
jvm_memory_bytes_max{area="heap",} 2.5952256E8
jvm_memory_bytes_max{area="nonheap",} 1.3631488E8
jvm_memory_pool_bytes_used{pool="Code Cache",} 1152704.0
jvm_memory_pool_bytes_used{pool="Par Eden Space",} 1.6724904E7
jvm_memory_pool_bytes_used{pool="Par Survivor Space",} 760352.0
jvm_memory_pool_bytes_used{pool="CMS Old Gen",} 4061712.0
jvm_memory_pool_bytes_used{pool="CMS Perm Gen",} 2.023216E7
jvm_memory_pool_bytes_committed{pool="Code Cache",} 2555904.0
jvm_memory_pool_bytes_committed{pool="Par Eden Space",} 7.1630848E7
jvm_memory_pool_bytes_committed{pool="Par Survivor Space",} 8912896.0
jvm_memory_pool_bytes_committed{pool="CMS Old Gen",} 4.4761088E7
jvm_memory_pool_bytes_committed{pool="CMS Perm Gen",} 3.3353728E7
jvm_memory_pool_bytes_max{pool="Code Cache",} 5.0331648E7
jvm_memory_pool_bytes_max{pool="Par Eden Space",} 7.1630848E7
jvm_memory_pool_bytes_max{pool="Par Survivor Space",} 8912896.0
jvm_memory_pool_bytes_max{pool="CMS Old Gen",} 1.78978816E8
jvm_memory_pool_bytes_max{pool="CMS Perm Gen",} 8.5983232E7
jvm_gc_collection_seconds_sum{gc="ParNew",} 0.032
jvm_gc_collection_seconds_count{gc="ParNew",} 3.0
jvm_gc_collection_seconds_sum{gc="ConcurrentMarkSweep",} 0.035
jvm_gc_collection_seconds_count{gc="ConcurrentMarkSweep",} 1.0

These lines are always present, whatever rules I configure in my config file.

Is this deliberate ? Can I modify something in the exporter code to get rid of these ?

[Brian Brazil]

Can you share your config?

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[briche...@gmail.com]

Sure.

{
"rules": [
{"pattern": "java.lang<type=Memory><HeapMemoryUsage>(.+): .*" , "name": "java_lang_Memory_HeapMemoryUsage_$1"},
{"pattern": "java.lang<type=Memory><NonHeapMemoryUsage>(.+): .*" , "name": "java_lang_Memory_NonHeapMemoryUsage_$1"},

{"pattern": "java.lang<type=OperatingSystem><.*>OpenFileDescriptorCount: .*" , "name": "java_lang_OperatingSystem_OpenFileDescriptorCount"},
{"pattern": "java.lang<type=OperatingSystem><.*>ProcessCpuLoad: .*" , "name": "java_lang_OperatingSystem_ProcessCpuLoad"},

{"pattern": "java.lang<type=Threading><(.*)>ThreadCount: .*" , "name": "java_lang_Threading_ThreadCount"},

{"pattern": "kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_MessagesInPerSec_$1"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_BytesInPerSec_$1"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_BytesOutPerSec_$1"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=BytesRejectedPerSec><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_BytesRejectedPerSec_$1"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=FailedFetchRequestsPerSec><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_FailedFetchRequestsPerSec_$1"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=FailedProduceRequestsPerSec><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_FailedProduceRequestsPerSec_$1"},

{"pattern": "kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_MessagesInPerSec_$1_$2"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=(.+)><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_BytesInPerSec_$1_$2"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec, topic=(.+)><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_BytesOutPerSec_$1_$2"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=BytesRejectedPerSec, topic=(.+)><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_BytesRejectedPerSec_$1_$2"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=FailedFetchRequestsPerSec, topic=(.+)><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_FailedFetchRequestsPerSec_$1_$2"},
{"pattern": "kafka.server<type=BrokerTopicMetrics, name=FailedProduceRequestsPerSec, topic=(.+)><>(.+): .*" , "name": "kafka_server_BrokerTopicMetrics_FailedProduceRequestsPerSec_$1_$2"},

{"pattern": "kafka.log<type=Log, name=(.+), topic=(.+), partition=(.+)><>Value" , "name": "kafka_log_Log_$1_$2_$3"},

{"pattern": "kafka.cluster<type=Partition, name=(.+), topic=(.+), partition=(.+)><>Value" , "name": "kafka_cluster_Partition_$1_$2_$3"}
]
}

[Brian Brazil]

Ah right, those aren't coming from the JMX collector - those are from the Standard Exports which all client libraries expose.

Brian

}

[briche...@gmail.com]

So, nothing I can do to get rid of these ?

[briche...@gmail.com]

If I can't get rid of these lines, maybe I can format them according to my output format ?

[Brian Brazil]

On 29 June 2015 at 16:43, <briche...@gmail.com> wrote:

If I can't get rid of these lines, maybe I can format them according to my output format ?

The output format is the Prometheus Text Exposition format (see http://prometheus.io/docs/instrumenting/exposition_formats/), so this particular output format is not likely to change.

What are you trying to export to?

Brian

[briche...@gmail.com]

I'm trying to export these metrics to Influxdb 0.8.

I'd like the output format to be : <metric name> <numeric value>

[Brian Brazil]

On 29 June 2015 at 16:52, <briche...@gmail.com> wrote:

I'm trying to export these metrics to Influxdb 0.8.

I'd like the output format to be : <metric name> <numeric value>

I wouldn't be against adding a influxdb bridge to the java client, so that you could take advantage of the rich prometheus client library while not being ready to move to prometheus just yet. It'd then take some relatively simple plumbing to do what you want.

I'd suggest mapping Prometheus labels to InfluxDB tags, as they're the same data model.

[briche...@gmail.com]

Well, with Influxdb 0.8 it's recommended to encode most data in the serie name because attributes are not indexed.

In my particular case, I'd prefer that jmx exporter doesn't talk directly to Influxdb, because I want to be able to do some processing on the data before they go to influx (eg: I'm currently using hekad to fetch that data by http request, do some simple anomaly detection, then output to influx).

The best solution for my problem would be to just add some rules to specify how these beans must be outputted.

[Brian Brazil]

I don't think that intricate a use case is something we can directly support. Within the Prometheus ecosystem, your best option is probably to get it into a prometheus server and then out via console templates in the format you want.

May I ask what sort of anomaly detection you're doing?

[briche...@gmail.com]

Ok, I'll just write some lua code to parse the Prometheus Text Exposition format in hekad then.

As for anomaly detection, I'm using (https://hekad.readthedocs.org/en/v0.9.2/sandbox/index.html?highlight=anomaly#anomaly-detection-module) to detect abnormal variations of http status code in our server logs and in some of our services metrics.