Configuring Prometheus JMX exporter for Hadoop2

[abhishekd]

Hi,

I am trying to scrape metrics from following Hadoop2 daemons running on ec2 instance using Prometheus JMX exporter:

i. hadoop namenode

ii. hadoop datanode

iii. yarn resourcemanager

iv. yarn nodemanager

I am trying to run JMX exporter as a java agent with all the four daemons. For this I have added EXTRA_JAVA_OPTS in hadoop-env.sh and yarn-env.sh :

i. export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"

ii. export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"

iii. export YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"

iv. export YARN_NODEMANAGER_OPTS="$YARN_NODEMANAGER_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"

Sample prometheus_config.yml for a resourcemanager metric NumAllSources is as follows :

rules:

 - pattern: Hadoop<service=ResourceManager, name=MetricsSystem, sub=Stats><>NumAllSources

   name: sources

   labels:

    app_id: "hadoop_rm"

I am getting the following exception when I restart the resourcemanager or other daemons with new configs and java_opts :

Exception in thread "main" java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:382)

at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:397)

Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds

at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)

at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:128)

I saw a similar exception while using jmx_exporter to collect Kafka metrics. Any suggestions how to fix this?  

[Brian Brazil]

The javaagent is getting included twice, make sure it's only getting loaded once.

Brian

 

at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)

at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:128)

I saw a similar exception while using jmx_exporter to collect Kafka metrics. Any suggestions how to fix this?  

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/65ab584a-2280-4182-b96c-84286e911864%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io