Hadoop metric exporter for Prometheus-Grafana

[mark]

Hi

We have open source hadoop 2.7.3 installed on AWS EC2 instances. We already use redis and mongo exporters for promethous exporters for grafana.

We are trying to do the same to collect all the hadoop (namenode and datanode) metrices and display on grafana graphs. We could not find any specific exporter for hadoop metrices. We tried the JMX Exporter and tried to modify the yml file as below keeping in line with cassandra exporter but it is not exporting the metrices. We cant see anything on the web url. We do see similar list of metrices fpr redis but not for mongo.

We tried with:

startDelaySeconds: 0

hostPort: 127.0.0.1:1234

jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:1234/jmxrmi

ssl: false

lowercaseOutputName: false

lowercaseOutputLabelNames: false

whitelistObjectNames: ["org.apache.hadoop.metrics2:*"]

blacklistObjectNames: ["org.apache.hadoop.metrics2:type=ColumnFamily,*"]

rules:

  - pattern: "org.apache.hadoop.metrics2<type=(\\w+), name=(\\w+)><>Value: (\\d+)"

    name: hadoop_$1_$2

    value: $3

    valueFactor: 0.001

    labels: {}

    help: "Hadoop metric $1 $2"

    type: GAUGE

    attrNameSnakeCase: false

And command as :

java -javaagent:./jmx_prometheus_javaagent-0.9.jar=1234:config.yaml -jar hadoop-common-2.7.3.jar

But it is giving errors and not displaying anything. Maybe we are doing the basics itself wrong. 

Can we have some insight as to how do we use jmx exporters for hadoop metrices for Prometheus-Grafana.

Thanks

[Brian Brazil]

On 21 July 2017 at 10:47, mark <nishant....@gmail.com> wrote:

Hi

We have open source hadoop 2.7.3 installed on AWS EC2 instances. We already use redis and mongo exporters for promethous exporters for grafana.

We are trying to do the same to collect all the hadoop (namenode and datanode) metrices and display on grafana graphs. We could not find any specific exporter for hadoop metrices. We tried the JMX Exporter and tried to modify the yml file as below keeping in line with cassandra exporter but it is not exporting the metrices. We cant see anything on the web url. We do see similar list of metrices fpr redis but not for mongo.

We tried with:

startDelaySeconds: 0

 

hostPort: 127.0.0.1:1234

jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:1234/jmxrmi

You need to remove these two lines if you're running it as an agent.

Brian

 

ssl: false

lowercaseOutputName: false

lowercaseOutputLabelNames: false

whitelistObjectNames: ["org.apache.hadoop.metrics2:*"]

blacklistObjectNames: ["org.apache.hadoop.metrics2:type=ColumnFamily,*"]

rules:

  - pattern: "org.apache.hadoop.metrics2<type=(\\w+), name=(\\w+)><>Value: (\\d+)"

    name: hadoop_$1_$2

    value: $3

    valueFactor: 0.001

    labels: {}

    help: "Hadoop metric $1 $2"

    type: GAUGE

    attrNameSnakeCase: false

And command as :

java -javaagent:./jmx_prometheus_javaagent-0.9.jar=1234:config.yaml -jar hadoop-common-2.7.3.jar

But it is giving errors and not displaying anything. Maybe we are doing the basics itself wrong. 

Can we have some insight as to how do we use jmx exporters for hadoop metrices for Prometheus-Grafana.

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d32c0722-4d7a-4c7a-9025-b507865e0feb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[mark]

Thanks for the reply.

So I deleted those two lines from my config.yaml and tried with same command to start the agent. Now, I am getting this error:

2017-07-22 19:00:11.936:INFO:ipjsoejs.Server:jetty-8.y.z-SNAPSHOT

2017-07-22 19:00:11.978:INFO:ipjsoejs.AbstractConnector:Started SelectChann...@0.0.0.0:1234

no main manifest attribute, in hadoop-common-2.7.3.jar

Isn't hadoop-common-2.7.3.jar exposing the hadoop related jvm parameters? Please share some insights here.

Thanks

[Brian Brazil]

On 22 July 2017 at 20:02, mark <nishant....@gmail.com> wrote:

Thanks for the reply.

So I deleted those two lines from my config.yaml and tried with same command to start the agent. Now, I am getting this error:

2017-07-22 19:00:11.936:INFO:ipjsoejs.Server:jetty-8.y.z-SNAPSHOT

2017-07-22 19:00:11.978:INFO:ipjsoejs.AbstractConnector:Started SelectChann...@0.0.0.0:1234

no main manifest attribute, in hadoop-common-2.7.3.jar

Isn't hadoop-common-2.7.3.jar exposing the hadoop related jvm parameters? Please share some insights here.

I don't know hadoop, but I'd guessing you don't have the right jar.

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8f5f3b39-21f8-46c6-a562-da00a6e53f6b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[mark]

Might someone please share some pointers here. Which jar to use here for hadoop metrices?

On Friday, July 21, 2017 at 3:17:00 PM UTC+5:30, mark wrote:

[lauren...@gmail.com]

Hi,

Btw there's following alternative opensource exporter available (written in go): https://github.com/wyukawa/hadoop_exporter

It currently has support to gather a number of metrics for the namenode and journalnode, which we are already using in production including custom alerts.

Datanode support would be easy to add, I created an issue for that: https://github.com/wyukawa/hadoop_exporter/issues/4#issuecomment-330783849

Would be nice to get more people involved, to cover all HDFS related services, and more metrics !

best,

Laurent

[abhi...@qubole.com]

Hi Mark,

I am trying to use Prometheus JMX exporter to collect hadoop metrics similarly. Can you please take a look at the issue below if you've already figured it out:

https://groups.google.com/forum/#!topic/prometheus-users/dEuNmGsdIDI

On Friday, July 21, 2017 at 3:17:00 PM UTC+5:30, mark wrote: