Java/Scala Metrics in a Cluster/Mulitnode & Graphite Reporter

[nightwol...@gmail.com]

Hey folks,

Question about running metrics in an app on multiple machines/JVMs. 

If I have a application that runs multiple instances. How should I name my MetricsRegistries? Should they include some host name? 

I'm using the Graphite reporter so I'm curious if I have multiple instances reporting to the same key what happens? 

When using CodaHale Metrics in Java or Scala code a clustered environment what are the gotchas when reporting to Graphite?

If I have multiple instances of my application running and creating different metrics can Graphite cope - i.e. is reporting cumulative?

For example if I have AppInstance A and B. If B has a gauge reporting 1.2 and another reporting 1.3 - what would be the result in Ganglia? Will it be an average? or will one override the other.

Are counters cumulative?

Are timers cumulative?

Or should I somehow give each instance some tag to differentiate different JVM instances?

http://stackoverflow.com/questions/27087768/java-scala-metrics-codahale-cluster-mulitnode-graphite-reporter 

Cheers,

~N

[Mark Crossfield]

Hey NightWolf

At the moment Metrics is only able to output aggregated metrics per JVM. You mentioned Graphite, so I’ll talk about that—I’m not familiar with Ganglia although I have heard of it. Graphite only accepts one winning write per bucket per metric—it does not perform culumation / aggregation against a single metric—so you will want to namespace your metrics against a server name / JVM instance so that they don’t overwrite each other. You can achieve some level of aggregation on Graphite using functions or using the aggregation-rules mechanism, but this will be an aggregate of an already aggregated value, losing interesting features and giving equal weight to each instance (although the measurements may be dramatically unbalanced). For example, you might use the nPercentile function to take the median of the medians reported by Metrics for a timer namespaced by each JVM. If those JVM’s are under different load, the median you come up with would not be the median of the individual measurements.

So in summary, yes; namespace your metrics to differentiate between JVMs :)

Cheers, Mark

[Ryan Rupp]

I don't use graphite but I think the general approach here is that you use the GraphiteReporters prefix capability to prefix naming data about the host/jvm/application. See - https://github.com/dropwizard/metrics/blob/master/metrics-graphite/src/main/java/com/codahale/metrics/graphite/GraphiteReporter.java#L68

So, you could have some format like:

<cluster>.<host>.<metric_name>

So your prefix when creating the reporter would be whatever <cluster>.<host> is e.g.:

mycluster.host1

mycluster.host2

Then you end up with metrics in graphite like:

mycluster.host1.mytimer

mycluster.host2.mytimer