Translating Spring java metrics to Prometheus format

[matt...@gmail.com]

With a Spring boot app I can get it to automatically produce a http /metrics endpoint with data I'm interested in but it's a JSON string eg:

{"mem":189952,"mem.free":86871,"processors":4,"uptime":8838288,"instance.uptime":8823684,"systemload.average":0.0,"heap.committed":189952,"heap.init":32768,"heap.used":103080,"heap":458752,"threads.peak":20,"threads.daemon":18,"threads":20,"classes":5481,"classes.loaded":5481,"classes.unloaded":0,"gc.ps_scavenge.count":17,"gc.ps_scavenge.time":225,"gc.ps_marksweep.count":1,"gc.ps_marksweep.time":96,"httpsessions.max":-1,"httpsessions.active":0,"counter.status.200.greeting":6,"counter.status.200.health":1,"counter.status.200.metrics":560,"counter.status.200.star-star.favicon.ico":1,"counter.status.304.star-star.favicon.ico":1,"counter.status.404.star-star":1,"gauge.response.greeting":2.0,"gauge.response.health":226.0,"gauge.response.metrics":2.0,"gauge.response.star-star":6.0,"gauge.response.star-star.favicon.ico":25.0}

With the JMX Exporter there's a way to specify mappings to translate the received data into Prometheus format. Is there a way of specifying pre-processing mappings for a http endpoint?

Thanks

Matt Reynolds

[Julius Volz]

Hey Matt,

Unfortunately there's no way to do that (yet, at least) in the Prometheus server itself. The way we usually approach this is by writing an exporter, a bridge process which sits in between the exporting process and Prometheus and translates the metrics. See all the available ones: http://prometheus.io/docs/instrumenting/exporters/

In this specific case, if you're in control of the application code itself, perhaps it's even possible to transform the metrics data into Prometheus's format in-process, and expose them using the Prometheus client library?

Cheers,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Reynolds]

Hi Julius, thanks for the quick response. It looks like it should be fairly easy for me to get the app to produce it's metrics via JMX so I could use the JMX exporter then. I might also be able to figure out a way to provide an alternative metrics implementation and use the Prometheus Java client for that.

Do you think it would make sense in the future to be able to specify a mapping file on a prometheus conf and have it translate on input or offload that to a http exporter ?  I took a look at the statsd exporter and it looks like that could be the basis for a general http exporter so I might try and fork that and give it a go if you think it's the right direction. 

Given that I know nothing about golang, and I have plenty of other stuff on my plate, I wouldn't hold my breath though...

[Brian Brazil]

On 5 March 2015 at 00:35, Matt Reynolds <matt...@gmail.com> wrote:

Hi Julius, thanks for the quick response. It looks like it should be fairly easy for me to get the app to produce it's metrics via JMX so I could use the JMX exporter then. I might also be able to figure out a way to provide an alternative metrics implementation and use the Prometheus Java client for that.

Do you think it would make sense in the future to be able to specify a mapping file on a prometheus conf and have it translate on input or offload that to a http exporter ?  I took a look at the statsd exporter and it looks like that could be the basis for a general http exporter so I might try and fork that and give it a go if you think it's the right direction. 

The approach we're generally looking at is to have exporters that can take metrics from the various instrumentation systems  and expose them from the same process in the prometheus format. Which instrumentation system is this?

Brian

[Matt Reynolds]

java Spring Boot (via Spring Boot Actuator) which is also used by Spring cloud generates metrics that by default get exposed as JSON to a /metrics endpoint for a web project ( see http://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html ). Now I've just started looking into this and it looks like I may be able to switch the generated metrics to JMX and use that exporter or maybe (and this is just speculation right now) provide a Prometheus client implementation of a metrics interface. The end result is I can probably find a way to talk to Prometheus without having to write a new exporter. 

[Julius Volz]

Yeah, that's the most preferable solution because Prometheus will directly be able to scrape your target without any intermediary then. That ensures the most useful job/instance labels and also will enable you to do simple up-ness monitoring via the synthetic "up" metric (see http://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series).

[Brian Brazil]

On 5 March 2015 at 16:35, Matt Reynolds <matt...@gmail.com> wrote:

java Spring Boot (via Spring Boot Actuator) which is also used by Spring cloud generates metrics that by default get exposed as JSON to a /metrics endpoint for a web project ( see http://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html ). Now I've just started looking into this and it looks like I may be able to switch the generated metrics to JMX and use that exporter or maybe (and this is just speculation right now) provide a Prometheus client implementation of a metrics interface. The end result is I can probably find a way to talk to Prometheus without having to write a new exporter. 

It looks like that'll integrate automatically with codahale metrics, so once we have a codehale collector that'll all work.

Are you aware of the Prometheus Java simpleclient? It's our own metrics system, http://www.boxever.com/easy-java-instrumentation-with-prometheus describes how to use it.

Brian

[Matt Reynolds]

Yep - it looks like I can drop in codehale and use it to route the generated metrics for me via JMX or something else. I did see the java simpleclient and that was what I was thinking of to provide a metrics implementation - the Spring Boot libs generate a lot of metrics for you under the covers and I assume they write to an interface that you can swap in a provider for - implementation of "gauge" and "counter" but the codehale route looks the most flexible.

Thanks Brian and Julius

[Brian Brazil]

On 5 March 2015 at 18:51, Matt Reynolds <matt...@gmail.com> wrote:

Yep - it looks like I can drop in codehale and use it to route the generated metrics for me via JMX or something else. I did see the java simpleclient and that was what I was thinking of to provide a metrics implementation - the Spring Boot libs generate a lot of metrics for you under the covers and I assume they write to an interface that you can swap in a provider for - implementation of "gauge" and "counter" but the codehale route looks the most flexible.

Prometheus-style metrics are more expressive and useful than that provided by spring/codahale due to offering labels, a nicer api, and being correctly aggregatable.

You should use the java simpleclient where possible, and only take data from other instrumentation/metric systems when there's no other choice. The simpleclient includes JMV stats such as memory, GC and cpu usage for example.

Brian