Prometheus and HdrHistogram

[Ross Black]

Hi,

I have just started using prometheus, and had a few questions about sending data via the scrape interface.

I am converting some existing code in Java (using custom metrics instrumentation code) to expose data to prometheus (instead of sending it to graphite).
The existing instrumentation code only provides counters (implemented using AtomicLong) and histograms (implemented using HdrHistogram).
I have been using the protocol buffer protocol and have so far successfully exported data to prometheus, but was a bit uncertain about some details.

Counter
My custom counter instrumentation is slightly different to the prometheus implementation.  Even though the custom counter is always incrementing, it resets its counter value on each capture (so that it only reports the delta from the last report period).
- Will it work OK to report the delta each prometheus scrape, or do I need to keep a running tally?  (or instead report using prometheus Gauge?)
- What does prometheus do when a process restarts and the counter resets to 0?  (this was why the existing custom code reports deltas)

Histogram
-  I assume a histogram always expects a fixed (defined) set of ranges?
     Does this range have to be the same for all histograms with the same name?
     What happens if the range of the buckets change for a histogram?
- Do buckets have to be sent for all of the predefined ranges on every scrape, even if the bucket has zero count?
- Does anybody have any experience of sending data from HdrHistogram in the bucket format required by prometheus?
  HdrHistogram allows output of a number of different intervals (https://github.com/HdrHistogram/HdrHistogram#iteration) - linear and logarithmic seem most appropriate (also provides recorded intervals, and percentile intervals but I believe these change the bounds on every snapshot).  I was just intending to use log intervals, but wondered if anyone had experience with this mapping.
- If I understand correctly, the final bucket sent must always have an upper bound of +Inf?  When using protobuf, is using java.lang.Double#POSITIVE_INFINITY correct?  Does this bucket always have to be sent (even if empty)?

HdrHistogram metric
Would there be any interest in having HdrHistogram added as a basic metric to prometheus?  (I believe it has a defined wire format that is exceptionally compressed).
It would be a while before I could even contemplating having enough knowledge to write or assist writing something like that, but it would be exceptional useful for my usage since it would remove the need to map to specific bucket sizes.


Apologies for the long ramble. Thanks,
Ross

[Brian Brazil]

On Mon, Oct 19, 2015 at 11:32 AM, Ross Black <ross.w...@gmail.com> wrote:

Hi,

I have just started using prometheus, and had a few questions about sending data via the scrape interface.

I am converting some existing code in Java (using custom metrics instrumentation code) to expose data to prometheus (instead of sending it to graphite).
The existing instrumentation code only provides counters (implemented using AtomicLong) and histograms (implemented using HdrHistogram).
I have been using the protocol buffer protocol and have so far successfully exported data to prometheus, but was a bit uncertain about some details.

Is there some reason you're not using the official java client? https://github.com/prometheus/client_java

 

Counter
My custom counter instrumentation is slightly different to the prometheus implementation.  Even though the custom counter is always incrementing, it resets its counter value on each capture (so that it only reports the delta from the last report period).
- Will it work OK to report the delta each prometheus scrape, or do I need to keep a running tally?  (or instead report using prometheus Gauge?)

It should be a running tally. A delta each scrape is very fragile, as if there's more than one scraper you'll lose data.

 

- What does prometheus do when a process restarts and the counter resets to 0?  (this was why the existing custom code reports deltas)

The rate function handles this automatically.

 

Histogram
-  I assume a histogram always expects a fixed (defined) set of ranges?

Yes.

 

     Does this range have to be the same for all histograms with the same name?

Yes.

 

     What happens if the range of the buckets change for a histogram?

Monitoring for that histogram will break until all servers have the new value. This is generally okay in practice given how rarely it happens.

 

- Do buckets have to be sent for all of the predefined ranges on every scrape, even if the bucket has zero count?

Yes.

 

- Does anybody have any experience of sending data from HdrHistogram in the bucket format required by prometheus?

That wouldn't make sense, as HdrHistogram has dynamic buckets and thus can't be aggregated.

 

  HdrHistogram allows output of a number of different intervals (https://github.com/HdrHistogram/HdrHistogram#iteration) - linear and logarithmic seem most appropriate (also provides recorded intervals, and percentile intervals but I believe these change the bounds on every snapshot).  I was just intending to use log intervals, but wondered if anyone had experience with this mapping.

Usually you only want a handful, so manually specifying them is sufficient.

 

- If I understand correctly, the final bucket sent must always have an upper bound of +Inf?  When using protobuf, is using java.lang.Double#POSITIVE_INFINITY correct?  Does this bucket always have to be sent (even if empty)?

It's optional, if set it must match the count.

HdrHistogram metric
Would there be any interest in having HdrHistogram added as a basic metric to prometheus?  (I believe it has a defined wire format that is exceptionally compressed).

This would fall under Summary. See https://github.com/prometheus/client_java/issues/85

 

It would be a while before I could even contemplating having enough knowledge to write or assist writing something like that, but it would be exceptional useful for my usage since it would remove the need to map to specific bucket sizes.

I think what you want to do in this case is write a custom collector for the existing java simpleclient, rather than implementing a whole client yourself. https://github.com/prometheus/client_java/tree/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot has some examples.

Brian

 

Apologies for the long ramble. Thanks,
Ross

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Ross Black]

Thanks for the quick response Brian.

On Monday, 19 October 2015 21:50:14 UTC+11, Brian Brazil wrote:

On Mon, Oct 19, 2015 at 11:32 AM, Ross Black <ross.w...@gmail.com> wrote:

Hi,

I have just started using prometheus, and had a few questions about sending data via the scrape interface.

I am converting some existing code in Java (using custom metrics instrumentation code) to expose data to prometheus (instead of sending it to graphite).
The existing instrumentation code only provides counters (implemented using AtomicLong) and histograms (implemented using HdrHistogram).
I have been using the protocol buffer protocol and have so far successfully exported data to prometheus, but was a bit uncertain about some details.

Is there some reason you're not using the official java client? https://github.com/prometheus/client_java

Primarily because of the existing and working code.
My instrumentation requirements are very simple, but also require low overhead (especially regards memory allocation). HdrHistogram has proven to work exceptional well in this scenario to provide low-cost low-GC collection of timing of frequent paths that are instrumented.  There was also an element of avoiding lock-in of instrumented code to a particular monitoring/reporting system.

I was slightly confused about the status of the java client - old + new client, where the new client did not appear to support protobuf (which seems to be the recommended protocol)?
My application already uses java-nano for protobuf, and it seemed easier to generate protobuf messages from existing metrics.  Is the new client intending to use protobuf?

 

Counter
My custom counter instrumentation is slightly different to the prometheus implementation.  Even though the custom counter is always incrementing, it resets its counter value on each capture (so that it only reports the delta from the last report period).
- Will it work OK to report the delta each prometheus scrape, or do I need to keep a running tally?  (or instead report using prometheus Gauge?)

It should be a running tally. A delta each scrape is very fragile, as if there's more than one scraper you'll lose data.

 

- What does prometheus do when a process restarts and the counter resets to 0?  (this was why the existing custom code reports deltas)

The rate function handles this automatically.

 

Histogram
-  I assume a histogram always expects a fixed (defined) set of ranges?

Yes.

 

     Does this range have to be the same for all histograms with the same name?

Yes.

 

     What happens if the range of the buckets change for a histogram?

Monitoring for that histogram will break until all servers have the new value. This is generally okay in practice given how rarely it happens.

 

- Do buckets have to be sent for all of the predefined ranges on every scrape, even if the bucket has zero count?

Yes.

 

- Does anybody have any experience of sending data from HdrHistogram in the bucket format required by prometheus?

That wouldn't make sense, as HdrHistogram has dynamic buckets and thus can't be aggregated.

 

  HdrHistogram allows output of a number of different intervals (https://github.com/HdrHistogram/HdrHistogram#iteration) - linear and logarithmic seem most appropriate (also provides recorded intervals, and percentile intervals but I believe these change the bounds on every snapshot).  I was just intending to use log intervals, but wondered if anyone had experience with this mapping.

Usually you only want a handful, so manually specifying them is sufficient.

 

- If I understand correctly, the final bucket sent must always have an upper bound of +Inf?  When using protobuf, is using java.lang.Double#POSITIVE_INFINITY correct?  Does this bucket always have to be sent (even if empty)?

It's optional, if set it must match the count.

HdrHistogram metric
Would there be any interest in having HdrHistogram added as a basic metric to prometheus?  (I believe it has a defined wire format that is exceptionally compressed).

This would fall under Summary. See https://github.com/prometheus/client_java/issues/85

 

I have probably mis-understood this issue, but it appeared to be talking more about using HdrHistogram in the java client.  I was wondering about adding raw HdrHistogram support into the prometheus engine and sending HdrHistogram as wire protocol?
 

It would be a while before I could even contemplating having enough knowledge to write or assist writing something like that, but it would be exceptional useful for my usage since it would remove the need to map to specific bucket sizes.

I think what you want to do in this case is write a custom collector for the existing java simpleclient, rather than implementing a whole client yourself. https://github.com/prometheus/client_java/tree/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot has some examples.

I did start down that path initially, but concluded (probably incorrectly) that there was more code involved than simply rendering protobuf directly from my existing metrics objects.
I will have another look.

Thanks for the info.
Ross

[Brian Brazil]

On Mon, Oct 19, 2015 at 12:39 PM, Ross Black <ross.w...@gmail.com> wrote:

Thanks for the quick response Brian.

On Monday, 19 October 2015 21:50:14 UTC+11, Brian Brazil wrote:

On Mon, Oct 19, 2015 at 11:32 AM, Ross Black <ross.w...@gmail.com> wrote:

Hi,

I have just started using prometheus, and had a few questions about sending data via the scrape interface.

I am converting some existing code in Java (using custom metrics instrumentation code) to expose data to prometheus (instead of sending it to graphite).
The existing instrumentation code only provides counters (implemented using AtomicLong) and histograms (implemented using HdrHistogram).
I have been using the protocol buffer protocol and have so far successfully exported data to prometheus, but was a bit uncertain about some details.

Is there some reason you're not using the official java client? https://github.com/prometheus/client_java

Primarily because of the existing and working code.
My instrumentation requirements are very simple, but also require low overhead (especially regards memory allocation). HdrHistogram has proven to work exceptional well in this scenario to provide low-cost low-GC collection of timing of frequent paths that are instrumented. 

There's benchmarks up at https://github.com/prometheus/client_java/tree/master/benchmark. A basic increment should take no more than 20ns under high contention, and there should be no object allocation if you pre-fetch any children you need.

 

There was also an element of avoiding lock-in of instrumented code to a particular monitoring/reporting system.

The prometheus clients aren't intended to lock you in. The Python client can export to Graphite for example, and there's a New Relic bridge currently in review. These will be ported to Java in due course, the required hooks are already there.

 

I was slightly confused about the status of the java client - old + new client, where the new client did not appear to support protobuf (which seems to be the recommended protocol)?
My application already uses java-nano for protobuf, and it seemed easier to generate protobuf messages from existing metrics.  Is the new client intending to use protobuf?

The new simpleclient is the one you should use, protobuf only has benefits in cases where you care about every last drop of performance in exposition. Protobuf may be an optional extra for the simpleclient at some point.

 

That's require some thought, as what we fundamentally want is the ability to determine the quantile over any given time range. I suspect that HdrHistogram won't quite work with that or our data model more generally.

 

 

It would be a while before I could even contemplating having enough knowledge to write or assist writing something like that, but it would be exceptional useful for my usage since it would remove the need to map to specific bucket sizes.

I think what you want to do in this case is write a custom collector for the existing java simpleclient, rather than implementing a whole client yourself. https://github.com/prometheus/client_java/tree/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot has some examples.

I did start down that path initially, but concluded (probably incorrectly) that there was more code involved than simply rendering protobuf directly from my existing metrics objects.

It should be a similar amount of work. I recently improved the API for Python in this regard, and will be porting that to Java.

Brian

 

I will have another look.

Thanks for the info.
Ross

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io