Text-based histogram exposition format

[Damian Kamiński]

Hi,

I'm writing Prometheus client library and while trying to figure out how to expose histograms I found out that documentation is not really clear.

Docs: https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-example

Example shown there is the most basic one:

# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="0.2"} 100392
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

My questions are:
1. +Inf must be last? Or is it just a convention?
2. How about labels? "le" is the only allowed label or can there be more?
3. If there can be more labels then how about order of buckets?
4. "le" should be first or last label?

Is below example ok (everything ordered alphabetically so +Inf is a first bucket, additional label added)?

# TYPE latency_seconds histogram
latency_seconds_bucket{upstream="bar",le="+Inf"} 123
latency_seconds_bucket{upstream="bar",le="0.001"} 1
latency_seconds_bucket{upstream="bar",le="0.002"} 5
latency_seconds_bucket{upstream="bar",le="0.003"} 6
latency_seconds_bucket{upstream="foo",le="+Inf"} 234
latency_seconds_bucket{upstream="foo",le="0.001"} 1
latency_seconds_bucket{upstream="foo",le="0.002"} 4
latency_seconds_bucket{upstream="foo",le="0.003"} 5
latency_seconds_count{upstream="bar"} 123
latency_seconds_count{upstream="foo"} 234
latency_seconds_sum{upstream="bar"} 5
latency_seconds_sum{upstream="foo"} 8

[Stuart Clark]

1. There should be an inf bucket otherwise there might be entries that don't appear in any, but are included in the count.

2. You can have other labels like any other metric, for example by status.

3 & 4. Label and row ordering doesn't really matter.

What language are you using, as many of the client libraries have support for histograms which sorts all this out for you?

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Damian Kamiński]

We're using Lua + Nginx in high performance scenario (100k req/s+). We had to write our own implementation as there is no suitable clients open sourced.

> Label and row ordering doesn't really matter.

According to Prometheus docs: "The buckets of a histogram and the quantiles of a summary must appear in increasing numerical order of their label values (for the le or the quantile label, respectively)."

Source: https://prometheus.io/docs/instrumenting/exposition_formats/#histograms-and-summaries

Is documentation wrong in that case?

[Bjoern Rabenstein]

On 10.04.20 05:29, Damian Kamiński wrote:
>
> > Label and row ordering doesn't really matter.
>
> According to Prometheus docs: "The buckets of a histogram and the quantiles of
> a summary must appear in increasing numerical order of their label values (for
> the le or the quantile label, respectively)."
> Source: https://prometheus.io/docs/instrumenting/exposition_formats/#
> histograms-and-summaries
>
> Is documentation wrong in that case?

The documentation is probably "right" as in "a consumer has the right
to reject your histogram if it doesn't follow the
documentation". Having said that, the Prometheus server might still
try to be nice and interprete your input in a meaningful
way. Cf. https://en.wikipedia.org/wiki/Robustness_principle (which
also includes the problems with that – something that seems to just
work might suddenly get its faults exposed when it needs to talk to a
less tolerant consumer).

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in