Prometheus-style cumulative histograms

[Michael Hobbs]

Hi All,

Just looking to find some help from the community. I'm setting up mtail to track latency from an apache custom log format and it's all working fine. However I notice that the native histogram type is not cumulative and, as such, not what  I believe prometheus expects when it scrapes such metrics.

e.g.

My metrics look like:

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.01"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.025"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.05"} 3

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.075"} 2

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.1"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.25"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.5"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.75"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="1"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="10"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="+Inf"} 0

Whereas I would expect them to look like:

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.01"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.025"} 0

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.05"} 3

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.075"} 5

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.1"} 5

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.25"} 5

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.5"} 5

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="0.75"} 5

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="1"} 5

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="10"} 5

http_requests_seconds_bucket{endpoint_name="/is/e/o/<uuid>.jpeg",host_header="www.foo.com",method="GET",le="+Inf"} 5

For context:

https://prometheus.io/docs/concepts/metric_types/#histogram (which links to https://en.wikipedia.org/wiki/Histogram#Cumulative_histogram )

and described further in https://www.robustperception.io/why-are-prometheus-histograms-cumulative

Is this something that is known and accepted? Are people creating their own prometheus histograms by evaluating their metric and incrementing counters with the appropriate le buckets? Or am I doing something wrong with my mtail config?

Thanks,

Mike Hobbs

[Michael Hobbs]

Ignore me, I updated from r26 to r29 and now it works as expected :)

[Jamie Wilkinson]

Yay! I was about to asked what version because I just fixed that issue :)

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To post to this group, send email to mtail...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/c41a0d96-b166-4f9f-bb6d-9ea99e3bbf4f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.