Exemplar support

18 views
Skip to first unread message

Maximilian Tagher

unread,
May 27, 2025, 5:10:14 PMMay 27
to Prometheus Developers
Hi, I'm working on adding exemplar support for histograms to a Haskell prometheus client. I wanted to run my implementation by Prometheus folks to see if this all sounds correct:

1. For a given histogram with 5 buckets, store up to 1 exemplar per bucket. Don't store exemplars for +Inf samples.
2. There should be a minimum retention period, such that if you store two exemplars for the same bucket back-to-back, only the first exemplar is stored. My understanding is there are two main purposes of this:

a. Store fewer exemplars in prometheus
b. Letting callers know up-front if an exemplar is likely to make it to prometheus, so that they can sample the trace.

3. There should be a maximum retention period, such that an exemplar is no longer sent to prometheus if it's e.g. older than a minute. I'm less clear on why this is desired, but want to give a good explanation in docs.

Bryan Boreham

unread,
May 27, 2025, 5:38:45 PMMay 27
to Prometheus Developers
1. 1 exemplar per bucket is fine; "le +Inf" is also a bucket. 
(OpenMetrics insists "A bucket MUST NOT have more than one exemplar.", but I don't think the Prometheus code enforces this rule)
2. Prometheus will remember as many exemplars as it can, in a ring buffer.  
2b You can't always predict when Prometheus will next scrape. There might be multiple scrapers, e.g. an HA pair.

I'm not sure about 3; maybe some other client authors will comment.  As far as I know the Go client retains up to 1 exemplar per series forever.
Reply all
Reply to author
Forward
0 new messages