Exemplars & histogram buckets

[Vishwanath]

Hi - 

I instrumented an histogram with golang prom sdk and it includes exemplar. When i curl the metrics endpoint, i see below for my histogram with exemplar. As you can see, each bucket can have exemplar, and exemplar has time and they (exemplar) can be old.

When Prometheus with exemplar enabled is scraping this end-point, does it de-dup the exemplars so it doesn't ingest the older exemplars for some buckets, again, for every scrape ?

Thanks for reading and your help.

# HELP exemplar_my_histogram vishwa's test metric3 histogram
# TYPE exemplar_my_histogram histogram
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-3.0"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.9"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.8"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.6999999999999997"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.5999999999999996"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.4999999999999996"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.3999999999999995"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.2999999999999994"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.1999999999999993"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.099999999999999"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.9999999999999991"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.899999999999999"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.799999999999999"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.6999999999999988"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.5999999999999988"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.4999999999999987"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.3999999999999986"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.2999999999999985"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.1999999999999984"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.0999999999999983"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.9999999999999983"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.8999999999999984"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.7999999999999984"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.6999999999999984"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.5999999999999984"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.49999999999999845"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.39999999999999847"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.2999999999999985"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.19999999999999848"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.09999999999999848"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.5265566588595902e-15"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.10000000000000153"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.20000000000000154"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.30000000000000154"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.4000000000000016"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.5000000000000016"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.6000000000000015"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.7000000000000015"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.8000000000000015"} 0
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.9000000000000015"} 1 # {mytraceid="fa128244a04e3167"} 0.81342384094935 1.6916505472120025e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.0000000000000016"} 1
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.1000000000000016"} 1
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.2000000000000017"} 2 # {mytraceid="fa128244a04e3167"} 1.1297683801942673 1.6915601023219924e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.3000000000000018"} 4 # {mytraceid="fa128244a04e3167"} 1.2198116269608419 1.6920360561040416e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.400000000000002"} 5 # {mytraceid="fa128244a04e3167"} 1.358550014547222 1.6919264337043686e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.500000000000002"} 5
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.600000000000002"} 6 # {mytraceid="fa128244a04e3167"} 1.5173031242650117 1.691906796059689e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.7000000000000022"} 6
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.8000000000000023"} 7 # {mytraceid="fa128244a04e3167"} 1.7294844647831784 1.6919916590401986e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.9000000000000024"} 7
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.000000000000002"} 9 # {mytraceid="fa128244a04e3167"} 1.952000333558452 1.692091738121497e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.1000000000000023"} 10 # {mytraceid="fa128244a04e3167"} 2.096613085256622 1.6918952017111053e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.2000000000000024"} 13 # {mytraceid="fa128244a04e3167"} 2.1298900181088167 1.6919736919950492e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.3000000000000025"} 14 # {mytraceid="fa128244a04e3167"} 2.2400071381918214 1.691540474782527e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.4000000000000026"} 15 # {mytraceid="fa128244a04e3167"} 2.390128024808927 1.6917299980148625e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.5000000000000027"} 16 # {mytraceid="fa128244a04e3167"} 2.4096549758317476 1.692204001547885e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.6000000000000028"} 18 # {mytraceid="fa128244a04e3167"} 2.5060113851880983 1.6920551035233934e+09
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.700000000000003"} 18
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.800000000000003"} 18
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.900000000000003"} 18
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="3.000000000000003"} 18
exemplar_my_histogram_bucket{dim1="e",dim2="f",le="+Inf"} 69970 # {mytraceid="fa128244a04e3167"} 644.3222078518065 1.6922187573901772e+09
exemplar_my_histogram_sum{dim1="e",dim2="f"} 3.5056405397655797e+08
exemplar_my_histogram_count{dim1="e",dim2="f"} 69970

[Bryan Boreham]

> does it de-dup the exemplars 

Yes, an exemplar that is a duplicate of one in memory is discarded. 

> so it doesn't ingest the older exemplars for some buckets, again, for every scrape

Prometheus reads the values on every scrape. I don’t know how it could de-dup without reading them. 

Did you have some other meaning of “ingest” in mind?

I think there is an issue if the older exemplar is not in memory : https://github.com/prometheus/prometheus/issues/9377 

Bryan

[Vishwanath N]

thank you for your response Bryan. 

So this means it’s scraped and ingested but dedupe is done by tstb. thanks for pointing out the issue , it’s useful. 

---

From: prometheus...@googlegroups.com <prometheus...@googlegroups.com> on behalf of Bryan Boreham <bjbo...@gmail.com>
Sent: Thursday, August 17, 2023 1:46 AM
To: Prometheus Developers <prometheus...@googlegroups.com>
Subject: [prometheus-developers] Re: Exemplars & histogram buckets

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/f63e600d-e743-4937-bccb-29ee50aa3dbfn%40googlegroups.com.