Scraping multiple entries of the same metric with different values

26 views
Skip to first unread message

Panem78

unread,
Sep 15, 2020, 8:32:34 AM9/15/20
to Prometheus Users

Hello everyone ! I have the following question:
I have created a flask application that upon request in it's /metrics endpoint, retrieves specific values (strings) from Redis in the form
"test_service_metric{service="ui",component="graphs",env="mm"} 180" ,

concatenates them according to prometheus exposition format (each entry separated by newline character and a line at the end), and finally returns in the Response a string like the following:

"test_service_metric{service="ui",component="graphs",env="mm"} 160
test_service_metric{service="ui",component="graphs",env="mm"} 170
test_service_metric{service="ui",component="graphs",env="mm"} 180"

So what i am actually trying to achieve, is prometheus to retrieve all these entries by scraping (which refer to the same metric but with different values) and display them in the console/graph. But what actually happens, is that only one of them actually shows in Prometheus.

Is there something i am doing wrong ? Or because the scrape time is the same for all these entries, all but one are regarded as duplicates and removed ?

Thanks in advance and please let me know for any clarifications !

Brian Candler

unread,
Sep 15, 2020, 9:27:45 AM9/15/20
to Prometheus Users
Each timeseries has to have a different set of labels.  You've returned the same set of labels three times, so this is the same timeseries repeated three times, and prometheus rejects the additional data as duplicate.

You cannot "back fill" values in Prometheus.  That is, you cannot export points 160@t1, 170@t2, 180@t3 in the same scrape.  You can export value 160, and then on next scrape 170, and then on next scrape 180; and the timestamps assigned to those points by prometheus will be the times that the scrapes took place.

If you need to back-fill historical data, then prometheus is not for you.  However VictoriaMetrics might meet your needs, as it's very Prometheus-like (it implements a superset of the PromQL API) but *does* support back-filling in various different formats.

Panem78

unread,
Sep 15, 2020, 9:45:14 AM9/15/20
to Prometheus Users
Thanks a lot for your answer. This completely made it clear for me. 
Reply all
Reply to author
Forward
0 new messages