PromQL - vector cannot contain metrics with the same labelset

3,969 views
Skip to first unread message

Dan S

unread,
Jan 24, 2021, 4:37:08 PM1/24/21
to Prometheus Users
Hi,

Not sure what I'm doing wrong when trying to divide

count by (__name__) ({job="kubernetes-pods", app="scylla"})

scrape_series_added{} 70
scrape_samples_scraped{}  70
scylla_node_operation_mode{}  70
scrape_samples_post_metric_relabeling{} 70
scrape_duration_seconds{} 70
up{}  70

When I try to divide by 70 (to find number of time series per pod), I get an error:

count by (__name__) ({job="kubernetes-pods", app="scylla"}) / 70
Error executing query: vector cannot contain metrics with the same labelset

how can I get a basic divide working? so that I end up with something like:

scrape_series_added{} 1
scrape_samples_scraped{}  1
scylla_node_operation_mode{}  1
scrape_samples_post_metric_relabeling{} 1
scrape_duration_seconds{} 1
up{}  1

Basically I want to know exactly how many time series per metric each "app" is generating per pod

Thanks
Dan

Julius Volz

unread,
Jan 25, 2021, 7:40:06 AM1/25/21
to Dan S, Prometheus Users
Hi,

The problem you're running into is that a binary operator (division, in your case) removes the metric name (because the sample value no longer has that same metric name meaning after the transformation), but the metric name is the only distinguishing label in your case on these series. You could work around that by first copying the metric name to a non-special label, like "metric":

label_join(count by (__name__) ({job="kubernetes-pods", app="scylla"}), "metric", "", "__name__") / 70

Regards,
Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d3562084-7a0f-4012-82d2-218036b730a5n%40googlegroups.com.


--
Julius Volz
PromLabs - promlabs.com

Dan S

unread,
Jan 25, 2021, 5:28:33 PM1/25/21
to Prometheus Users
Awesome Julian, thanks!  Works great. 
Reply all
Reply to author
Forward
0 new messages