vector cannot contain metrics with the same labelset

[Adso “Cho” Castro]

Hey, been getting the error below when querying:

execution: vector cannot contain metrics with the same labelset

The query:

sum(rate(istio_requests_total{reporter="source", destination_workload="dummy-data", response_code=~"2.."}[$_range])) by (destination_workload) / sum(rate(istio_requests_total{reporter="source", destination_workload="dummy-data"}[$_range])) by (destination_workload)

Funny thing is, this is a Thanos installation. I have a Cortex installation with the same data and this query works fine.

Thanks

[Julius Volz]

Nothing in the query itself should be able to cause a duplicate labelset in a normal PromQL execution, so I assume that something is wrong with the underlying Thanos setup or code. Maybe it somehow manages to select the same duplicate time series from multiple Prometheus servers (or the long-term storage), because something around the external labels normally used for differentiating series isn't configured or working properly?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b08f43a8-fd11-4775-838e-350f144df8e8n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Adso “Cho” Castro]

Hey Julius, morning

Found a few warn logs from thanos-receive, maybe a possible root cause?

- "Error on ingesting samples with different values but same timestamp"

- "Error on ingesting out-of-order samples"

[Julius Volz]

That's the TSDB complaining about a related problem where multiple series with the same label identity are likely coming in. But that shouldn't lead to PromQL queries receiving the same series multiple times in a vector selector unless something is broken in the TSDB / Thanos data merging layer underneath PromQL. Could you share all the details of your Thanos setup (how many Prometheus servers with which external labels do you have, how are you merging them together with Thanos, which Thanos and Prometheus versions are you using, are you using object storage for long-term data, etc.)? Might at some point be a better question / bug report for the Thanos folks though.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/36531430-934f-4334-a78c-66abb1b16aa7n%40googlegroups.com.

[Adso “Cho” Castro]

Oh I see. I agree with that idea, I'm gonna move this discussion to their github issues.

Thank you!

[Adso “Cho” Castro]

Julius, just to keep you updated about this topic, it was a bug from Thanos 0.31.0. I rolled back to the 0.30.0 e the query worked fine. Looks like there is a thread about the issue already and Plotka is looking into it.

Thank you again!