Is it possible to find out how many samples are used to serve a single Prometheus Query?

[pin...@hioscar.com]

Or queries on average?

Thanks,

[Julius Volz]

Hi,

So on one hand, there's two ways to get query stats about a specific query or all queries:

a) When appending "&stats=1" to the PromQL HTTP API call, like https://demo.promlabs.com/api/v1/query?query=up&timestamp=1644357133.302&stats=1, the returned JSON includes some timing statistics,

b) When enabling the query log file (config option global.query_log_file), the query log file contains those stats for every executed query.

However, these stats currently don't include the number of touched samples :(

But the PromQL engine is already tracking the total number of touched samples as part of making sure that the maximum configured sample limit (flag "--query.max-samples") is not exceeded. So it would probably not be hard to expose those stats as well, it's just not done yet.

Cheers,

Julius

On Tue, Feb 8, 2022 at 5:33 PM 'pin...@hioscar.com' via Prometheus Users <promethe...@googlegroups.com> wrote:

Or queries on average?

Thanks,

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f9353c9a-2054-4bf0-b9ee-4619fde45ffen%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Pingjia Shan]

Thanks! 

We're trying to evaluate if we should move to AWS managed Prometheus and finding out how many samples are used by our queries is an important factor to calculate costs.

[Julius Volz]

Makes sense!

There's a couple of existing issues/PRs around this that would need to be pushed forward somehow:

- https://github.com/prometheus/prometheus/issues/10181

- https://github.com/prometheus/prometheus/issues/6668

- https://github.com/prometheus/prometheus/pull/6890