PromQL redirection

[Samit Jain]

We've multiple segregated data systems which support PromQL, each storing metrics for different class of applications, infra, etc.

We would like to explore the possibility of abstracting promql over these systems, such that user can run a query without knowing about the different backends. The options we considered below use something of a brute force approach and won't scale:

1. support remote read API in all backends and configure Prometheus to remote read from all of them.
2. send PromQL query to all backends and merge the results.

I think a system where there is an external 'router' component which knows where different time series' are stored (some sort of an index table) and uses it to query the right backend would be worth exploring. We can presume for now that the time series are unique across all backends. Do you know of something like this exists in some form, or some literature on this that we could build upon?

[Brian Candler]

Have a look at https://github.com/jacksontj/promxy

But I don't think it's yet clever enough to avoid querying servers that couldn't possibly match the query.  (Presumably it could only do that if your PromQL was specific enough with its labels)

[Ben Kochie]

This is somewhat how Thanos works.

It has a gRPC federating protocol that allows you to pull in metric data from multiple TSDBs and glue the results together in the Thanos Query engine.

The Thanos Sidecar already supports Prometheus remote read protocol, so it could be a shim for what you need.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f3c551fc-cfa2-496e-84e4-c96e26843244n%40googlegroups.com.