Sharing selected data between 2 Prometheis

[yagyans...@gmail.com]

I have a use-case wherein I need to send some filtered data (metrics that match a regex) from one Prometheus to another.
What is the suggested approach for such cases? I was thinking of using remote_read (over remote_write because remote_read is better in terms of resource utilization), but I couldn't find anything in the docs to suggest that I can filter what data to read while using remote_read.
Prometheus Version - v2.34.0

[Brian Candler]

Federation might match your use case, since you can filter metrics by regex. But effectively you are rescraping the data, so the timestamps won't be the same. (Although sometimes this is an advantage, e.g. if you actually want lower resolution data in your copy).

Otherwise, look at remote write, using write_relabel_configs to filter the data, which will copy the data exactly, and buffer it if the remote system is temporarily down. I don't understand why you discarded that option - what makes you think resource utilization is poor for prom A remote-writing to prom B?

Remote read is not an option for syncing data as far as I know - only for performing queries from a remote data source.  There is "backfill" but it is only for historical data; it doesn't work for the head block.

[yagyans...@gmail.com]

Federation might match your use case, since you can filter metrics by regex. But effectively you are rescraping the data, so the timestamps won't be the same. (Although sometimes this is an advantage, e.g. if you actually want lower resolution data in your copy).

>> Hm, I did not think about timestamps being slightly different.

Otherwise, look at remote write, using write_relabel_configs to filter the data, which will copy the data exactly, and buffer it if the remote system is temporarily down. I don't understand why you discarded that option - what makes you think resource utilization is poor for prom A remote-writing to prom B?

>> According to the official docs and quite a few blogs, remote_write increases the resource utilization of source Prometheus by ~ 25%. That is the only reason I'm trying to avoid that.

Remote read is not an option for syncing data as far as I know - only for performing queries from a remote data source.  There is "backfill" but it is only for historical data; it doesn't work for the head block.

>> Maybe, I used the word sync incorrectly. I actually want to query some data available in Prometheus 1 in my 2nd Prometheus and use them for some autoscaling use-case.

[yagyans...@gmail.com]

@Brain
Is there a way at all to do this via remote_read?

[Brian Candler]

I don't know if Prometheus itself implements the remote read protocol as a storage endpoint (I've never come across it).

If it could, I guess you could set up the 2nd prometheus to remote_read the 1st.  But then, you might as well just query the 1st directly. It might only help you if you need to do a single PromQL query which joins metrics on the 1st and the 2nd.

Prometheus does implement the remote write protocol as a storage endpoint, if you enable it with a flag.

[Brian Candler]

On Sunday, 3 September 2023 at 16:59:22 UTC+1 Brian Candler wrote:

I don't know if Prometheus itself implements the remote read protocol as a storage endpoint (I've never come across it).

To answer my own question, it does: see https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/

However "This is not currently considered part of the stable API and is subject to change even between non-major version releases of Prometheus."  (although it has said this since May 2020)