Prometheus federation vs remote_read

[sayf eddine Hammemi]

Hello,

I have a special use case and I want to ask the community opinion on the implementation

Usecase: I want to expose all metrics from a given cluster (not k8s cluster) to a prometheus server through only one endpoint.

multiple components exposing metrics ---> single-endpoint --> main prometheus

I have thought of 3 solutions but with some caveats

Solution1: use a tool that merge metrics from multiple endpoints and expose them at once while injecting missing labels like instance and job

Solution2: Setup a local prometheus server that scrape all metrics and use its /federate endpoint as a single endpoint for the main prometheus server to scrape

Solution3: Use remote_read config option and point it to the local-prometheus/api/v1/read to ingest the local prometheus metrics

Caveats

Solution1 is more like a dirty hack because metrics with different scrape intervals will be complicated to handle through only one endpoint

Solution2: the main prometheus server will timestamp the metrics again from the federation (not sure if federate  endpoint exposes metrics timestamp in that case i may honor timestamps)

Solution3: This looks the best solution so far as I read that remote_read will be more efficient that scraping ? (I may be mistaken)

Which solution you think it is the best, Any other suggestion is welcome

Best regards

[Brian Candler]

It depends what you mean by "single endpoint".

You can install exporter_exporter: you'll get a single endpoint listening on a single port, which proxies to the various backend exporters.  This means you only need to open one firewall port.

However you'll still have to scrape it multiple times, with a different param module=XXX for each backend.

[sayf eddine Hammemi]

The use case is about using as minimal configuration in the main prometheus as possible which means only one configured endpoint, so exporter_exporter is a no go for me.

[Brian Candler]

OK - that's an unusual requirement, since you can build your scrape config from a script.

I don't think your solution 3 will work.  Remote_read sends queries to the remote_engine when handling promQL queries, but I don't think it ingests into the local TSDB.

[sayf eddine Hammemi]

hmm correct the main prometheus can access the data but there no offline store.
Now I am wondering does federation preserve timestamp in recent version or I ll end up with double timestamped data

thnx

[Ben Kochie]

https://github.com/RobustPerception/PushProx supports auto-registration, so you should meet your minimal configuration requirements.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/03ccfb42-d0bb-47f2-8993-9314be4b61af%40googlegroups.com.