Historical Data

[Adso Castro]

Hey,

What's your opinion about sending Prometheus metrics to a Data Lake or something? My team has a PoC about it and I'm trying to find a way if this is possible or a pure waste of time. The goal is to serve historical data to other teams, basically. By the way, I'm not talking about a prometheus based long term retention, like Thanos.

Thanks.

[Brian Candler]

Is it possible? Yes. There are many remote_write integrations available, see https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

Is it a waste of time? Only you can answer that.  It depends what you want to use it for, and what value you get out of it.

I use remote_write from several satellite prometheus servers, to a central victoriametrics server with a longer retention period configured.  It gives me a useful backup of the data, and a single place I can run queries with a global view.  Also, if those queries are expensive, they don't affect the prometheus servers themselves.

[Adso Castro]

Oh, another person of culture I see. Nice to see a fellow VM user :)

We're running the cluster version of VM too, 6 month retention by the way. 

This is a PoC we're running to understand if we can put those historical series to some use.

Thanks Brian!