How to read the Prometheus db data.

[Sandeep Rao Kokkirala]

Hi Team,

can i read the data from TSDB for prometheus. 

Currently i have set the retention for 30 days. our cluster have 44 nodes, It occupied 300GB space as of now i.e for 20 days. 

I want to ensure the data get overriden after 30 days and size to maintain max 500GB. 

retention: 30d(Current value set). 

Though deployment level we set the storage size due k8 restrication it will extend beyond the value what we set. 

https://github.com/kubernetes/kubernetes/issues/61641

Appreciate if one aware of this and how to read the data from prometheusdb and any other inputs?

[Brian Candler]

Currently i have set the retention for 30 days. our cluster have 44 nodes, It occupied 300GB space as of now i.e for 20 days. 

I want to ensure the data get overriden after 30 days and size to maintain max 500GB. 

There is an experimental flag --storage.tsdb.retention.size for that. See:

https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects

As for "reading the data from prometheus TSDB", I guess you want the query API:

https://prometheus.io/docs/prometheus/latest/querying/api/

[Sandeep Rao Kokkirala]

Thanks Brain. I have enabled this and tested from grafana i am able to see only 4 days data .Is that mean on 5th day is it overriding the data of 1st day?

[Brian Candler]

Data is stored in chunks, and old chunks are deleted when the configured retention limit (of days and/or size) has been reached.  It's not exactly "overwriting" data, it's just deleting old files and creating new ones.

[Sandeep Rao Kokkirala]

Thanks Brain. 

On Tuesday, June 16, 2020 at 4:49:45 PM UTC+8, Sandeep Rao Kokkirala wrote: