Prometheus storage wrt Pushgateway metrics

[anuj tyagi]

Hi All,

I have a question for a use case. If we are pushing batch jobs metrics to Pushgateway.

Some of the job groups of those metrics are getting pushed to pushgateway every 24 hrs. So, metric values are updating once in a day. 

There are other job groups pushing to pushgateway every 15 seconds, and updating metrics values every 15 seconds in Pushgateway. 

Eg.
Backup_timestamp: xxxxx
Backup_files_count: xxxx
so values are getting updated for same metrics. So, All these requests are overwriting the metrics value so not much increase in storage with time. 

Now, Prometheus is scraping all the jobs every 30 seconds. Even job groups with metrics getting pushed every 24 hrs at a time in Pushgateway, prometheus is scraping every 30 seconds. 

Do you think Pushgateway scraping with such short time interval adding storage even though metric value stay same for 24 hrs.

For this reason, one way is to clean Pushgateway job which are older than maybe few seconds like 50 seconds. So, Prometheus will not scrape job at all. This way I can save Prometheus storage and scraping effort? 

Consider I'm pushing 10k metrics in total part of different job groups. Half of those are getting pushed/updated to Pushgateway only once in a day? 

So, the question is how much it impact on storage of Prometheus if Prometheus scraping metrics from Pushgateway every 30 seconds with no change in value for 1 day.

[Stuart Clark]

The storage usage for a metric that isn't changing is next to nothing,
so I wouldn't worry about it. What you describe would be exactly how I'd
expect Pushgateway to be behaving - some metrics are updated more
frequently and others less, but they are always there and being scraped
at the same frequency.

--
Stuart Clark

[anuj tyagi]

Yes. So, Prometheus is scraping at every scrape interval. Is it adding to Prometheus storage at every scrape interval even if no change in value in Pushgateway metrics? 

--

Regards,

Anuj Tyagi