"no space left on device" issue in prometheus pod

Vaibhav Ingulkar

unread,

Jun 26, 2024, 1:24:03 PM (11 days ago) Jun 26

to Prometheus Users

Hi Team,

We are facing below error in prometheus pod and there after it stops pushing data to mimir -

target=http://pushgateway:9091/metrics msg="appending scrape report failed" err="write to WAL: log samples: write /prometheus/wal/123456: no space left on device"

Prometheus Version : 2.50.1

When verified by pod login it shows /tmp directory becomes full i.e. 11/11 Gi space is consumed. ALso tried to reduced retention period to 1h but still issue persist and /tmp directory gets full.

Shriram Datar

unread,

Jul 1, 2024, 3:07:54 AM (6 days ago) Jul 1

to Prometheus Users

Increase the space.

Vaibhav Ingulkar

unread,

Jul 1, 2024, 3:40:52 AM (6 days ago) Jul 1

to Prometheus Users

Hi @Shiram,

Where can I increase the space? I searched in helm vhart but not found any configuration to increase /tmp directory space.

Can you please suggest on it?

Reply all

Reply to author

Forward