"no space left on device" issue in prometheus pod

Vaibhav Ingulkar

unread,

Jun 26, 2024, 1:24:03 PM6/26/24

to Prometheus Users

Hi Team,

We are facing below error in prometheus pod and there after it stops pushing data to mimir -

target=http://pushgateway:9091/metrics msg="appending scrape report failed" err="write to WAL: log samples: write /prometheus/wal/123456: no space left on device"

Prometheus Version : 2.50.1

When verified by pod login it shows /tmp directory becomes full i.e. 11/11 Gi space is consumed. ALso tried to reduced retention period to 1h but still issue persist and /tmp directory gets full.

Shriram Datar

unread,

Jul 1, 2024, 3:07:54 AM7/1/24

to Prometheus Users

Increase the space.

Vaibhav Ingulkar

unread,

Jul 1, 2024, 3:40:52 AM7/1/24

to Prometheus Users

Hi @Shiram,

Where can I increase the space? I searched in helm vhart but not found any configuration to increase /tmp directory space.

Can you please suggest on it?

Vaibhav Ingulkar

unread,

Jul 17, 2024, 3:08:39 AM7/17/24

to Prometheus Users

Hi Team,

Any updates?

Bryan Boreham

unread,

Jul 18, 2024, 1:56:14 PM7/18/24

to Prometheus Users

You need to place the /prometheus/ directory on its own volume, not have it write to the same place as /tmp.

Look for "persistentVolume" in the helm chart, or in the documentation for it. I don't use helm myself.

Bryan

Vaibhav Ingulkar

unread,

Aug 6, 2024, 2:41:14 AM8/6/24

to Prometheus Users

Thanks Bryan,

Enabled pvc as suggested it's working fine.

Reply all

Reply to author

Forward