Prometheus compaction failed and no space left

Annonyme1

unread,

Jul 2, 2020, 12:01:16 PM7/2/20

to Prometheus Users

hi ! I suddenly couldn't have no data on my grafana dashboard then i got that error message while : systemctl status prometheus :

level=error ts=2020-07-02T15:18:41.764Z caller=db.go:604 component=tsdb msg="compaction failed" err="persist head block: write compaction: write chunks: no space left on device"

How can i solve that please ?

Stuart Clark

unread,

Jul 2, 2020, 12:02:32 PM7/2/20

to Annonyme1, Prometheus Users

That is saying you have run out of disk space wherever Prometheus is
configured to store the database.

--
Stuart Clark

Annonyme1

unread,

Jul 2, 2020, 12:06:28 PM7/2/20

to Prometheus Users

What shall i do to solve this ? i'am running my prometheus on OVH server

can i just resize my local disk ? or any other suggestion ?

Stuart Clark

unread,

Jul 2, 2020, 12:09:18 PM7/2/20

to Annonyme1, Prometheus Users

On 2020-07-02 17:06, Annonyme1 wrote:
> What shall i do to solve this ? i'am running my prometheus on OVH
> server
> can i just resize my local disk ? or any other suggestion ?
>

For Prometheus to load you'll need to either delete something to reduce
space usage (which could include the Prometheus database, losing data)
or expand the available space by resizing disks.

Once it is running again you can prevent future issues by looking at the
retention settings (by time or remaining space).

You can also produce dashboards and alerts to know about trends and know
before you have errors.

--
Stuart Clark

Annonyme1

unread,

Jul 3, 2020, 3:41:04 AM7/3/20

to Prometheus Users

Hi How can i delete something from prometheus database ?
How can i see the size of my prometheus database and it's available free size ?

i'am prometheus noob here !

can you provide me with some tuto to produce that dashboard ?

Brian Candler

unread,

Jul 7, 2020, 7:47:05 AM7/7/20

to Prometheus Users

On Friday, 3 July 2020 08:41:04 UTC+1, Annonyme1 wrote:

Hi How can i delete something from prometheus database ?

https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series

How can i see the size of my prometheus database and it's available free size ?

Simplest just to use standard Unix tools. "Free size" is only limited by your available disk space on the partition where prometheus stores its files.