/usr/local/bin/prometheus --version
prometheus, version 2.20.0 (branch: HEAD, revision: e5a06b483527d4fe0704b8fa3a2b475b661c526f)
build user: r
build date: 20200722-18:51:45
go version: go1.14.6
sudo du -Pshx /var/lib/prometheus/*
73G /var/lib/prometheus/chunks_head
0 /var/lib/prometheus/lock
0 /var/lib/prometheus/queries.active
120G /var/lib/prometheus/wal
Data partition was filled a some point, we increased it but since than noticing that wal partition is growing all the time. In the logs seeing
prometheus[2172]: level=info ts=2020-08-17T19:14:25.361Z caller=head.go:760 component=tsdb msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
Also - seeing these errors:
level=error ts=2020-08-17T19:26:19.628Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"