Data partition growing

67 views
Skip to first unread message

deln...@gmail.com

unread,
Aug 26, 2020, 3:23:10 AM8/26/20
to Prometheus Users
Capture-wal-corruption.PNG
/usr/local/bin/prometheus --version
prometheus, version 2.20.0 (branch: HEAD, revision: e5a06b483527d4fe0704b8fa3a2b475b661c526f)
  build user:       r
  build date:       20200722-18:51:45
  go version:       go1.14.6

 sudo du -Pshx /var/lib/prometheus/*
73G     /var/lib/prometheus/chunks_head
0       /var/lib/prometheus/lock
0       /var/lib/prometheus/queries.active
120G    /var/lib/prometheus/wal

Data partition was filled a some point, we increased it but since than noticing that wal partition is growing all the time. In the logs seeing 

prometheus[2172]: level=info ts=2020-08-17T19:14:25.361Z caller=head.go:760 component=tsdb msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"

Also - seeing these errors:

level=error ts=2020-08-17T19:26:19.628Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"


Reply all
Reply to author
Forward
0 new messages