Compaction not running

17 views

Skip to first unread message

Kevin Black

unread,

May 13, 2020, 11:38:24 PM5/13/20

to Prometheus Users

I have a prometheus 2.16.0 instance that does not seem to even try to compact blocks anymore. It's been running fine for some time, has free cpu/memory/disk (both space and inodes), but new blocks arent' created. I see a <blockid>.tmp is created in our data directories, but a few minutes later it goes away. The logs do not show any errors or messages related to compact.go duritng this. I've increased logging, and completely flushed the wal directory, but new blocks still aren't written. I have a secondary server with identical configs, and it works fine, so this does not seem to be load related. Any ideas or tips on what I can try to get this healthy?

Brian Brazil

unread,

May 14, 2020, 4:07:48 AM5/14/20

to Kevin Black, Prometheus Users

On Thu, 14 May 2020 at 04:38, Kevin Black <kevinbla...@gmail.com> wrote:

I have a prometheus 2.16.0 instance that does not seem to even try to compact blocks anymore. It's been running fine for some time, has free cpu/memory/disk (both space and inodes), but new blocks arent' created. I see a <blockid>.tmp is created in our data directories, but a few minutes later it goes away. The logs do not show any errors or messages related to compact.go duritng this. I've increased logging, and completely flushed the wal directory, but new blocks still aren't written. I have a secondary server with identical configs, and it works fine, so this does not seem to be load related. Any ideas or tips on what I can try to get this healthy?