compaction failed: delete persisted head block after failed db reload

62 views

Skip to first unread message

Robin Pharaoh

unread,

Apr 3, 2020, 11:19:56 AM4/3/20

to Prometheus Users

We are using prometheus version 2.15.2 and started seeing compaction failures that are causing all database files after the initial issue to fail to load.

In the graph this looks like a large gap from the start of failure to where the in-memory database begins.

I've ******** out exact names, in console directory names and timestamps are displayed.

level=info ts=2020-04-02T19:00:06.111Z caller=compact.go:496 component=tsdb msg="write block" mint=******** maxt=******** ulid=************ duration=5.648508245s
level=info ts=2020-04-02T19:00:07.180Z caller=head.go:668 component=tsdb msg="head GC completed" duration=115.070561ms
level=error ts=2020-04-02T19:00:13.304Z caller=db.go:609 component=tsdb msg="compaction failed" err="delete persisted head block after failed db reload:*****************: unlinkat /prometheus/****************/chunks: directory not empty"
level=error ts=2020-04-02T19:00:14.572Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory"
level=error ts=2020-04-02T19:01:18.920Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory"
level=error ts=2020-04-02T19:02:27.367Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory"
level=error ts=2020-04-02T19:03:45.459Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory"