Promethus Deleted Data blocks !!
36 views
Skip to first unread message
Midhun K
Apr 26, 2020, 10:47:31 PM4/26/20
to Prometheus Users
Issue : Prometheus Restarting with below error
``` level=error ts=2020-04-27T00:09:41.712Z caller=main.go:740 err="opening storage failed: block dir: \"/prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ\": open /prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ/meta.json: no such file or directory" ```
``` level=error ts=2020-04-27T00:09:41.712Z caller=main.go:740 err="opening storage failed: block dir: \"/prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ\": open /prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ/meta.json: no such file or directory" ```
Observation : 1. Checked the Datablocks & findout we lost around 300GB of datablocks.
2. We have one disk space alert , Assuming that disk got full before invoking this issue and something trigger the deleting on old datablock the disk got free.
2. We have one disk space alert , Assuming that disk got full before invoking this issue and something trigger the deleting on old datablock the disk got free.
3. Moved the corrupted datablocks to /tmp directory so promethus start working but old metrics are lost & After start its using high resource 2X CPU load & high Memory utilisation.
Q1. Any Idea Why this error got triggered ?
Q2. What is the Normal behaviour of Promethus if no disk space is available for storing the metrics ?
Q3 . If its deleted automatically then any parameters are available to tweak this configurations ?
Brian Candler
Apr 27, 2020, 2:32:25 AM4/27/20
to Prometheus Users
Prometheus version?
Reply all
Reply to author
Forward
0 new messages