Thanos compactor issue

[Pratibha Channamsetty]

My thanos compactor is halting when ever it is compacting level2 . 

This is error I see.

 msg="critical error detected; halting" err="compaction: 2 errors: group 0@16765046448932964094: compact blocks

[16765046448932964094/01FSWJYPEV0HC9BW2XYJJC8RM7 @16765046448932964094/01FSWSE82E4WDYHTTQB3M2CZMB]: 2 errors: populate block: add series: read symbols: invalid checksum;

When I delete 1 or 2 blocks from s3 it resume compaction  and again halt with next chunk of blocks. I am worried I will loose all data.

Any help will be great.

[Ben Kochie]

Sounds like you have corrupted blocks.

There is a not well documented feature that allows you to ignore blocks for compaction.

thanos tools bucket mark --marker=no-compact-mark.json --details="Broken block" --id=01FSWJYPEV0HC9BW2XYJJC8RM7 --id=01FSWSE82E4WDYHTTQB3M2CZMB

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/522f5f52-c811-4464-8f32-eefb87463b76n%40googlegroups.com.