Crash in WAL "last page of the wal is torn, filling it with zeros"

[Prudhvi Acharya]

Prometheus version: 2.8.0
Alertmanager version: 0.15.3
host size
CPU - 4
MEM - 64GB
Scrape-interval - 60 seconds
Running in Kubernetes

args:

        - --storage.tsdb.retention.time=720h
        - --config.file=/etc/prometheus-shared/prometheus.yml
        - --storage.tsdb.path=/data
        - --web.console.libraries=/etc/prometheus/console_libraries
        - --web.console.templates=/etc/prometheus/consoles
        - --web.enable-lifecycle
        - --log.level=info
        - --storage.tsdb.max-block-duration=2h
        - --storage.tsdb.min-block-duration=2h

• Logs:

level=info ts=2019-08-19T12:43:49.011977607Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.8.0, branch=HEAD, revision=59369491cfdfe8dcb325723d6d28a837887a07b9)"
level=info ts=2019-08-19T12:43:49.012122816Z caller=main.go:322 build_context="(go=go1.11.5, user=root@4c4d5c29b71f, date=20190312-07:46:58)"
level=info ts=2019-08-19T12:43:49.012173765Z caller=main.go:323 host_details="(Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 infra-tsdb-server-0 (none))"
level=info ts=2019-08-19T12:43:49.012219427Z caller=main.go:324 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-08-19T12:43:49.012259029Z caller=main.go:325 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-08-19T12:43:49.01383382Z caller=main.go:640 msg="Starting TSDB ..."
level=info ts=2019-08-19T12:43:49.013991874Z caller=web.go:418 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=warn ts=2019-08-19T12:43:49.01957651Z caller=wal.go:116 component=tsdb msg="last page of the wal is torn, filling it with zeros" segment=/data/wal/00000102

[Simon Pasquier]

In general it is best to not tweak storage.tsdb.min-block-duration and
storage.tsdb.max-block-duration. Can you check whether your Prometheus
pod gets OOMed? I suspect that the "last page of the wal is torn,
filling it with zeros" message is because Prometheus is killed
abruptly.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1f21b3c4-b1ac-4786-a10c-044c1f6182bd%40googlegroups.com.