err="opening storage failed: block dir: unexpected end of JSON input"

[Nial mccallister]

when I try to start prometheus I get this error in the logs

err="opening storage failed: block dir: \"/var/lib/prometheus/metrics2/01EWX0Y64BRZ749WK3AFSKJ98K\": unexpected end of JSON input"

[Nial mccallister]

full log

Feb 02 20:35:41 Hydrogen systemd[1]: Started Monitoring system and time series database.

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.574969007Z caller=main.go:300 msg="Starting Prometheus" version="(version=2.7.1+ds, branch=debian/sid, revision=2.7.1+ds-3)"

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.575740934Z caller=main.go:301 build_context="(go=go1.11.5, user=pkg-go-ma...@lists.alioth.debian.org, date=20190210-18:10:02)"

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.576023121Z caller=main.go:302 host_details="(Linux 5.4.83-v7+ #1379 SMP Mon Dec 14 13:08:57 GMT 2020 armv7l Hydrogen (none))"

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.576268746Z caller=main.go:303 fd_limits="(soft=8192, hard=8192)"

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.576487132Z caller=main.go:304 vm_limits="(soft=unlimited, hard=unlimited)"

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.638680725Z caller=web.go:411 component=web msg="Start listening for connections" address=0.0.0.0:9090

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.638458225Z caller=main.go:618 msg="Starting TSDB ..."

Feb 02 20:35:44 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:44.983999319Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1610344800000 maxt=1610409600000 ulid=01EVT8KEW2FBT7VEDA3FMQ3EW2

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.025813746Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1610409600000 maxt=1610474400000 ulid=01EVW6CE38RRV1SQW1702SGGBB

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.050734892Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1610474400000 maxt=1610539200000 ulid=01EVY461P9CB9QY47RQ2T5EF9J

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.092900048Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1610539200000 maxt=1610604000000 ulid=01EW01Z3ENEW83R9PCKFBTNCXP

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.111339215Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1610604000000 maxt=1610668800000 ulid=01EW1ZRMHH8DDJAEBRER4D6V8B

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.154788954Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1610668800000 maxt=1610719200000 ulid=01EWG4YHPGPYPXWV77ZFSRPW1S

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.197002548Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1611136800000 maxt=1611187200000 ulid=01EWHN0THG5BTYJDV1W4FYF9YQ

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.238659475Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1611187200000 maxt=1611252000000 ulid=01EWKBYFK4QRQTZ4NYFPZH3ZMN

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.263393694Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1611252000000 maxt=1611316800000 ulid=01EWN9RAHT9J3MEJPZ0AVPQP5T

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.282775517Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1611316800000 maxt=1611381600000 ulid=01EWQ7HJ8FR4D4X0QNXJYJDKKX

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.301695256Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1611381600000 maxt=1611446400000 ulid=01EWS5B3EYP7DWCJHPMNZ48H43

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.320386454Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1611446400000 maxt=1611511200000 ulid=01EWV34MVFTPX7Y917V0ZDHR3K

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.33257859Z caller=main.go:487 msg="Stopping scrape discovery manager..."

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.332825725Z caller=main.go:501 msg="Stopping notify discovery manager..."

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.332952913Z caller=main.go:523 msg="Stopping scrape manager..."

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.333081923Z caller=manager.go:737 component="rule manager" msg="Stopping rule manager..."

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.333049267Z caller=main.go:483 msg="Scrape discovery manager stopped"

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.333200413Z caller=manager.go:743 component="rule manager" msg="Rule manager stopped"

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.333277808Z caller=main.go:497 msg="Notify discovery manager stopped"

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.333333642Z caller=notifier.go:521 component=notifier msg="Stopping notification manager..."

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.333467236Z caller=main.go:677 msg="Notifier manager stopped"

Feb 02 20:35:45 Hydrogen prometheus[437]: level=info ts=2021-02-02T20:35:45.347324267Z caller=main.go:517 msg="Scrape manager stopped"

Feb 02 20:35:45 Hydrogen prometheus[437]: level=error ts=2021-02-02T20:35:45.347679163Z caller=main.go:686 err="opening storage failed: block dir: \"/var/lib/prometheus/metrics2/01EWX0Y64BRZ749WK3AFSKJ98K\": unexpected end of JSON input"

[Nial mccallister]

output of ls -l

pi@Hydrogen:~ $ ls -l /var/lib/prometheus/metrics2/01EWX0Y64BRZ749WK3AFSKJ98K

total 904

drwxr-xr-x 2 prometheus prometheus   4096 Jan 25 15:00 chunks

-rw-r--r-- 1 prometheus prometheus 916885 Jan 25 15:00 index

-rw-r--r-- 1 prometheus prometheus      0 Jan 26 20:49 meta.json

-rw-r--r-- 1 prometheus prometheus      9 Jan 25 15:00 tombstone

[Stuart Clark]

Did the server running Prometheus run out of storage?

-- 
Stuart Clark

[Nial mccallister]

12GB of storage left, this is a raspberry pi btw

[Stuart Clark]

On 02/02/2021 23:28, Nial mccallister wrote:
> 12GB of storage left, this is a raspberry pi btw

It looks like you have data corruption. If it wasn't due to running out
of disk space did anything else happen (power loss, etc.) that could
have caused this?

As the data is corrupted you probably will need to remove that block and
lose the data contained. If you have a backup you could try a recovery.

--
Stuart Clark

[Nial mccallister]

how do I remove the block, just delete the folder?

[Julius Volz]

Yes, you can just delete any block directory individually and you will lose only the time range of that block and it won't affect anything else.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ac222bf7-19e0-478a-8ffc-fb2a008ffa0cn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com