No joy from Prometheus snapshot

54 views
Skip to first unread message

Mike Spreitzer

unread,
Jul 6, 2020, 9:36:18 PM7/6/20
to Prometheus Users
I am running Prometheus 2.19.1, configured with a retention time of 8 hours.  I took a snapshot, but when I run a prometheus server against the snapshot the server says it has no data.  What am I doing wrong?

The regular server is running in a Kubernetes Pod, with `--storage.tsdb.path` set to a hostPath type of Volume.

Here is a typescript showing a survey of the data directory before the snapshot, then the snapshot operation:

sysop@r26data0:~$ date; find /var/lib/prometheus/; curl -X POST http://localhost:30909/api/v1/admin/tsdb/snapshot
Mon Jul  6 23:25:23 UTC 2020
/var/lib/prometheus/
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/index
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/chunks
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/chunks/000001
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/meta.json
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/tombstones
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/index
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/chunks
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/chunks/000001
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/meta.json
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/tombstones
/var/lib/prometheus/chunks_head
/var/lib/prometheus/chunks_head/000080
/var/lib/prometheus/chunks_head/000079
/var/lib/prometheus/chunks_head/000078
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/index
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/chunks
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/chunks/000001
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/meta.json
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/tombstones
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/index
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/chunks
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/chunks/000001
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/meta.json
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/tombstones
/var/lib/prometheus/wal
/var/lib/prometheus/wal/00000572
/var/lib/prometheus/wal/00000566
/var/lib/prometheus/wal/checkpoint.00000562
/var/lib/prometheus/wal/checkpoint.00000562/00000000
/var/lib/prometheus/wal/00000565
/var/lib/prometheus/wal/00000569
/var/lib/prometheus/wal/00000568
/var/lib/prometheus/wal/00000574
/var/lib/prometheus/wal/00000563
/var/lib/prometheus/wal/00000567
/var/lib/prometheus/wal/00000573
/var/lib/prometheus/wal/00000571
/var/lib/prometheus/wal/00000564
/var/lib/prometheus/wal/00000570
/var/lib/prometheus/queries.active
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/index
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/chunks
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/chunks/000001
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/meta.json
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/tombstones
/var/lib/prometheus/lock
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/index
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/chunks
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/chunks/000001
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/meta.json
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/tombstones
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/index
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/chunks
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/chunks/000001
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/meta.json
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/tombstones
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/index
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/chunks
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/chunks/000001
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/meta.json
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/tombstones
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/index
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/chunks
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/chunks/000001
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/meta.json
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/tombstones
{"status":"success","data":{"name":"20200706T232523Z-207403def63a5b6f"}}sysop@r26data0:~$

Next is a typescript showing what I see after the snapshot.

sysop@r26data0:~$ date; find /var/lib/prometheus/
Mon Jul  6 23:26:06 UTC 2020
/var/lib/prometheus/
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/index
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/chunks
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/chunks/000001
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/meta.json
/var/lib/prometheus/01ECHNMH8MRPZYHM8KXKBE60ZA/tombstones
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/index
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/chunks
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/chunks/000001
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/meta.json
/var/lib/prometheus/01ECJH3E8ENEWVK8E41G5XWM71/tombstones
/var/lib/prometheus/snapshots
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHNMH8MRPZYHM8KXKBE60ZA
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHNMH8MRPZYHM8KXKBE60ZA/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHNMH8MRPZYHM8KXKBE60ZA/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHNMH8MRPZYHM8KXKBE60ZA/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHNMH8MRPZYHM8KXKBE60ZA/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHNMH8MRPZYHM8KXKBE60ZA/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJH3E8ENEWVK8E41G5XWM71
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJH3E8ENEWVK8E41G5XWM71/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJH3E8ENEWVK8E41G5XWM71/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJH3E8ENEWVK8E41G5XWM71/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJH3E8ENEWVK8E41G5XWM71/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJH3E8ENEWVK8E41G5XWM71/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK75HTV8VDRA51493JGBH51
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK75HTV8VDRA51493JGBH51/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK75HTV8VDRA51493JGBH51/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK75HTV8VDRA51493JGBH51/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK75HTV8VDRA51493JGBH51/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK75HTV8VDRA51493JGBH51/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJYTWRJZG6YSGH3XBTJK6CX
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJYTWRJZG6YSGH3XBTJK6CX/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJYTWRJZG6YSGH3XBTJK6CX/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJYTWRJZG6YSGH3XBTJK6CX/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJYTWRJZG6YSGH3XBTJK6CX/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJYTWRJZG6YSGH3XBTJK6CX/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJ3BZRM3SP7XQAZRJ867C30
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJ3BZRM3SP7XQAZRJ867C30/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJ3BZRM3SP7XQAZRJ867C30/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJ3BZRM3SP7XQAZRJ867C30/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJ3BZRM3SP7XQAZRJ867C30/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJ3BZRM3SP7XQAZRJ867C30/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJQZ5GB6TGV21QMHCWZ3JFY
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJQZ5GB6TGV21QMHCWZ3JFY/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJQZ5GB6TGV21QMHCWZ3JFY/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJQZ5GB6TGV21QMHCWZ3JFY/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJQZ5GB6TGV21QMHCWZ3JFY/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJQZ5GB6TGV21QMHCWZ3JFY/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJA7Q18RT0VW9S5HKSVDP0J
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJA7Q18RT0VW9S5HKSVDP0J/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJA7Q18RT0VW9S5HKSVDP0J/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJA7Q18RT0VW9S5HKSVDP0J/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJA7Q18RT0VW9S5HKSVDP0J/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECJA7Q18RT0VW9S5HKSVDP0J/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHWG8GCAZWMW1JDZB9ERQ0N
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHWG8GCAZWMW1JDZB9ERQ0N/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHWG8GCAZWMW1JDZB9ERQ0N/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHWG8GCAZWMW1JDZB9ERQ0N/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHWG8GCAZWMW1JDZB9ERQ0N/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHWG8GCAZWMW1JDZB9ERQ0N/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK5PQETDE8VDRSR6NZ6FGXQ
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK5PQETDE8VDRSR6NZ6FGXQ/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK5PQETDE8VDRSR6NZ6FGXQ/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK5PQETDE8VDRSR6NZ6FGXQ/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK5PQETDE8VDRSR6NZ6FGXQ/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECK5PQETDE8VDRSR6NZ6FGXQ/tombstones
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHERT02S0HSGNA5ENN4MZ35
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHERT02S0HSGNA5ENN4MZ35/index
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHERT02S0HSGNA5ENN4MZ35/chunks
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHERT02S0HSGNA5ENN4MZ35/chunks/000001
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHERT02S0HSGNA5ENN4MZ35/meta.json
/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f/01ECHERT02S0HSGNA5ENN4MZ35/tombstones
/var/lib/prometheus/chunks_head
/var/lib/prometheus/chunks_head/000080
/var/lib/prometheus/chunks_head/000079
/var/lib/prometheus/chunks_head/000078
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/index
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/chunks
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/chunks/000001
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/meta.json
/var/lib/prometheus/01ECJYTWRJZG6YSGH3XBTJK6CX/tombstones
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/index
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/chunks
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/chunks/000001
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/meta.json
/var/lib/prometheus/01ECJ3BZRM3SP7XQAZRJ867C30/tombstones
/var/lib/prometheus/wal
/var/lib/prometheus/wal/00000572
/var/lib/prometheus/wal/00000566
/var/lib/prometheus/wal/checkpoint.00000562
/var/lib/prometheus/wal/checkpoint.00000562/00000000
/var/lib/prometheus/wal/00000565
/var/lib/prometheus/wal/00000569
/var/lib/prometheus/wal/00000568
/var/lib/prometheus/wal/00000574
/var/lib/prometheus/wal/00000563
/var/lib/prometheus/wal/00000567
/var/lib/prometheus/wal/00000573
/var/lib/prometheus/wal/00000571
/var/lib/prometheus/wal/00000564
/var/lib/prometheus/wal/00000570
/var/lib/prometheus/queries.active
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/index
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/chunks
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/chunks/000001
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/meta.json
/var/lib/prometheus/01ECJQZ5GB6TGV21QMHCWZ3JFY/tombstones
/var/lib/prometheus/lock
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/index
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/chunks
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/chunks/000001
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/meta.json
/var/lib/prometheus/01ECJA7Q18RT0VW9S5HKSVDP0J/tombstones
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/index
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/chunks
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/chunks/000001
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/meta.json
/var/lib/prometheus/01ECHWG8GCAZWMW1JDZB9ERQ0N/tombstones
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/index
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/chunks
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/chunks/000001
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/meta.json
/var/lib/prometheus/01ECK5PQETDE8VDRSR6NZ6FGXQ/tombstones
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/index
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/chunks
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/chunks/000001
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/meta.json
/var/lib/prometheus/01ECHERT02S0HSGNA5ENN4MZ35/tombstones

Next I tried running another Prometheus server against that snapshot, with an empty config file.  Note that the log message say nothing about any data.

sysop@r26data0:/var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f$ sudo -u nobody ~/prometheus --storage.tsdb.path==.  --web.enable-admin-api --config.file=$HOME/prom-config/config.yaml
level=info ts=2020-07-07T01:16:49.251Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2020-07-07T01:16:49.251Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.19.1, branch=HEAD, revision=eba3fdcbf0d378b66600281903e3aab515732b39)"
level=info ts=2020-07-07T01:16:49.251Z caller=main.go:338 build_context="(go=go1.14.4, user=root@62700b3d0ef9, date=20200618-16:35:26)"
level=info ts=2020-07-07T01:16:49.251Z caller=main.go:339 host_details="(Linux 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 r26data0 (none))"
level=info ts=2020-07-07T01:16:49.252Z caller=main.go:340 fd_limits="(soft=65535, hard=65535)"
level=info ts=2020-07-07T01:16:49.252Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-07-07T01:16:49.254Z caller=main.go:678 msg="Starting TSDB ..."
level=info ts=2020-07-07T01:16:49.254Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-07-07T01:16:49.258Z caller=head.go:645 component=tsdb msg="Replaying WAL and on-disk memory mappable chunks if any, this may take a while"
level=info ts=2020-07-07T01:16:49.259Z caller=head.go:706 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2020-07-07T01:16:49.259Z caller=head.go:709 component=tsdb msg="WAL replay completed" duration=815.925µs
level=info ts=2020-07-07T01:16:49.261Z caller=main.go:694 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-07-07T01:16:49.261Z caller=main.go:695 msg="TSDB started"
level=info ts=2020-07-07T01:16:49.261Z caller=main.go:799 msg="Loading configuration file" filename=/home/sysop/prom-config/config.yaml
level=info ts=2020-07-07T01:16:49.268Z caller=main.go:827 msg="Completed loading of configuration file" filename=/home/sysop/prom-config/config.yaml
level=info ts=2020-07-07T01:16:49.268Z caller=main.go:646 msg="Server is ready to receive web requests."

But when I look, it has no time series:

sysop@r26data0:~$ curl http://localhost:9090/api/v1/metadata
{"status":"success","data":{}}sysop@r26data0:~$

I also used its web UI.  I visited "/status", and got the following.




Runtime Information

Uptime 2020-07-07 01:16:49.253648426 +0000 UTC
Working Directory /var/lib/prometheus/snapshots/20200706T232523Z-207403def63a5b6f
Configuration reload Successful
Last successful configuration reload 2020-07-07 01:16:49 +0000 UTC
WAL corruptions 0
Goroutines 32
GOMAXPROCS 8
GOGC
GODEBUG
Storage Retention 15d

Build Information

Version 2.19.1
Revision eba3fdcbf0d378b66600281903e3aab515732b39
Branch HEAD
BuildUser root@62700b3d0ef9
BuildDate 20200618-16:35:26
GoVersion go1.14.4

Alertmanagers

Endpoint

Head Stats

Number Of Series Number of Chunks Current Max Time Current Min Time
0 0 -9223372036854775808 9223372036854775807
Total Query Time: 0.000 Seconds

Highest Cardinality Labels

Name Count

Highest Cardinality Metric Names

Name Count

Label Names With Highest Cumulative Label Value Length

Name Length

Most Common Label Pairs

NameCount









Thanks,
Mike

Mike Spreitzer

unread,
Jul 6, 2020, 10:01:28 PM7/6/20
to Prometheus Users
I suspected the relative directory and then noticed the double equal in the prometheus command line.  So I erased the old snapshot and tried again.

sysop@r26data0:/var/lib/prometheus/snapshots$ ls
sysop@r26data0:/var/lib/prometheus/snapshots$ curl -X POST http://localhost:30909/api/v1/admin/tsdb/snapshot
{"status":"success","data":{"name":"20200707T015331Z-b7fbfbcafd915bb"}}sysop@r26data0:/var/lib/prometheus/snapshots$
sysop@r26data0:/var/lib/prometheus/snapshots$ ls -la
total 12
drwxr-xr-x  3 nobody nogroup 4096 Jul  7 01:53 .
drwxr-xr-x 14 nobody   65533 4096 Jul  7 00:59 ..
drwxr-xr-x 12 nobody nogroup 4096 Jul  7 01:53 20200707T015331Z-b7fbfbcafd915bb

Next I run the server again.  This time it logs messages about finding data blocks.

sysop@r26data0:/var/lib/prometheus/snapshots/20200707T015331Z-b7fbfbcafd915bb$ sudo -u nobody ~/prometheus --storage.tsdb.path=$PWD  --web.enable-admin-api --config.file=$HOME/prom-config/config.yaml
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.19.1, branch=HEAD, revision=eba3fdcbf0d378b66600281903e3aab515732b39)"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:338 build_context="(go=go1.14.4, user=root@62700b3d0ef9, date=20200618-16:35:26)"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:339 host_details="(Linux 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 r26data0 (none))"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:340 fd_limits="(soft=65535, hard=65535)"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-07-07T01:57:08.664Z caller=main.go:678 msg="Starting TSDB ..."
level=info ts=2020-07-07T01:57:08.664Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594015200000 maxt=1594022400000 ulid=01ECHNMH8MRPZYHM8KXKBE60ZA
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594022400000 maxt=1594029600000 ulid=01ECHWG8GCAZWMW1JDZB9ERQ0N
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594029600000 maxt=1594036800000 ulid=01ECJ3BZRM3SP7XQAZRJ867C30
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594036800000 maxt=1594044000000 ulid=01ECJA7Q18RT0VW9S5HKSVDP0J
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594044000000 maxt=1594051200000 ulid=01ECJH3E8ENEWVK8E41G5XWM71
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594051200000 maxt=1594058400000 ulid=01ECJQZ5GB6TGV21QMHCWZ3JFY
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594058400000 maxt=1594065600000 ulid=01ECJYTWRJZG6YSGH3XBTJK6CX
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594065600000 maxt=1594072800000 ulid=01ECK5PQETDE8VDRSR6NZ6FGXQ
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594072800000 maxt=1594080000000 ulid=01ECKCJEPAJ19AJ7DXJY6K6P6B
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1594080000000 maxt=1594086829277 ulid=01ECKFMSZ5XDWD5296R3C8ZXZ6
level=info ts=2020-07-07T01:57:08.688Z caller=head.go:645 component=tsdb msg="Replaying WAL and on-disk memory mappable chunks if any, this may take a while"
level=info ts=2020-07-07T01:57:08.688Z caller=head.go:706 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2020-07-07T01:57:08.688Z caller=head.go:709 component=tsdb msg="WAL replay completed" duration=365.54µs
level=info ts=2020-07-07T01:57:08.690Z caller=main.go:694 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-07-07T01:57:08.690Z caller=main.go:695 msg="TSDB started"
level=info ts=2020-07-07T01:57:08.690Z caller=main.go:799 msg="Loading configuration file" filename=/home/sysop/prom-config/config.yaml
level=info ts=2020-07-07T01:57:08.694Z caller=main.go:827 msg="Completed loading of configuration file" filename=/home/sysop/prom-config/config.yaml
level=info ts=2020-07-07T01:57:08.694Z caller=main.go:646 msg="Server is ready to receive web requests."

The last data block ends about 5 minutes ago.

sysop@r26data0:~$ date --date @1594086829
Tue Jul  7 01:53:49 UTC 2020

But still no data.

sysop@r26data0:~$ curl http://localhost:9090/api/v1/metadata
{"status":"success","data":{}}sysop@r26data0:~$

The web UI shows the following for "/status".

Runtime Information

Uptime 2020-07-07 01:57:08.663553256 +0000 UTC
Working Directory /var/lib/prometheus/snapshots/20200707T015331Z-b7fbfbcafd915bb
Configuration reload Successful
Last successful configuration reload 2020-07-07 01:57:08 +0000 UTC
0 0 1594086829277 1594086829277

Mike Spreitzer

unread,
Jul 6, 2020, 10:19:29 PM7/6/20
to Prometheus Users
Interestingly, the Prometheus server running against the new snapshot logged some compactions after about a minute.

...
level=info ts=2020-07-07T01:57:08.694Z caller=main.go:646 msg="Server is ready to receive web requests."
level=info ts=2020-07-07T01:58:18.185Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1594015200000 maxt=1594036800000 ulid=01ECKFX8HKYEP8Y2YPX2WB0PC3 sources="[01ECHNMH8MRPZYHM8KXKBE60ZA 01ECHWG8GCAZWMW1JDZB9ERQ0N 01ECJ3BZRM3SP7XQAZRJ867C30]" duration=9.494006165s
level=info ts=2020-07-07T01:58:27.860Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1594036800000 maxt=1594058400000 ulid=01ECKFXHV47D51X0GREQM6QZE3 sources="[01ECJA7Q18RT0VW9S5HKSVDP0J 01ECJH3E8ENEWVK8E41G5XWM71 01ECJQZ5GB6TGV21QMHCWZ3JFY]" duration=9.64782793s
level=info ts=2020-07-07T01:58:37.619Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1594058400000 maxt=1594080000000 ulid=01ECKFXV9PSG91VCTAMAVKZ1YR sources="[01ECJYTWRJZG6YSGH3XBTJK6CX 01ECK5PQETDE8VDRSR6NZ6FGXQ 01ECKCJEPAJ19AJ7DXJY6K6P6B]" duration=9.725054346s
level=info ts=2020-07-07T01:58:53.953Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1594015200000 maxt=1594080000000 ulid=01ECKFY4TFCY2J3KQ30VSZFRKS sources="[01ECKFX8HKYEP8Y2YPX2WB0PC3 01ECKFXHV47D51X0GREQM6QZE3 01ECKFXV9PSG91VCTAMAVKZ1YR]" duration=16.305728942s

This server logged finding 10 blocks when it started (see my previous email); those compactions compacted the first 9 blocks into one.

Mike Spreitzer

unread,
Jul 7, 2020, 1:19:24 AM7/7/20
to Prometheus Users
 I was surprised to find that despite the negative signs noted above, this thing _does_ answer queries!

sysop@r26data0:~$ curl 'http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_chunks&time=2020-07-06T14:00:00Z'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"prometheus_tsdb_head_chunks","app":"prometheus","instance":"10.242.236.4:9090","job":"kubernetes-service-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"default"},"value":[1594044000,"992698"]}]}}sysop@r26data0:~$


This prompts a follow-up question.  Is there an HTTP API query I can make that will get the list that pops up in the web UI as shown in the attached screenshot?  As noted above, it is _not_ "/api/v1/metadata".

Screen Shot 2020-07-07 at 1.16.38 AM.png


Thanks,
Mike

Brian Brazil

unread,
Jul 7, 2020, 4:11:25 AM7/7/20
to Mike Spreitzer, Prometheus Users
On Tue, 7 Jul 2020 at 06:19, Mike Spreitzer <mspr...@us.ibm.com> wrote:
 I was surprised to find that despite the negative signs noted above, this thing _does_ answer queries!

sysop@r26data0:~$ curl 'http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_chunks&time=2020-07-06T14:00:00Z'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"prometheus_tsdb_head_chunks","app":"prometheus","instance":"10.242.236.4:9090","job":"kubernetes-service-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"default"},"value":[1594044000,"992698"]}]}}sysop@r26data0:~$


This prompts a follow-up question.  Is there an HTTP API query I can make that will get the list that pops up in the web UI as shown in the attached screenshot? 

That's the label values api for __name__.
 
As noted above, it is _not_ "/api/v1/metadata".

Metric metadata is only stored in memory, so you need to have active scrapes for it to show up.

Brian
 

Screen Shot 2020-07-07 at 1.16.38 AM.png


Thanks,
Mike

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/91bd5af1-dd7a-40ad-b6c2-92215e57e7fbo%40googlegroups.com.


--
Reply all
Reply to author
Forward
0 new messages