Prometheus pod keeps restarting

[Anoop]

Hi,

I have configured Prometheus in Kubernetes with 2 replicas. One of the two pods keeps restarting, but the other pod is working fine without any issues. I could not find any errors when describing the pod. Resource utilization is normal. The node has enough resources and storage available to run the Prometheus pod and store the data. The Prometheus log for the affected pod is provided below..

ts=2023-10-04T20:18:42.029Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1696298400000 maxt=1696305600000 ulid=01HBSYGH69GH5KVHETGXCVA0K6
ts=2023-10-04T20:18:42.030Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock
ts=2023-10-04T20:18:43.722Z caller=head.go:488 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2023-10-04T20:18:43.722Z caller=head.go:522 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=25.416µs
ts=2023-10-04T20:18:43.722Z caller=head.go:528 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2023-10-04T20:18:48.538Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=97
ts=2023-10-04T20:21:37.477Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=97
ts=2023-10-04T20:24:44.730Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=97
ts=2023-10-04T20:28:52.391Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=97
ts=2023-10-04T20:32:49.630Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=97
ts=2023-10-04T20:33:32.135Z caller=main.go:776 level=warn msg="Received SIGTERM, exiting gracefully..."
ts=2023-10-04T20:33:32.135Z caller=main.go:799 level=info msg="Stopping scrape discovery manager..."
ts=2023-10-04T20:33:32.135Z caller=main.go:813 level=info msg="Stopping notify discovery manager..."
ts=2023-10-04T20:33:32.135Z caller=main.go:835 level=info msg="Stopping scrape manager..."
ts=2023-10-04T20:33:32.135Z caller=main.go:829 level=info msg="Scrape manager stopped"
ts=2023-10-04T20:33:32.135Z caller=main.go:795 level=info msg="Scrape discovery manager stopped"
ts=2023-10-04T20:33:32.135Z caller=main.go:809 level=info msg="Notify discovery manager stopped"
ts=2023-10-04T20:33:32.135Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..."
ts=2023-10-04T20:33:32.135Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped"
ts=2023-10-04T20:33:32.135Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2023-10-04T20:33:32.135Z caller=main.go:1055 level=info msg="Notifier manager stopped"
ts=2023-10-04T20:36:25.242Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=97
ts=2023-10-04T20:37:16.354Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=97
ts=2023-10-04T20:37:18.434Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=97
ts=2023-10-04T20:37:20.454Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=97
ts=2023-10-04T20:37:22.493Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=9 maxSegment=97
ts=2023-10-04T20:37:24.548Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=10 maxSegment=97
ts=2023-10-04T20:37:26.625Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=11 maxSegment=97
ts=2023-10-04T20:37:28.597Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=12 maxSegment=97
ts=2023-10-04T20:37:30.632Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=13 maxSegment=97
ts=2023-10-04T20:37:32.638Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=14 maxSegment=97
ts=2023-10-04T20:37:34.585Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=15 maxSegment=97
ts=2023-10-04T20:37:36.726Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=16 maxSegment=97
ts=2023-10-04T20:37:38.708Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=17 maxSegment=97
ts=2023-10-04T20:37:40.706Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=18 maxSegment=97
ts=2023-10-04T20:37:42.688Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=19 maxSegment=97
ts=2023-10-04T20:37:44.635Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=20 maxSegment=97
ts=2023-10-04T20:37:46.616Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=21 maxSegment=97
ts=2023-10-04T20:37:48.611Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=22 maxSegment=97
ts=2023-10-04T20:37:53.567Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=23 maxSegment=97
ts=2023-10-04T20:37:57.835Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=24 maxSegment=97
ts=2023-10-04T20:37:59.312Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=25 maxSegment=97
ts=2023-10-04T20:38:00.824Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=26 maxSegment=97
ts=2023-10-04T20:38:02.312Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=27 maxSegment=97
ts=2023-10-04T20:38:03.740Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=28 maxSegment=97
ts=2023-10-04T20:38:05.160Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=29 maxSegment=97
ts=2023-10-04T20:38:06.648Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=30 maxSegment=97
ts=2023-10-04T20:38:08.184Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=31 maxSegment=97
ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=32 maxSegment=97
ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=33 maxSegment=97
ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=34 maxSegment=97
ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=35 maxSegment=97
ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=36 maxSegment=97
ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=37 maxSegment=97
ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=38 maxSegment=97
ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=39 maxSegment=97
ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=40 maxSegment=97
ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=41 maxSegment=97
ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=42 maxSegment=97
ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=43 maxSegment=97
ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=44 maxSegment=97
ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=45 maxSegment=97
ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=46 maxSegment=97
ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=47 maxSegment=97
ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=48 maxSegment=97
ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=49 maxSegment=97
ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=50 maxSegment=97
ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=51 maxSegment=97
ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=52 maxSegment=97
ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=53 maxSegment=97
ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=54 maxSegment=97
ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=55 maxSegment=97
ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=56 maxSegment=97
ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=57 maxSegment=97
ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=58 maxSegment=97
ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=59 maxSegment=97
ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=60 maxSegment=97
ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=61 maxSegment=97
ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=62 maxSegment=97
ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=63 maxSegment=97
ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=64 maxSegment=97
ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=65 maxSegment=97
ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=66 maxSegment=97
ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=67 maxSegment=97
ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=68 maxSegment=97
ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=69 maxSegment=97
ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=70 maxSegment=97
ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=71 maxSegment=97
ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=72 maxSegment=97
ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=73 maxSegment=97
ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=74 maxSegment=97
ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=75 maxSegment=97
ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=76 maxSegment=97
ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=77 maxSegment=97
ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=78 maxSegment=97
ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=79 maxSegment=97
ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=80 maxSegment=97
ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=81 maxSegment=97
ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=82 maxSegment=97
ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=83 maxSegment=97
ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=84 maxSegment=97
ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=85 maxSegment=97
ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=86 maxSegment=97
ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=87 maxSegment=97
ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=88 maxSegment=97
ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=89 maxSegment=97
ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=90 maxSegment=97
ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=91 maxSegment=97
ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=92 maxSegment=97
ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=93 maxSegment=97
ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=94 maxSegment=97
ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=95 maxSegment=97
ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=96 maxSegment=97
ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=97 maxSegment=97
ts=2023-10-04T20:38:09.167Z caller=head.go:605 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=911.162µs wal_replay_duration=19m25.444199083s total_replay_duration=19m25.445156166s
ts=2023-10-04T20:38:22.160Z caller=main.go:945 level=info fs_type=EXT4_SUPER_MAGIC
ts=2023-10-04T20:38:22.160Z caller=main.go:948 level=info msg="TSDB started"
ts=2023-10-04T20:38:22.160Z caller=main.go:1067 level=info msg="See you next time!"

Any thoughts?

Thanks,

Anoop

[Ben Kochie]

ts=2023-10-04T20:33:32.135Z caller=main.go:776 level=warn msg="Received SIGTERM, exiting gracefully..."

The time between your first log line and this line is about 15 minutes. Kubernetes is timing out the startup and terminating your process.

Your startup is _very_ slow. Without any other information it's too hard to say why it's slow. Do you have CPU limits? What kind of storage? Too many variables.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fee699aa-d41d-4cd1-b70b-9adc8dc0db20n%40googlegroups.com.

[Anoop Mohan]

Hi Ben,

I have not configured the CPU/Memory limit values. Only the CPU/Memory requests values are configured.

For storage, we are using portworx. After deleting the wal directory, the pod suddenly came up and started running. No restart for the last 21 hours.

However, I can expect the restart after 1 or 2 days. That is what's been happening for more than 2 weeks.

Thanks & Regards,

Anoop

[Anoop]

Hey,

I am still getting the same issue. Anymore suggestions?