Readiness probe failed: HTTP probe failed with statuscode: 503

[anis ben hassine]

Hello Guys,

1. I'm getting for the first time a Readiness Probe issue with ly Prom running on AWS EKS, the pod was running nice since weeks ant it suddenly turns into failing.

$ kdpon monitoring prometheus (Kube pod describe command )

Unhealthy  4m (x182535 over 6d)  kubelet, ip-100-xxxxxxxx.ec2.internal  Readiness probe failed: HTTP probe failed with statuscode: 503

 2. However, seems my logs are doing well, and restarting pods doesn't fix the issue but this is a normal. behaviour for readiness Prob

    

$ klon monitoring prometheus (Kube log command )

level=info ts=2020-05-25T09:38:56.653582783Z caller=main.go:243 msg="Starting Prometheus" version="(version=2.6.1, branch=HEAD, revision=b639fe140c1f71b2cbad3fc322b17efe60839e7e)"

level=info ts=2020-05-25T09:38:56.653650678Z caller=main.go:244 build_context="(go=go1.11.4, user=root@4c0e286fe2b3, date=20190115-19:12:04)"

level=info ts=2020-05-25T09:38:56.653691321Z caller=main.go:245 host_details="(Linux 4.14.154-128.181.amzn2.x86_64 #1 SMP Sat Nov 16 21:49:00 UTC 2019 x86_64 prometheus-0 (none))"

level=info ts=2020-05-25T09:38:56.653727427Z caller=main.go:246 fd_limits="(soft=65536, hard=65536)"

level=info ts=2020-05-25T09:38:56.653762149Z caller=main.go:247 vm_limits="(soft=unlimited, hard=unlimited)"

level=info ts=2020-05-25T09:38:56.65457275Z caller=main.go:561 msg="Starting TSDB ..."

level=info ts=2020-05-25T09:38:56.655755388Z caller=web.go:429 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2020-05-25T09:38:56.817276722Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1583940235972 maxt=1583971200000 ulid=01E36HT83DK3TG22QWE02AP50X

level=info ts=2020-05-25T09:38:56.822556254Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1583971200000 maxt=1584165600000 ulid=01E3C4BCPDEWX998E1EMR8RWTB

level=info ts=2020-05-25T09:38:56.826661565Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1584165600000 maxt=1584360000000 ulid=01E3HXRADQ09H3PD3KHPS5GAXK

level=info ts=2020-05-25T09:38:56.831874475Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1584360000000 maxt=1584554400000 ulid=01E3QQ4PWYR2D0TF0ERWR7KTZA

level=info ts=2020-05-25T09:38:56.837772421Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1584554400000 maxt=1584748800000 ulid=01E3XGHHPWR4QHYZBZGGSR41FS

level=info ts=2020-05-25T09:38:56.842330027Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1584748800000 maxt=1584943200000 ulid=01E439YBW2HYS58CK67540JRDP

level=info ts=2020-05-25T09:38:56.847023184Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1584943200000 maxt=1585137600000 ulid=01E493B0FCHAT74EZFVX63AA8J

level=info ts=2020-05-25T09:38:56.852126117Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1585137600000 maxt=1585332000000 ulid=01E4EWV0G5ZV7G0M5TH3XEV3X5

level=info ts=2020-05-25T09:38:56.856215058Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1585332000000 maxt=1585526400000 ulid=01E4MP49E017G3SF46C4MXND0A

level=info ts=2020-05-25T09:38:56.860292298Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1585526400000 maxt=1585720800000 ulid=01E4TFMEZ1YMEM8PJAQJA9WZA7

level=info ts=2020-05-25T09:38:56.865162002Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1585720800000 maxt=1585915200000 ulid=01E508Y5F1V3RBPRMPVYMQCFQM

level=info ts=2020-05-25T09:38:56.869511088Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1585915200000 maxt=1586109600000 ulid=01E562AXM9EHH3BV11MQYHGX80

I'm trying to figure if the issue is rrmated to Code or infrastructure layer

Thanks you in advance for you Help

Best,

ABH