Prometheus operator pod is not running

15 views

Skip to first unread message

Devops Engineers

unread,

Jun 29, 2020, 12:03:17 AM6/29/20

to Prometheus Users

Hello Everyone

Actually my prometheus operator readiness failed because of this i am unable to open my UI .I have checked the logs but i couldn't able to find any error messages related to prometheus

Could you please check the below logs and do the needful .

LOGS :
X567H40N2WD
level=info ts=2020-06-28T18:03:33.702Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1591905600000 maxt=1591912800000 ulid=01EAJT7KCKCSRT16H7CEWC6X3B
level=info ts=2020-06-28T18:03:33.702Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1591912800000 maxt=1591920000000 ulid=01EAK261BQVKXF14BQMBHKFFH9
level=info ts=2020-06-28T18:03:33.702Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1591920000000 maxt=1591927200000 ulid=01EAKA1RYKVM6WM4K0P1CRZDHJ
level=info ts=2020-06-28T18:03:33.703Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1591855200000 maxt=1591876800000 ulid=01EAKB8V8EPRGJMZQAB8K7FEJV
level=info ts=2020-06-28T18:03:33.703Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1591927200000 maxt=1591934400000 ulid=01EAKGZNYZN22MPA84RWWYBEYP
level=info ts=2020-06-28T18:03:54.940Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2020-06-28T18:04:11.343Z caller=head.go:608 component=tsdb msg="WAL checkpoint loaded"
level=info ts=2020-06-28T18:04:11.344Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=32993 maxSegment=34175
level=info ts=2020-06-28T18:04:11.344Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=32994 maxSegment=34175
level=info ts=2020-06-28T18:04:11.344Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=32995 maxSegment=34175

kubdetl events :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m default-scheduler Successfully assigned monitoring/prometheus-prometheus-operator-prometheus-0 to ip-172-16-9-79.ec2.internal
Normal SuccessfulAttachVolume 17m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c841d132-a29f-11e9-8e27-021d0af0a494"
Normal Pulling 17m kubelet, ip-172-11.2.2.ec2.internal pulling image "quay.io/prometheus/prometheus:v2.15.2"
Normal Pulling 17m kubelet, ip- 172-11.2.2 .ec2.internal pulling image "quay.io/coreos/configmap-reload:v0.0.1"
Normal Created 17m kubelet, ip- 172-11.2.2 .ec2.internal Created container
Normal Started 17m kubelet, ip- 172-11.2.2 .ec2.internal Started container
Normal Pulling 17m kubelet, ip- 172-11.2.2 .ec2.internal pulling image "quay.io/coreos/prometheus-config-reloader:v0.37.0"
Normal Pulled 17m kubelet, ip- 172-11.2.2 .ec2.internal Successfully pulled image "quay.io/coreos/prometheus-config-reloader:v0.37.0"
Normal Created 17m kubelet, ip- 172-11.2.2 .ec2.internal Created container
Normal Pulled 17m kubelet, ip- 172-11.2.2 .ec2.internal Successfully pulled image "quay.io/prometheus/prometheus:v2.15.2"
Normal Started 17m kubelet, ip- 172-11.2.2 ec2.internal Started container
Normal Pulled 17m kubelet, ip- 172-11.2.2 .ec2.internal Successfully pulled image "quay.io/coreos/configmap-reload:v0.0.1"
Normal Created 17m kubelet, ip 172-11.2.2 .ec2.internal Created container
Normal Started 17m kubelet, ip- 172-11.2.2 .ec2.internal Started container
Normal Pulling 17m kubelet, ip- 172-11.2.2 ec2.internal pulling image "thanosio/thanos:v0.7.0"
Normal Pulled 17m kubelet, ip- 172-11.2.2 .ec2.internal Successfully pulled image "thanosio/thanos:v0.7.0"
Normal Created 17m kubelet, ip- 172-11.2.2 .ec2.internal Created container
Normal Started 17m kubelet, ip- 172-11.2.2 ec2.internal Started container
Warning Unhealthy 2m12s (x179 over 17m) kubelet, ip- 172-11.2.2 .ec2.internal Readiness probe failed: HTTP probe failed with statuscode: 503