Re: [prometheus-users] K8s Prometheus Crash Looping

[Ben Kochie]

"Out of order sample from remote write".

You have remote write clients that are sending out of order/overlapping data.

https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus/

This is not likely to cause a crash IIRC. Have you checked to see if you have allocated enough memory to your Prometheus? What does Kubernetes say is the crash reason?

In all probability, this is simply an OOM issue. Give your instance more memory.

On Tue, May 23, 2023 at 10:55 AM Nick Carlton <nick.ca...@gmail.com> wrote:

Hello,

I have a Prometheus deployment in Azure using the kube-prometheus-stack helm chart which receives lots of remote writes. This runs fine normally but every couple of months or so, will randomly crash and start this crash loop cycle. It does usually self-resolve but takes Prom out for a couple of days.

We use a PVC with Prometheus with a size of 200GB and with the access mode "ReadWriteOnce".

Basically what I am seeing happen is, Prometheus Starts, complains about some storage chunks and does a WAL replay. Then it does start, however, we get thousands of messages about remote write "out of bounds" and then it crashes and loops again. Its will continue like this even if the pods are deleted and re-created. 

See Logs:

ts=2023-05-23T08:12:03.968Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.42.0, branch=HEAD, revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" ts=2023-05-23T08:12:03.968Z caller=main.go:561 level=info build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, date=20230201-07:53:32)" ts=2023-05-23T08:12:03.968Z caller=main.go:562 level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu May 19 18:52:16 UTC 2022 x86_64 prometheus-prometheus-kube-prometheus-prometheus-0 (none))" ts=2023-05-23T08:12:03.968Z caller=main.go:563 level=info fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:12:03.968Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" ts=2023-05-23T08:12:03.970Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090 ts=2023-05-23T08:12:03.970Z caller=main.go:993 level=info msg="Starting TSDB ..." ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1683892800000 maxt=1683957600000 ulid=01H0A5J8MWQ152H85DYY6F32PK ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1683957600000 maxt=1684022400000 ulid=01H0C38Y02M29M82TXK2V7NN7G ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684022400000 maxt=1684087200000 ulid=01H0E12DTC6PWH4J6R7B1H36XW ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684087200003 maxt=1684152000000 ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684152000003 maxt=1684216800000 ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684216800000 maxt=1684281600000 ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684281600003 maxt=1684346400000 ulid=01H0NZ78AACK9BPD034VWH7SJS ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684346400000 maxt=1684411200000 ulid=01H0QP2AA05C33ZVAAVDBT2E43 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684411200004 maxt=1684476000000 ulid=01H0STTM76GGEKKYAMWV1KNTPT ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684476000004 maxt=1684540800000 ulid=01H0VRMBDAC5D5M6TEVHRP80PS ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684540800043 maxt=1684605600000 ulid=01H0XPDYZ3D2X8XJ97B97M4BDH ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684605600043 maxt=1684670400000 ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684735200044 maxt=1684742400000 ulid=01H11AZP33QHEGWR2631MYMY3M ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684742400044 maxt=1684749600000 ulid=01H11HVD455FTTADM149WAHAH0 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684670400032 maxt=1684735200000 ulid=01H11J11TMN4XWACE1Q9AJ1KXC ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684749600043 maxt=1684756800000 ulid=01H11RQ4AHY8PVQRCM58CKTVZF ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684756800043 maxt=1684764000000 ulid=01H11ZJVV4E276NWR3370HFN00 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684764000000 maxt=1684771200000 ulid=01H12712JWFQMYW8FR8YXGQN8Y ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684771200000 maxt=1684778400000 ulid=01H12DAKVHHHGEY3TKY0FPQSVZ ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684778400013 maxt=1684785600000 ulid=01H12M7CW13EEQRVRN6J4ZD7AK ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684785600070 maxt=1684792800000 ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684792800033 maxt=1684800000000 ulid=01H131XPKE4VB6RMRQF3S67M0Y ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684800000010 maxt=1684807200000 ulid=01H138S7B8Y8Q14YKJZ5SRXTAX ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684807200000 maxt=1684814400000 ulid=01H13FXP6EB7T1KSRR8P5GMFP3 ts=2023-05-23T08:12:03.971Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684814400043 maxt=1684821600000 ulid=01H13PQXNNJNH4GETATBWHPGYJ ts=2023-05-23T08:12:03.971Z caller=tls_config.go:271 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090 ts=2023-05-23T08:12:04.257Z caller=head.go:564 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" ts=2023-05-23T08:12:04.318Z caller=head.go:592 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, last chunk: [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" ts=2023-05-23T08:12:04.318Z caller=head.go:831 level=info component=tsdb msg="Deleting mmapped chunk files" ts=2023-05-23T08:12:04.319Z caller=head.go:837 level=info component=tsdb msg="Deleting mmapped chunk files" ts=2023-05-23T08:12:04.319Z caller=head.go:840 level=info component=tsdb msg="Deletion of corrupted mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, last chunk: [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" ts=2023-05-23T08:12:04.325Z caller=head.go:608 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=67.431625ms ts=2023-05-23T08:12:04.325Z caller=head.go:614 level=info component=tsdb msg="Replaying WAL, this may take a while" ts=2023-05-23T08:12:14.136Z caller=head.go:650 level=info component=tsdb msg="WAL checkpoint loaded" ts=2023-05-23T08:12:14.145Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17588 maxSegment=17630 ts=2023-05-23T08:12:14.197Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17589 maxSegment=17630 ts=2023-05-23T08:12:14.216Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17590 maxSegment=17630 ts=2023-05-23T08:12:14.250Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17591 maxSegment=17630 ts=2023-05-23T08:12:14.293Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17592 maxSegment=17630 ts=2023-05-23T08:12:15.026Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17593 maxSegment=17630 ts=2023-05-23T08:12:15.063Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17594 maxSegment=17630 ts=2023-05-23T08:12:15.188Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17595 maxSegment=17630 ts=2023-05-23T08:12:15.287Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17596 maxSegment=17630 ts=2023-05-23T08:12:15.378Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17597 maxSegment=17630 ts=2023-05-23T08:12:15.457Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17598 maxSegment=17630 ts=2023-05-23T08:12:15.543Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17599 maxSegment=17630 ts=2023-05-23T08:12:15.619Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17600 maxSegment=17630 ts=2023-05-23T08:12:15.745Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17601 maxSegment=17630 ts=2023-05-23T08:12:15.890Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17602 maxSegment=17630 ts=2023-05-23T08:12:15.977Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17603 maxSegment=17630 ts=2023-05-23T08:12:16.057Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17604 maxSegment=17630 ts=2023-05-23T08:12:16.156Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17605 maxSegment=17630 ts=2023-05-23T08:12:16.157Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17606 maxSegment=17630 ts=2023-05-23T08:12:17.091Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17607 maxSegment=17630 ts=2023-05-23T08:12:18.271Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17608 maxSegment=17630 ts=2023-05-23T08:12:18.449Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17609 maxSegment=17630 ts=2023-05-23T08:12:18.787Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17610 maxSegment=17630 ts=2023-05-23T08:12:19.671Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17611 maxSegment=17630 ts=2023-05-23T08:12:19.894Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17612 maxSegment=17630 ts=2023-05-23T08:12:20.178Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17613 maxSegment=17630 ts=2023-05-23T08:12:20.344Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17614 maxSegment=17630 ts=2023-05-23T08:12:22.299Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17616 maxSegment=17630 ts=2023-05-23T08:12:23.180Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17617 maxSegment=17630 ts=2023-05-23T08:12:23.343Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17618 maxSegment=17630 ts=2023-05-23T08:12:23.599Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17619 maxSegment=17630 ts=2023-05-23T08:12:23.764Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17620 maxSegment=17630 ts=2023-05-23T08:12:24.128Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17621 maxSegment=17630 ts=2023-05-23T08:12:24.317Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17622 maxSegment=17630 ts=2023-05-23T08:12:24.463Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17623 maxSegment=17630 ts=2023-05-23T08:12:24.809Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17624 maxSegment=17630 ts=2023-05-23T08:12:25.088Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17625 maxSegment=17630 ts=2023-05-23T08:12:25.360Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17626 maxSegment=17630 ts=2023-05-23T08:12:25.642Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17627 maxSegment=17630 ts=2023-05-23T08:12:25.664Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17628 maxSegment=17630 ts=2023-05-23T08:12:25.884Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17629 maxSegment=17630 ts=2023-05-23T08:12:25.884Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17630 maxSegment=17630 ts=2023-05-23T08:12:25.884Z caller=head.go:722 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=9.811449959s wal_replay_duration=11.747824132s wbl_replay_duration=215ns total_replay_duration=21.626738156s ts=2023-05-23T08:12:28.177Z caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC ts=2023-05-23T08:12:28.177Z caller=main.go:1017 level=info msg="TSDB started" ts=2023-05-23T08:12:28.177Z caller=main.go:1197 level=info msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml ts=2023-05-23T08:12:28.181Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-operator/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info component="discovery manager notify" discovery=kubernetes config=config-0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:12:28.221Z caller=main.go:1234 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=43.976379ms db_storage=1.549µs remote_storage=1.501µs web_handler=594ns query_engine=1.035µs scrape=207.016µs scrape_sd=565.387µs notify=13.403µs notify_sd=116.569µs rules=39.2593ms tracing=5.19µs ts=2023-05-23T08:12:28.221Z caller=main.go:978 level=info msg="Server is ready to receive web requests." ts=2023-05-23T08:12:28.221Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." ts=2023-05-23T08:13:17.453Z caller=write_handler.go:109 level=error component=web msg="Out of order sample from remote write" err="out of bounds" series ... timestamp=1684825890472 ... ts=2023-05-23T08:13:17.547Z caller=write_handler.go:109 level=error component=web msg="Out of order sample from remote write" err="out of bounds" series ... timestamp=1684808929971 ts=2023-05-23T08:13:29.090Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.42.0, branch=HEAD, revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" ts=2023-05-23T08:13:29.090Z caller=main.go:561 level=info build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, date=20230201-07:53:32)" ts=2023-05-23T08:13:29.090Z caller=main.go:562 level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu May 19 18:52:16 UTC 2022 x86_64 prometheus-prometheus-kube-prometheus-prometheus-0 (none))" ts=2023-05-23T08:13:29.090Z caller=main.go:563 level=info fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:13:29.090Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" ts=2023-05-23T08:13:29.090Z caller=query_logger.go:80 level=info component=activeQueryTracker msg="These queries didn't finish in prometheus' last run:" queries=[829607] ts=2023-05-23T08:13:29.091Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090 ts=2023-05-23T08:13:29.092Z caller=main.go:993 level=info msg="Starting TSDB ..." ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1683892800000 maxt=1683957600000 ulid=01H0A5J8MWQ152H85DYY6F32PK ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1683957600000 maxt=1684022400000 ulid=01H0C38Y02M29M82TXK2V7NN7G ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684022400000 maxt=1684087200000 ulid=01H0E12DTC6PWH4J6R7B1H36XW ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684087200003 maxt=1684152000000 ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684152000003 maxt=1684216800000 ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684216800000 maxt=1684281600000 ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684281600003 maxt=1684346400000 ulid=01H0NZ78AACK9BPD034VWH7SJS ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684346400000 maxt=1684411200000 ulid=01H0QP2AA05C33ZVAAVDBT2E43 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684411200004 maxt=1684476000000 ulid=01H0STTM76GGEKKYAMWV1KNTPT ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684476000004 maxt=1684540800000 ulid=01H0VRMBDAC5D5M6TEVHRP80PS ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684540800043 maxt=1684605600000 ulid=01H0XPDYZ3D2X8XJ97B97M4BDH ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684605600043 maxt=1684670400000 ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 ts=2023-05-23T08:13:29.093Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684735200044 maxt=1684742400000 ulid=01H11AZP33QHEGWR2631MYMY3M ts=2023-05-23T08:13:29.093Z caller=tls_config.go:271 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684742400044 maxt=1684749600000 ulid=01H11HVD455FTTADM149WAHAH0 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684670400032 maxt=1684735200000 ulid=01H11J11TMN4XWACE1Q9AJ1KXC ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684749600043 maxt=1684756800000 ulid=01H11RQ4AHY8PVQRCM58CKTVZF ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684756800043 maxt=1684764000000 ulid=01H11ZJVV4E276NWR3370HFN00 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684764000000 maxt=1684771200000 ulid=01H12712JWFQMYW8FR8YXGQN8Y ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684771200000 maxt=1684778400000 ulid=01H12DAKVHHHGEY3TKY0FPQSVZ ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684778400013 maxt=1684785600000 ulid=01H12M7CW13EEQRVRN6J4ZD7AK ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684785600070 maxt=1684792800000 ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684792800033 maxt=1684800000000 ulid=01H131XPKE4VB6RMRQF3S67M0Y ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684800000010 maxt=1684807200000 ulid=01H138S7B8Y8Q14YKJZ5SRXTAX ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684807200000 maxt=1684814400000 ulid=01H13FXP6EB7T1KSRR8P5GMFP3 ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684814400043 maxt=1684821600000 ulid=01H13PQXNNJNH4GETATBWHPGYJ ts=2023-05-23T08:13:29.365Z caller=head.go:564 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" ts=2023-05-23T08:13:29.426Z caller=head.go:608 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=60.526867ms ts=2023-05-23T08:13:29.426Z caller=head.go:614 level=info component=tsdb msg="Replaying WAL, this may take a while" ts=2023-05-23T08:13:38.994Z caller=head.go:650 level=info component=tsdb msg="WAL checkpoint loaded" ts=2023-05-23T08:13:39.003Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17588 maxSegment=17631 ts=2023-05-23T08:13:39.038Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17589 maxSegment=17631 ts=2023-05-23T08:13:39.055Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17590 maxSegment=17631 ts=2023-05-23T08:13:39.081Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17591 maxSegment=17631 ts=2023-05-23T08:13:39.120Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17592 maxSegment=17631 ts=2023-05-23T08:13:40.778Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17593 maxSegment=17631 ts=2023-05-23T08:13:40.811Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17594 maxSegment=17631 ts=2023-05-23T08:13:40.928Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17595 maxSegment=17631 ts=2023-05-23T08:13:41.015Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17596 maxSegment=17631 ts=2023-05-23T08:13:41.091Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17597 maxSegment=17631 ts=2023-05-23T08:13:41.158Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17598 maxSegment=17631 ts=2023-05-23T08:13:41.234Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17599 maxSegment=17631 ts=2023-05-23T08:13:41.300Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17600 maxSegment=17631 ts=2023-05-23T08:13:41.406Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17601 maxSegment=17631 ts=2023-05-23T08:13:41.543Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17602 maxSegment=17631 ts=2023-05-23T08:13:41.619Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17603 maxSegment=17631 ts=2023-05-23T08:13:41.689Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17604 maxSegment=17631 ts=2023-05-23T08:13:41.770Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17605 maxSegment=17631 ts=2023-05-23T08:13:41.771Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17606 maxSegment=17631 ts=2023-05-23T08:13:42.359Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17607 maxSegment=17631 ts=2023-05-23T08:13:42.843Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17608 maxSegment=17631 ts=2023-05-23T08:13:43.023Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17609 maxSegment=17631 ts=2023-05-23T08:13:43.362Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17610 maxSegment=17631 ts=2023-05-23T08:13:44.268Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17611 maxSegment=17631 ts=2023-05-23T08:13:44.500Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17612 maxSegment=17631 ts=2023-05-23T08:13:44.792Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17613 maxSegment=17631 ts=2023-05-23T08:13:44.958Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17614 maxSegment=17631 ts=2023-05-23T08:13:45.175Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17615 maxSegment=17631 ts=2023-05-23T08:13:46.881Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17616 maxSegment=17631 ts=2023-05-23T08:13:48.160Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17617 maxSegment=17631 ts=2023-05-23T08:13:48.529Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17618 maxSegment=17631 ts=2023-05-23T08:13:49.071Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17619 maxSegment=17631 ts=2023-05-23T08:13:49.463Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17620 maxSegment=17631 ts=2023-05-23T08:13:49.864Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17621 maxSegment=17631 ts=2023-05-23T08:13:50.052Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17622 maxSegment=17631 ts=2023-05-23T08:13:50.181Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17623 maxSegment=17631 ts=2023-05-23T08:13:51.235Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17627 maxSegment=17631 ts=2023-05-23T08:13:51.258Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17628 maxSegment=17631 ts=2023-05-23T08:13:51.476Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17629 maxSegment=17631 ts=2023-05-23T08:13:51.707Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17630 maxSegment=17631 ts=2023-05-23T08:13:51.707Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17631 maxSegment=17631 ts=2023-05-23T08:13:51.707Z caller=head.go:722 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=9.568658965s wal_replay_duration=12.712461039s wbl_replay_duration=240ns total_replay_duration=22.341708746s ts=2023-05-23T08:13:54.095Z caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC ts=2023-05-23T08:13:54.095Z caller=main.go:1017 level=info msg="TSDB started" ts=2023-05-23T08:13:54.095Z caller=main.go:1197 level=info msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml ts=2023-05-23T08:13:54.098Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/2 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-prometheus/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info component="discovery manager notify" discovery=kubernetes config=config-0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:13:54.136Z caller=main.go:1234 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=41.309545ms db_storage=946ns remote_storage=1.338µs web_handler=633ns query_engine=1.422µs scrape=229.86µs scrape_sd=593.335µs notify=15.321µs notify_sd=186.832µs rules=36.799825ms tracing=4.126µs ts=2023-05-23T08:13:54.136Z caller=main.go:978 level=info msg="Server is ready to receive web requests." ts=2023-05-23T08:13:54.136Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." ts=2023-05-23T08:13:58.936Z caller=write_handler.go:109 level=error component=web msg="Out of order sample from remote write" err="out of bounds" series ... timestamp=1684808929971 ... ts=2023-05-23T08:14:16.023Z caller=write_handler.go:109 level=error component=web msg="Out of order sample from remote write" err="out of bounds" series ... timestamp=1684825987289 ts=2023-05-23T08:14:34.749Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.42.0, branch=HEAD, revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" ts=2023-05-23T08:14:34.749Z caller=main.go:561 level=info build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, date=20230201-07:53:32)" ts=2023-05-23T08:14:34.749Z caller=main.go:562 level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu May 19 18:52:16 UTC 2022 x86_64 prometheus-prometheus-kube-prometheus-prometheus-0 (none))" ts=2023-05-23T08:14:34.749Z caller=main.go:563 level=info fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:14:34.749Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" ts=2023-05-23T08:14:34.750Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090 ts=2023-05-23T08:14:34.752Z caller=main.go:993 level=info msg="Starting TSDB ..." ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1683892800000 maxt=1683957600000 ulid=01H0A5J8MWQ152H85DYY6F32PK ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1683957600000 maxt=1684022400000 ulid=01H0C38Y02M29M82TXK2V7NN7G ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684022400000 maxt=1684087200000 ulid=01H0E12DTC6PWH4J6R7B1H36XW ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684087200003 maxt=1684152000000 ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684152000003 maxt=1684216800000 ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684216800000 maxt=1684281600000 ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684281600003 maxt=1684346400000 ulid=01H0NZ78AACK9BPD034VWH7SJS ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684346400000 maxt=1684411200000 ulid=01H0QP2AA05C33ZVAAVDBT2E43 ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684411200004 maxt=1684476000000 ulid=01H0STTM76GGEKKYAMWV1KNTPT ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684476000004 maxt=1684540800000 ulid=01H0VRMBDAC5D5M6TEVHRP80PS ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684540800043 maxt=1684605600000 ulid=01H0XPDYZ3D2X8XJ97B97M4BDH ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684605600043 maxt=1684670400000 ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684735200044 maxt=1684742400000 ulid=01H11AZP33QHEGWR2631MYMY3M ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684742400044 maxt=1684749600000 ulid=01H11HVD455FTTADM149WAHAH0 ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684670400032 maxt=1684735200000 ulid=01H11J11TMN4XWACE1Q9AJ1KXC ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684749600043 maxt=1684756800000 ulid=01H11RQ4AHY8PVQRCM58CKTVZF ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684756800043 maxt=1684764000000 ulid=01H11ZJVV4E276NWR3370HFN00 ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684764000000 maxt=1684771200000 ulid=01H12712JWFQMYW8FR8YXGQN8Y ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684771200000 maxt=1684778400000 ulid=01H12DAKVHHHGEY3TKY0FPQSVZ ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684778400013 maxt=1684785600000 ulid=01H12M7CW13EEQRVRN6J4ZD7AK ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684785600070 maxt=1684792800000 ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684792800033 maxt=1684800000000 ulid=01H131XPKE4VB6RMRQF3S67M0Y ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684800000010 maxt=1684807200000 ulid=01H138S7B8Y8Q14YKJZ5SRXTAX ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684807200000 maxt=1684814400000 ulid=01H13FXP6EB7T1KSRR8P5GMFP3 ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1684814400043 maxt=1684821600000 ulid=01H13PQXNNJNH4GETATBWHPGYJ ts=2023-05-23T08:14:34.753Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090 ts=2023-05-23T08:14:34.753Z caller=tls_config.go:271 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090 ts=2023-05-23T08:14:35.063Z caller=head.go:564 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" ts=2023-05-23T08:14:35.125Z caller=head.go:592 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, last chunk: [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" ts=2023-05-23T08:14:35.125Z caller=head.go:831 level=info component=tsdb msg="Deleting mmapped chunk files" ts=2023-05-23T08:14:35.129Z caller=head.go:837 level=info component=tsdb msg="Deleting mmapped chunk files" ts=2023-05-23T08:14:35.129Z caller=head.go:840 level=info component=tsdb msg="Deletion of corrupted mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, last chunk: [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" ts=2023-05-23T08:14:35.135Z caller=head.go:608 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=71.923626ms ts=2023-05-23T08:14:35.135Z caller=head.go:614 level=info component=tsdb msg="Replaying WAL, this may take a while" ts=2023-05-23T08:14:44.651Z caller=head.go:650 level=info component=tsdb msg="WAL checkpoint loaded" ts=2023-05-23T08:14:44.659Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17588 maxSegment=17632 ts=2023-05-23T08:14:44.710Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17589 maxSegment=17632 ts=2023-05-23T08:14:44.729Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17590 maxSegment=17632 ts=2023-05-23T08:14:44.763Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17591 maxSegment=17632 ts=2023-05-23T08:14:44.894Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17592 maxSegment=17632 ts=2023-05-23T08:14:46.470Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17593 maxSegment=17632 ts=2023-05-23T08:14:46.503Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17594 maxSegment=17632 ts=2023-05-23T08:14:46.616Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17595 maxSegment=17632 ts=2023-05-23T08:14:46.710Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17596 maxSegment=17632 ts=2023-05-23T08:14:46.796Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17597 maxSegment=17632 ts=2023-05-23T08:14:46.876Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17598 maxSegment=17632 ts=2023-05-23T08:14:46.955Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17599 maxSegment=17632 ts=2023-05-23T08:14:47.025Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17600 maxSegment=17632 ts=2023-05-23T08:14:47.135Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17601 maxSegment=17632 ts=2023-05-23T08:14:47.252Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17602 maxSegment=17632 ts=2023-05-23T08:14:47.336Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17603 maxSegment=17632 ts=2023-05-23T08:14:47.406Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17604 maxSegment=17632 ts=2023-05-23T08:14:47.496Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17605 maxSegment=17632 ts=2023-05-23T08:14:47.499Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17606 maxSegment=17632 ts=2023-05-23T08:14:48.079Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17607 maxSegment=17632 ts=2023-05-23T08:14:48.573Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17608 maxSegment=17632 ts=2023-05-23T08:14:48.745Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17609 maxSegment=17632 ts=2023-05-23T08:14:49.082Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17610 maxSegment=17632 ts=2023-05-23T08:14:49.967Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17611 maxSegment=17632 ts=2023-05-23T08:14:50.190Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17612 maxSegment=17632 ts=2023-05-23T08:14:52.558Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17616 maxSegment=17632 ts=2023-05-23T08:14:54.144Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17617 maxSegment=17632 ts=2023-05-23T08:14:54.490Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17618 maxSegment=17632 ts=2023-05-23T08:14:55.119Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17619 maxSegment=17632 ts=2023-05-23T08:14:55.293Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17620 maxSegment=17632 ts=2023-05-23T08:14:55.633Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17621 maxSegment=17632 ts=2023-05-23T08:14:55.826Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17622 maxSegment=17632 ts=2023-05-23T08:14:55.976Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17623 maxSegment=17632 ts=2023-05-23T08:14:56.335Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17624 maxSegment=17632 ts=2023-05-23T08:14:56.615Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17625 maxSegment=17632 ts=2023-05-23T08:14:56.882Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17626 maxSegment=17632 ts=2023-05-23T08:14:57.171Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17627 maxSegment=17632 ts=2023-05-23T08:14:57.193Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17628 maxSegment=17632 ts=2023-05-23T08:14:57.402Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17629 maxSegment=17632 ts=2023-05-23T08:14:57.616Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17630 maxSegment=17632 ts=2023-05-23T08:14:57.718Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17631 maxSegment=17632 ts=2023-05-23T08:14:57.718Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=17632 maxSegment=17632 ts=2023-05-23T08:14:57.718Z caller=head.go:722 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=9.515250193s wal_replay_duration=13.067369985s wbl_replay_duration=205ns total_replay_duration=22.654583208s ts=2023-05-23T08:15:00.133Z caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC ts=2023-05-23T08:15:00.133Z caller=main.go:1017 level=info msg="TSDB started" ts=2023-05-23T08:15:00.134Z caller=main.go:1197 level=info msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-alertmanager/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus-kube-prometheus-kube-scheduler/0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info component="discovery manager notify" discovery=kubernetes config=config-0 msg="Using pod service account via in-cluster config" ts=2023-05-23T08:15:00.176Z caller=main.go:1234 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=42.962628ms db_storage=1.094µs remote_storage=1.525µs web_handler=583ns query_engine=1.243µs scrape=197.887µs scrape_sd=650.383µs notify=16.671µs notify_sd=181.635µs rules=38.073939ms tracing=4.418µs ts=2023-05-23T08:15:00.177Z caller=main.go:978 level=info msg="Server is ready to receive web requests." ts=2023-05-23T08:15:00.177Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..."

Sometimes Prom will just start up again without warning and other times I see this after all the out of bounds errors:

ts=2023-05-23T08:18:58.284Z caller=main.go:828 level=warn msg="Received SIGTERM, exiting gracefully..." ts=2023-05-23T08:18:58.284Z caller=main.go:852 level=info msg="Stopping scrape discovery manager..." ts=2023-05-23T08:18:58.284Z caller=main.go:866 level=info msg="Stopping notify discovery manager..." ts=2023-05-23T08:18:58.284Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." ts=2023-05-23T08:18:58.284Z caller=main.go:848 level=info msg="Scrape discovery manager stopped" ts=2023-05-23T08:18:58.284Z caller=main.go:862 level=info msg="Notify discovery manager stopped"

Is anyone able to assist with what is going wrong here?

Thanks

Nick

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ab9d7a7f-b4d8-49ee-b702-fda49a590102n%40googlegroups.com.