Remote write dying after some time.

[Christian Oelsner]

Hello guys.

I am scraping some metrics wich ar ethen shipped of to an Elastic agent to be ingested into Elasticsearch. All seems fine to start with, but after some time, metrics stop comming in, and the prometheus logs shows a lot of entries like this:

ts=2023-01-18T10:51:46.125Z caller=dedupe.go:112 component=remote level=warn remote_name=010ca8 url=http://agent-svc.observability.svc.cluster.local:9201/write msg="Failed to send batch, retrying" err="Post \"http://agent-svc.observability.svc.cluster.local:9201/write\": context deadline exceeded"

ts=2023-01-18T10:51:20.364Z caller=dedupe.go:112 component=remote level=debug remote_name=010ca8 url=http://agent-svc.observability.svc.cluster.local:9201/write msg="Not downsharding due to being too far behind"

I am guessing that Prometheus is trying to tell me something, but i just dont know what.

Checking the TSDB status on the prom UI it tells me that Number of series is 8439 wich does not sound like a lot.

Any help would be very appreciated.

Best regards

Christian Oelsner

[Brian Candler]

Looks to me like a problem at the receiver end (i.e. the middleware Elastic agent, or Elasticsearch itself), i.e. that side has stopped accepting data.

Try looking at logs of these to determine why they are no longer accepting data.

[Christian Oelsner]

Hi Brian,

Thanks for your input, it as always apreaciated.

I will try to have the observability team enable some debug logging on the agent, and see if i can spot something.

Regards

Christian Oelsner

[Christian Oelsner]

Hi Brian,

Sorry about the latency.
It was indeed an issue with the receiver, which is sorted now by the team that manages the Elastic agent.

Best regards

Christian Oelsner