write: broken pipe"

[weidong zou]

version : prometheus:v2.28.1

Suddenly this problem occurred in my prometheus,Causes the memory to burst.

Something like this:

• level=error ts=2021-10-23T08:31:39.022Z caller=api.go:1491 component=web msg="error writing response" bytesWritten=0 err="write tcp 10.194.15.xx:9090->10.194.25.xx:57378: write: broken pipe"
• level=error ts=2021-10-23T08:31:41.561Z caller=api.go:1491 component=web msg="error writing response" bytesWritten=0 err="write tcp 10.194.15.xx:9090->10.194.24.xx:23920: write: broken pipe"
• level=error ts=2021-10-23T08:31:42.893Z caller=api.go:1491 component=web msg="error writing response" bytesWritten=0 err="write tcp 10.194.15.xx:9090->10.194.25.xx:10736: write: broken pipe"

What is the problem？

[Ben Kochie]

Without any more info, it looks like someone / something is abusing your Prometheus server API endpoint. Or something has changed such that your API endpoint is very slow.

I would look at the "prometheus_http_request_duration_seconds" histogram to find out if there is a specific API endpoint that is being abused.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ddbefbe2-d13b-410e-b90b-12ae837abbd6n%40googlegroups.com.

[Ben Kochie]

Thanos uses the remote read API, those look like direct query_range requests, aka graph queries.

On Sat, Oct 23, 2021 at 11:41 AM weidong zou <54939...@gmail.com> wrote:

thanks ,really like this.

I added the thanos sidecar in prometheus, I might have to check the reason for the thanos query.