POST /loki/api/v1/push (500) Response: empty ring and context canceled

[Andrew Kwok]

Hello,

I have built a test EKS environment to apply Loki for a few weeks, so far I have created 10s of Loki’s pods using Monolithic mode, each Loki’s pod received logs from multiple ec2 instances.

However, I have found promtail keep returning error messages below,

error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): empty ring"
or
error sending batch, will retry" status=-1 error="Post \"loki:port/loki/api/v1/push\": context deadline exceeded"

When I check the logs from Loki, I have found similar error messages:

level=warn ts=2022-11-07T08:59:39.648738164Z caller=logging.go:86 traceID=414f3905fdec9c5b orgID=fake msg="POST /loki/api/v1/push (500) 5.863871ms Response: \"empty ring\\n\" ws: false; Content-Length: 267464; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; X-Amzn-Trace-Id: Root=id; X-Forwarded-For: IP; X-Forwarded-Port: PORT; X-Forwarded-Proto: http; "

level=warn ts=2022-11-07T09:23:23.193157476Z caller=logging.go:86 traceID=0a15f11b53b377ce orgID=fake msg="POST /loki/api/v1/push (500) 9.998004113s Response: \"context canceled\\n\" ws: false; Content-Length: 235297; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; X-Amzn-Trace-Id: Root=id; X-Forwarded-For: IP; X-Forwarded-Port: PORT; X-Forwarded-Proto: http; "

After this issue started, Grafana failed to call resources on and off, which means Loki’s pod is reachable in a short period of time, started collecting logs busily, and failed again.

I am not sure why this situation happened in some of my Loki pods only as the config are nearly the same.

Can anyone know how to solve this problem? What does “empty ring” mean?

Thank you!

Regards,

Andrew