Connection Reset By Peer - Redis 7.0.6

[Ankit Gupta]

Hello Experts,

We are executing certain test on redis 7.0.6 with 1000 clients and 500 pipeline, and observing periodic connection resets in redis.log

Please see the logs:
11:M 23 Jan 2023 06:20:24.145 - Reading from client: Connection reset by peer
11:M 23 Jan 2023 06:20:24.145 - Client closed connection id=10268 addr=?:0 laddr=172.17.0.3:6381 fd=139 name= age=16 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=73 omem=1496792 tot-mem=1520088 events=rw cmd=get user=default redir=-1 resp=2
11:M 23 Jan 2023 06:20:24.145 - Reading from client: Connection reset by peer

We have a few queries around this

• When would redis reset the connection
• What can be expected when the connection is reset
• Why would redis reset an existing connection
• Can we minmize these resets,
• What will be the impact on consumer application if connections are reset by redis

Looking forward for your expert opinion.

[Greg Andrews]

"Connection reset by peer" means exactly what the second log file line says:  Your Redis server didn't close the TCP connection, the client closed it.  The answer to your question on how to minimize the disconnections depends on what's causing them.  Why would a client that connected to your Redis server decide to disconnect?

The first step toward understanding why the client disconnected is:  What kind of client is this?

• Is it a monitoring system that connects to test that your Redis server is available?
• Is it another Redis server process that's replicating data with this Redis server (in a primary/replica [used to be called master/slave] relationship)?
• Is it a client that connects to your Redis server to read or write data?

The expected behavior for each of the above type of client is often different, and the action you'll need to take to prevent the 'reset by peer" messages will be different for each of them.

Your other questions can't be answered without knowing what type of clients are generating these messages.  In the case of other Redis servers, your monitoring system will tell you which ones are exiting/crashing (and therefore dropping their connections), and you can match those timestamps and IP addresses to the messages in your log.  In the case of your monitoring system, testing and changing the code that connects/disconnects may be required.  (it's common for monitoring systems to just disconnect instead of sending a QUIT command and waiting for the server to acknowledge)

[Greg Andrews]

A quick note about monitoring systems:  In some cases, you may have your clients pass through a load balancer to reach your Redis servers.  Load balancers often have subsystems that test whether your Redis servers is available, so connections will be forwarded to the servers that are up.  The probes that load balancers perform are often very simple ones, just connecting and disconnecting without sending a clean QUIT command.  This sort of probe can produce the kind of 'connection reset by peer' message you see in your logs.

This isn't the only reason the reset message can appear in logfiles.  I'm mentioning it because load balancer health probes are often forgotten when thinking about the list of client connections that can produce errors in Redis logfiles.

On Thursday, February 2, 2023 at 9:31:07 PM UTC-8 g.an...@gmail.com wrote:

[Ankit Gupta]

Hello dear,

Thanks for the prompt response. 

Yes, in this it's a client(another redis server) from which we are running redis-benchmark.

[Greg Andrews]

Sorry, I can't tell if you're saying you're running redis-benchmark directly against the redis server whose logfile lines you posted, or your running redis-benchmark against a different redis server that's connected (and disconnecting) to the server whose logfile lines you posted. There's also no information about side effects on the Redis servers when you're running the benchmark tests.  High cpu usage?  Growth in RAM usage?  What does your monitoring show?

[TAPAS RANJAN]

hello sir,

  I got you from the google Redis group,, I have some queries, I cleared the redis administration certificate. Can you please guide me on what type of trouble I had to face and how to resolve those real-time issues in administration work?