"-ERR Connection refused" from redis server

[Yongsheng Wu]

Hi,

Occasionally we got "-ERR Connection refused" from the redis server. I wonder whether someone can provide some insight on when this error could be thrown by the server, and what would be the right thing to do in this scenario.

Thanks a bunch!

Yongsheng

[Josiah Carlson]

It can be thrown if your server is overwhelmed by command execution, network utilization, too many incoming connections, too many connections from a single machine (your client machine may be raising the error, not your server) if you're not properly reusing connections, too much memory used by your Redis server, or sometimes during snapshot or AOF rewriting ...

How often are you seeing a connection refused message? How many connections are you making to your Redis server every second/minute/hour (you can calculate this by watching the output of INFO's total_connections_received line)? Do the connection refused messages come in bursts, typically when Redis is snapshotting, rewriting the AOF, or during a slave connection/reconnection?

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Yongsheng Wu]

In our case, the server is not busy not all, both cpu usage and network usage are moderate, there are plenty of free memory, and the reconnection rate is around 10 per min. So it seems that this could have been caused by AOF rewriting, or slave connection/reconnection (snapshotting has been disabled).

Just to be certain we are talking about the same thing, we had the socket connection established, and the error message was read from the socket, so it was from redis server.

It sounds like that in this case, we could do some backoff, and retry with the same connection?

Thanks,
Yongsheng

[Yiftach Shoolman]

Do you see anything in the Redis log regarding AOF ? do you also use snapshot, if so what is the fork time ?.

In general, if your issues are around data persistence you can disable data-persistence at the master and enable it at your salve. Alternatively, you can increase your storage i.o throughput by raid.

If your issues are not around data-persistence, please look if you see something in the SLOWLOG.

Last but not least, make sure your configured client TO isn't too small. 

--

Yiftach Shoolman
+972-54-7634621

[Josiah Carlson]

"Connection refused" is not read from the server. It is an error message produced by the libraries your client is using to connect to Redis. If Redis was writing "Connection refused" to your client, that would imply that the connection had first been accepted, then written to, then closed. That would be very silly, unless you had passwords enabled and sent the wrong password.

If you are reconnecting 10 times/minute on average, that's pretty good. How many connections are you seeing refused?

Generally speaking, if you see a connection refused, you can definitely try backing off. But as Yiftach mentioned, check your logs and SLOWLOG to verify one way or another.

 - Josiah

[Yongsheng Wu]

Thanks for help, guys! I really appreciate it.

@Joshiah, you are right, this error comes from the proxy layer we have between redis and our application servers, not from redis server. Initially I thought in this particular case, we didn''t use proxy but rather directly talking to our redis servers, hence the confusion.

Yongsheng