Sentinel process connections increasing

[nigel coutinho]

We are noticing an issue on our production systems where the connections to our sentinel process keep steadily increasing. 

The apps and sentinel process run on different servers. 

Netstat output shows that the connections are from the application servers.

From the application side we see only a set number of open connections to the sentinel hosts.

Sometimes the app servers loose connection to sentinel and end up re-opening another connection. However the socket at the sentinel end stays open.

This issue forces us to have to periodically restart the sentinel instance.

Has anyone seen this issue ? We are running 2.8.24 version of Redis/Sentinel.

[Greg Andrews]

There was a very similar thread on the mailing list a couple of months ago:   https://groups.google.com/d/topic/redis-db/G4amckfHGDw/discussion

The summary is that you most likely have a network device (like a firewall) between your clients and your Sentinels that is breaking the connections after an idle time, or after a maximum allowed time.  Since the Sentinel service always waits for a query from the client, it won't notice that a connection is broken by the device.  The connection merely seems to be idle, so Sentinel never closes the connection.  The client notices the next time it tries to send a command and gets an error from the TCP stack.

Find the device between your clients and Sentinels that's breaking the connections and find out why it's breaking the connections.  If it thinks the connections are idle, then you can either configure it for a longer idle time, or code your clients to query Sentinel more often so the device won't see them as idle.  If it thinks the connections have stayed up too long, then you can either configure it to allow longer connection times, or perhaps code your client to disconnect and re-connect to Sentinel after a period of time.  The deliberate disconnections will be noticed by the Sentinel service, and it will close the connections.  The connections will not keep accumulating.

  -Greg

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.