Redis client times out when connecting

[Federico Nusymowicz]

I have a Redis cluster running on ElastiCache.

Multiple processes connect to the Redis cluster. Each process lives within a Docker container. The processes are not all the same -- I have a web process, a worker process, etc.

After a few days of operating normally, some of my web processes begin to time out when connecting to Redis. When I ssh into an affected web container and try to reach the cluster via redis-cli, my connection to the cluster times out. This tells me that the issue affects the entire container, and not just the web process.

When I use redis-cli from any other container, I connect without problems.

My web processes create new connections as needed, and close old connections when they're idle for a long time. My guess is that any given Docker container can open a certain number of connections before reaching some kind of limit. After a few days, my web containers reach that limit.

Any idea how to go about fixing this?

--

One more detail: for some reason, resetting my Redis cluster fixes the issue across all webcontainers. Maybe the Redis server imposes a limit on how many connections can be opened from a given IP address?

[Josiah Carlson]

Redis does have total connection limits, but Redis does not limit on a per-IP basis. I suspect that AWS has hidden/removed the 'CONFIG' command, which would let you 'GET' the 'maxclients' option, which is the maximum number of connected clients allowed.

When your web process is having problems connecting to Redis, can you create a connection from one of your other Docker containers? Have you tried connecting to Redis from another server entirely? Have you checked the number of open connections in your web container with 'lsof' to verify that your outgoing ephemeral ports are not filled? Are you *sure* you are pooling and re-using your Redis connections?

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Federico Nusymowicz]

Hi Josiah, answers inline:

On Wednesday, July 1, 2015 at 11:25:10 PM UTC-7, Josiah Carlson wrote:

Redis does have total connection limits, but Redis does not limit on a per-IP basis. I suspect that AWS has hidden/removed the 'CONFIG' command, which would let you 'GET' the 'maxclients' option, which is the maximum number of connected clients allowed.

When your web process is having problems connecting to Redis, can you create a connection from one of your other Docker containers?

Yes. I can connect from other containers (on other machines) just fine. I'm only running 1 container per machine for now.

 

Have you tried connecting to Redis from another server entirely?

Yes. Other servers connect just fine.

 

Have you checked the number of open connections in your web container with 'lsof' to verify that your outgoing ephemeral ports are not filled?

Good question -- I think this is related to my problem.

"lsof | wc -l" hovers around 2600. After a while, this number suddenly drops to ~500.

"netstat -s" says there are "3284 active connections openings" and "14440 passive connection openings". From what I can tell, active & passive connection openings keep growing without bound.

 

Are you *sure* you are pooling and re-using your Redis connections?

AFAIK, yes. I connect to Redis using redis-rb within the connection-pool gem. I use nginx + Phusion Passenger for my web process.

[Josiah Carlson]

On Thu, Jul 2, 2015 at 11:38 AM, Federico Nusymowicz <fede...@gmail.com> wrote:

Hi Josiah, answers inline:

On Wednesday, July 1, 2015 at 11:25:10 PM UTC-7, Josiah Carlson wrote:

Redis does have total connection limits, but Redis does not limit on a per-IP basis. I suspect that AWS has hidden/removed the 'CONFIG' command, which would let you 'GET' the 'maxclients' option, which is the maximum number of connected clients allowed.

When your web process is having problems connecting to Redis, can you create a connection from one of your other Docker containers?

Yes. I can connect from other containers (on other machines) just fine. I'm only running 1 container per machine for now.

 

Have you tried connecting to Redis from another server entirely?

Yes. Other servers connect just fine.

 

Have you checked the number of open connections in your web container with 'lsof' to verify that your outgoing ephemeral ports are not filled?

Good question -- I think this is related to my problem.

"lsof | wc -l" hovers around 2600. After a while, this number suddenly drops to ~500.

I should have been more specific, run "sudo lsof -i" to get the IP-related connections. To discover the number of connections that are waiting to close, grep for "CLOSE_WAIT". To get the number of connections established, grep for "ESTABLISHED". You'll also want to do this in the host outside the container.

"netstat -s" says there are "3284 active connections openings" and "14440 passive connection openings". From what I can tell, active & passive connection openings keep growing without bound.

I believe you are misinterpreting the output of "netstat -s". These numbers *should* grow without bound over time, assuming connections eventually time out and need to be re-established. They don't represent the current number of connections established, just the total number of outgoing and incoming connections that have ever been established. Even so, that number of connections to redis (even established) shouldn't be a problem, unless the cluster you rented is made up of slow machines. What instance size are you using?

Are you *sure* you are pooling and re-using your Redis connections?

AFAIK, yes. I connect to Redis using redis-rb within the connection-pool gem. I use nginx + Phusion Passenger for my web process.

I don't know enough about Phusion Passenger to know whether it is doing anything right for Ruby. A quick search through the docs/code leads me to a part that seemed to talk about preforking, which is never a good sign regarding connection pooling.

In Redis, what does INFO say for 'total_connections_received'? What does it say if you wait 1 minute and check it again?

 - Josiah

[Federico Nusymowicz]

The error hasn't reoccurred yet. I turned on TCP keepalives both client- and server-side, which may have fixed the issue.

If the error pops up again, I'll try moving away from connection pools within Phusion Passenger. I'll switch to one global connection per forked process, which seems to be the standard approach.

If that fails, I'll try "sudo lsof -i" and INFO within Redis and get back to you.

Thank you for all your help Josiah!

Best,

Federico

[Andrew Rembrandt]

Hi,

I've been seeing the same can't connect from localhost, but am able to connect remotely to the redis instance with MS's windows port of redis, v2.8.19.

(running on windows 2003 R2 x86)

Relevant info details are:

[connected_clients, 644]

[blocked_clients, 0]

[total_connections_received, 5415]

Ironically, I was able to telnet to port 6379 and issue commands fine from localhost - something was awry with the client libraries (both redis-cli from v2.8.19 and StackExchange.Redis were unable to connect).

Hope this helps anyone else who see's a re-occurrence of this.

Regards,

Andrew