I have a Redis cluster running on ElastiCache.
Multiple processes connect to the Redis cluster. Each process lives within a Docker container. The processes are not all the same -- I have a web
process, a worker
process, etc.
After a few days of operating normally, some of my web
processes begin to time out when connecting to Redis. When I ssh
into an affected web
container and try to reach the cluster via redis-cli
, my connection to the cluster times out. This tells me that the issue affects the entire container, and not just the web
process.
When I use redis-cli
from any other container, I connect without problems.
My web
processes create new connections as needed, and close old connections when they're idle for a long time. My guess is that any given Docker container can open a certain number of connections before reaching some kind of limit. After a few days, my web
containers reach that limit.
Any idea how to go about fixing this?
--
One more detail: for some reason, resetting my Redis cluster fixes the issue across all web
containers. Maybe the Redis server imposes a limit on how many connections can be opened from a given IP address?
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.
Redis does have total connection limits, but Redis does not limit on a per-IP basis. I suspect that AWS has hidden/removed the 'CONFIG' command, which would let you 'GET' the 'maxclients' option, which is the maximum number of connected clients allowed.When your web process is having problems connecting to Redis, can you create a connection from one of your other Docker containers?
Have you tried connecting to Redis from another server entirely?
Have you checked the number of open connections in your web container with 'lsof' to verify that your outgoing ephemeral ports are not filled?
Are you *sure* you are pooling and re-using your Redis connections?
Hi Josiah, answers inline:
On Wednesday, July 1, 2015 at 11:25:10 PM UTC-7, Josiah Carlson wrote:Redis does have total connection limits, but Redis does not limit on a per-IP basis. I suspect that AWS has hidden/removed the 'CONFIG' command, which would let you 'GET' the 'maxclients' option, which is the maximum number of connected clients allowed.When your web process is having problems connecting to Redis, can you create a connection from one of your other Docker containers?Yes. I can connect from other containers (on other machines) just fine. I'm only running 1 container per machine for now.Have you tried connecting to Redis from another server entirely?Yes. Other servers connect just fine.Have you checked the number of open connections in your web container with 'lsof' to verify that your outgoing ephemeral ports are not filled?Good question -- I think this is related to my problem."lsof | wc -l" hovers around 2600. After a while, this number suddenly drops to ~500.
"netstat -s" says there are "3284 active connections openings" and "14440 passive connection openings". From what I can tell, active & passive connection openings keep growing without bound.
Are you *sure* you are pooling and re-using your Redis connections?AFAIK, yes. I connect to Redis using redis-rb within the connection-pool gem. I use nginx + Phusion Passenger for my web process.