Max connections in Redis

[chilumb...@gmail.com]

Is there a command or a way to find out what the maximum number of connections on my redis cluster is? I have a redis cluster with multiple client connections, and i would like to find out what was the maximum  number of client connections my redis cluster has so far dealt with with processing requests.

[Marc Gravell]

At the moment, I think the only way you would have that would be to poll and record the connected_clients value from "info clients" against all nodes separately - and look at that data. There is also a total_connections_received value in "info stats", but no "max_connected_clients" at the current time, AFAIK.

Marc

[chilumb...@gmail.com]

Thanks Marc for the reply.

For my three maser nodes in the cluster, the each have a over 6 million total_connetions_received. The numbers seems pretty high.

Do these numbers indicate that the connections are still open, or just a record of a counter each time the connections is established (and the number is never reset but continues to increment with each new or repeated connection)

[Greg Andrews]

The total_connections_received value does not indicate the count of currently open connections.  It is the number of connections accepted by that Redis daemon since it started.  Some of those connections are probably open at the moment you sent the INFO command, but most likely not all of them.  You can take this and divide it by the uptime_in_seconds value, to see if your client software is pooling connections (good) or not (usually not good).

In the client section of the INFO output there is the connected_clients value, which is probably what you're actually interested in.

  -Greg

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[The Baldguy]

On Sunday, November 8, 2015 at 10:16:16 PM UTC-6, chilumb...@gmail.com wrote:

Is there a command or a way to find out what the maximum number of connections on my redis cluster is? I have a redis cluster with multiple client connections, and i would like to find out what was the maximum  number of client connections my redis cluster has so far dealt with with processing requests.

I have to ask what you're actually after as you can easily get down the wrong path here. Total connections ever handled isn't really representative expect for pathological cases where the client opens the connection for each command or sets versus the right way of open connection and reuse it.

What I'd recommend here is to to track your current connection count over time. You can do this vi the `info clients` command which returns the following as an example:

# Clients

connected_clients:4

client_longest_output_list:0

client_biggest_input_buf:0

blocked_clients:0

This will tell you much more useful information, particular if tracked over time. You should periodically (1 minute?) call this and store it in a DB (which can be another Redis instance, storing it a variety of ways) to look at how each of these items changes. The `connected_clients` piece will give you a concurrency figure. The `blocked_clients` being above 0 is indicative of clients issuing commands which have a negative performance ripple effect.

You will need to do this on every master in your cluster. You haven't specified if you're using Redis Cluster or simple replication (which I refer to as a 'pod'), but in either case doing this across all masters and then summing them will give you the broad spectrum. However I have found the per-master figures are more illustrative. If you have a master consistent running much higher (I'd go for determining and measuring standard deviations here) than the others you likely have found a "hot spot" - where many more keys map to a single shard.

Now all of that said, you can do the same thing but add in the `total_connections` and track it over time but it won't be consistent. The reason? Failovers. Failovers will essentially reset that as the out of band slave is promoted and thus the reported connection count will drop - often dramatically. This is another reason I don't recommend using that number as a metric for your system. You'll also see this if an instance restarts. Thus it isn't reliably accurate.

Cheers,

Bill

[George Chilumbu]

Thanks for the explanation Greg. Makes sense now.

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/m5_Kk2qe15c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.

[George Chilumbu]

@Bill, i will try your suggestions about tracking the client connections. 

By the way, i am using the redis cluster with three master nodes each with a single slave.

The reason for posting this question is because we are receiving a lot of the following error message below from our predis logs:

 Error while reading from the Line Server [. tcp: //10.0.0.80: 6379 ] 

Since redis is single threaded, i was suspecting that; as the number of clients grow, the percentage of resource time given to each client decreases and each client spends an increasing amount of time waiting for their share of Redis server time. And in the process of waiting in the command queue for too long, the connections somehow timeout. This is the reason i am interest in understanding the redis connections in this post. 

It is also important to note that the redis timeout on all nodes in the cluster is set to 0, the tcp-keepalie is 60, and the tcp-backlog is set to 65535 same as the system's somaxconn value.

On predis, we have the read_write_timeout set to -1, and the timeout set to 0.

The linux open files is set to 1024000 on each node in the cluster, and the max user processes is set to 10240 (these van be viewed using the "ulimit -a" command in the linux terminal). The txqueuelen is set to the default 1000 when you run ifconfig. I am using ubuntu 14.04 with 8GB RAM and 40GN disk. 

Periodically, we have cron job performing bursts of heavy writes from our social media. I am also suspecting this might be resulting in blocking the redis daemon with the burst of writes and as other clients send more queries. So maybe when redis clients send a large number of commands to the redis server in a short period of time, this clocks the redis server and resulting in the error message show above. 

These are our theories now. But we are not sure how to prove them. Or we are not sure if they might be the root cause of this error message.

[Marc Gravell]

The first thing to check would be "slowlog get" - see if it is showing any *particularly* expensive things going on (like "keys", etc)

The second thing to check would be the server log - see if it is showing anything like persistence or replication at the time it is failing.

The number of connections is an interesting hypothesis, and one that you should be able to investigate by just  recording the connected clients on a poll - see how big it is getting. I would not, however, think this is a likely cause of problems. I'd be far more interested in the ops/s, for example (or record the total ops processed on the poll, and see if it jumps).

Marc

Regards,

Marc

[chilumb...@gmail.com]

Thanks for the feedback Marc.

I now have a cron job running every minute and recording the values of total_commands_processed, total_connections_received and client_connections using the "redis-cli info" command.

I will let the cronjob run for a day then analyze the results and try to see look for any spikes especially with the total_commands_processed.