maxclients question

[Roshan Shetty]

Hi,

We have maxclients commented in our config file. However we are seeing connection timeout messages when number of connections exceed this range.

This bothers me because we are on a production system and config set does not work for maxclients parameter.

I checked files descriptors which is already something like 64K. Also I disabled syn flooding cookie (server was reporting syn flood from web host) but if someone can clarify why this may be happening? Thank you.

Thanks
Roshan

[Pieter Noordhuis]

Exceeds which range? The one that is commented in the config file?
Redis has no limit on the maximum number of clients by default, so
Redis will itself will not be the cause when you don't specify
anything in the configuration file. If it is specified AND the limit
is enforces, incoming connections that exceed the limit will receive
an error message and be closed immediately. Because you are talking
about timeouts, this is unlikely to be related to the `maxclients`
directive.

You say you changed the maximum number of file descriptors as well:
did you do so on a system level and/or on the user level (`ulimit
-n`)?

If you provide this and any more information (such as Redis version,
output of INFO, operating system, TCP tweaks in the kernel), we might
be able to help out.

Cheers,
Pieter

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
>

[Luis P]

I would like to confirm this in our case. I was testing Redis 2.2.12
with tcmalloc on 64bit Ubuntu 10.04. Specifically, this is an m1.large
EC2 instance based on the ami-63be790a AMI. The contents of redis.conf
are
daemonize no
port 6379
timeout 300
loglevel notice
logfile /dev/null
syslog-enabled yes
syslog-ident redis
syslog-facility local0
save 900 1
save 300 10
save 60 10000
dbfilename dump.rdb
dir ./

The maximum number of open file descriptors have been increased to
100000, as seen in the output of ulimit -a:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 100000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

As for TCP tweaks in the kernel, this is the content of /etc/
sysctl.conf
vm.overcommit_memory = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.ip_forward = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.lo.send_redirects = 0
net.ipv4.conf.eth0.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 3000
net.core.netdev_max_backlog = 3000

I connected to the redis server using the redis-py python package.
Each client is doing operations on the redis server. Everything is
fine up to 1000 connections. When I tested 1200 connections, the CPU
usage of the redis server stayed at 100%, and "redis-cli info" hanged
until I killed the test clients.

Weirdly, there is no problem if I try to test idle clients using
"redis-benchmark -c 1200 -I". There is also no output in the redis log
file to indicate an error has occurred.

Thank you for your time and for any help you could give.

On Aug 22, 8:30 pm, Pieter Noordhuis <pcnoordh...@gmail.com> wrote:
> Exceeds which range? The one that is commented in the config file?

> Redis has no limit on the maximum number ofclientsby default, so

[Didier Spezia]

Hi Luis,

>net.core.somaxconn = 3000 

>net.core.netdev_max_backlog = 3000 

This looks a bit suspicious to me.

The default value of 128 set in most distros is usually fine.

Anyway, Redis limits the listen backlog to 511 (magic constant also used by nginx).

So even if you put 3000, it will still be limited to much less.

The redis-benchmark program makes use of 50 ms delays every 64 connections

in order to open a large number of connections in a short timeframe without any

error despite the limited backlog.

Do your python clients all try to open their connection to Redis simultaneously?

Do you have one connection per Python client?

Are these clients running on the same box as the Redis server?

If we suppose the 1000 client processes are launched at the same time,

each of them taking about 5 MB of resident memory, the OS has to

fork 1000 processes and allocate about 5 GB of memory for the clients.

This is a lot of work for a VM with less that 8 GB and 4 vCPUs.

In this situation, it would not surprise me if Redis, one process

among others, failed to grab the CPU it would need to be kept

responsive ...

Regards,

Didier.

[Luis P]

Hello Didier,

Thanks for the reply. I will try to retest with net.core.somaxconn and
net.core.netdev_max_backlog unchanged, but I think it wouldn't affect
the result.

The python clients are located on a different instance. The script I
used for testing created 200 threads that connect to the redis server
and issue commands simultaneously. So I am running 6 of those scripts,
1200 connections, when the problem occurred.

I retested using an m2.xlarge instance. Since it has bigger compute
units per core, I thought it could handle a higher number of
connections, but I got the same result.

[Didier Spezia]

Sorry, my assumptions were completely wrong.

Did you try redis-benchmark in local or on the client machine?

Is Redis CPU at 100% when it is unresponsive?

If you can still make Redis hang, perhaps you can use the

procedure described in this issue to try to understand

where Redis is stuck ...

http://code.google.com/p/redis/issues/detail?id=648

Regards,

Didier.

[Luis P]

Hello Didier,

Thanks for the suggestion. I was able to pinpoint the problem to
redis having too many open files. I think there's a problem with the
way it is started at bootup, since if I try to start it from the
command line then it works fine. Sorry for the trouble.

[Luis P]

Hello. I am now encountering a problem where the maxclients is limited
to 10234. It seems to be related to the issue described here:
http://code.google.com/p/redis/issues/detail?id=372

In the log file there is only an error message saying "Error
allocating resources for the client". I can confirm that the number of
file descriptors limit has been increased to 100000. I can see in the
code the line that outputs the error message but I don't understand
what triggers it. Thanks for your time.

[Didier Spezia]

Hi Luis,

in file ae.h you have:

#define AE_SETSIZE (1024*10)    /* Max number of fd supported */

You may want to try to increase this limit and recompile Redis.

Regards,

Didier.

[Luis P]

Hi Didier,

That solved the problem, thank you very much.