Unable to change number of open files for redis-server process

[Itai Ganot]

Hi all,

There's a problem I'm having a hard time to solve.
There's a redis server in the company which gets metrics from monitoring tools in the company.
Quite often, servers are unable to send metrics to the redis and when I look at the logs I can see the following error:
"ERR max number of clients reached"
Max number of clients is affected by the number of open files which the system allows, so in order to fix the issue the number of open files should be increased and that's where my problem is... even though I've increased the number of open files everywhere I could think of, it seems like the increase isn't applied.

Here's where I've increased it:
root@sensu:~# grep nofile /etc/security/limits.conf
# - nofile - max number of open files
* soft nofile 100001
* hard nofile 100001
root soft nofile 100001
root hard nofile 100001

root@sensu:~# grep nofile /etc/security/limits.d/90-nofiles.conf
* soft nofile 100000
* hard nofile 100000
redis soft nofile 100001
redis hard nofile 100001
root soft nofile 100001
root hard nofile 100001

root@sensu:~# su - redis -c "ulimit -n"
100001

root@sensu:~# grep -i ulimit /etc/init.d/redis-server
ULIMIT=100001
if [ -n "$ULIMIT" ]
ulimit -n 100001
root@sensu:~# grep file.max /etc/sysctl.conf
#fs.file-max = 70000
fs.file-max = 100001

root@sensu:~# grep maxclient /etc/redis/redis.conf
maxclients 100001

And you can see it says 100001 everywhere...
but, when I check the limits of the process itself:

root@sensu:~# ps -ef | grep redis
root 15426 11910 0 08:59 pts/2 00:00:00 grep --color=auto redis
redis 30537 1 0 Mar05 ? 00:14:41 /usr/bin/redis-server sensu.company.com:6379
root@sensu:~# cat /proc/30537/limits | grep "open files"
Max open files 4081 4081 files
root@sensu:~#

I don't understand where the limit of 4081 is coming from, do you have any idea?

Thanks in advance,

Itai

[hva...@gmail.com]

A process receives its limits when it is first created (also known as "spawned" or "forked").  If you start (or restart) your Redis server process after logging into the server, then the Redis server process usually receives its limits from your interactive shell process.  Your interactive shell process received its limits (set in /etc/security/limits.conf) when you logged into the server.

So if you changed the limits.conf file and didn't restart your Redis server daemon, it won't receive any new limits.  It's also true in many cases that, after changing the limits.conf file, you will need to log out of the server and back in, so your interactive shell receives the new limits and can pass them to Redis when you restart Redis.  (this inheriting limits from your interactive shell is changing as Linux distros switch to systemd to start/stop/restart daemons like Redis)

In the past, when I was changing limits on a daemon like this, I had to log out and back in, then restart the daemon.  Then I could be sure the changes were picked up.

As I hinted a couple of sentences ago, if your servers are using systemd to start/stop/restart daemons, you may need to check the systemd configuration to see if it's forcing the file descriptor limits to something other than the limits.conf setting.

[Itai Ganot]

Hi,

I've rebooted the server and still the process gets the limit of 4081.

systemd indeed manages the services but as you can see no limit is configured there:

root@sensu:/etc/systemd# grep -v '^#' *.conf

bootchart.conf:

bootchart.conf:[Bootchart]

journald.conf:

journald.conf:[Journal]

logind.conf:

logind.conf:[Login]

resolved.conf:

resolved.conf:[Resolve]

system.conf:

system.conf:[Manager]

timesyncd.conf:

timesyncd.conf:[Time]

user.conf:

user.conf:[Manager]

Please, do you have any more suggestions? I have to solve this issue

Thanks

Itai

[hva...@gmail.com]

I did a google search for "systemd" "limit" and "open files", and found these:

  http://serverfault.com/questions/628610/increasing-nproc-for-processes-launched-by-systemd-on-centos-7

  http://unix.stackexchange.com/questions/281684/configuring-ulimit-of-max-open-files-for-daemonized-nginx-master-process

Which indicates systemd doesn't pay attention to the /etc/security/limits.conf file.  (This would be completely consistent with the things I've heard about systemd)

The first question is about CentOS 7 and the second is about Debian.  These will cover many of the distros currently in use.  (I included both links because other people using other distros will probably also be interested in this)

  -Greg

[Itai Ganot]

Hi,

I've read the links and tried what was suggested in the answers but to no avail.. the process is still limited by 4081, any other thoughts regarding this issue?

Thanks

Itai

[hva...@gmail.com]

I don't have any suggestions except to start your Redis without systemd and see what the file limits are, in order to confirm that systemd is creating this limit.  Once it's confirmed, persue the systemd configuration to make it set the proper limit on Redis.

[Itai Ganot]

It has been solved by editing redis.service systemd file and then reloading the configuration.

Thank you.

Itai

[George L]

For CentOS 7 systemd redis service at least via Remi yum repo you can just do a sed replace of the predefined limit set in  /etc/systemd/system/redis.service.d/limit.conf :)

sed -i 's|LimitNOFILE=.*|LimitNOFILE=262144|' /etc/systemd/system/redis.service.d/limit.conf

[kaleeswaran s]

I am facing the similar issue, What are all the changes that you have done on the redis.service file.

[hva...@gmail.com]

Itai's sed command changed the LimitNOFILE line to have LimitNOFILE=262144 in the /etc/systemd/system/redis.service.d/limit.conf file.

[hva...@gmail.com]

I wrote Itai, but it was actually George L who submitted the sed command.