Running beanstalkd in ubuntu 14.04 LTS in Amazon EC2 has issues if root's ulimit is not set properly, how to init the beanstalkd process with a pre-defined ulimit

[Collin Hayden]

Hi all,

I am currently running instances on the latest ubuntu 14.04.1 LTS image in amazon ec2.  I am having several issues that I have discovered and need some help in confirming how the application currently works and functions.

First, does the application run or hand off to root?  I have noticed that when using the package from ubuntu (1.9-2ubuntu1), that the installation contains both the /etc/init.d/beanstalkd script and the /etc/default/beanstalkd file to make optional adjustments to the configuration.  I have played around with persistence and not persistence and it works fine.  However I did notice that the logging in the init script is not setup yet.  I was able to set the logging to a file by modifying the init script so that it redirects stdout and err to a file.  I noticed that the init script is set to run the application as the user "beanstalkd".  I set the ulimit for the beanstalkd user to 64000 for soft and hard open files since we were hitting the limit of 1024 (the default in linux).  However, when I use the benchmark tool and open example 5000 connections I noticed that beanstalkd is able to open the connections but then in the log it hands off it root and root's user starts complaining about too many open files.  If I don't set my logging in the init script and leave it as the default install, the root entries go to the console and because there are thousands of connections it ends up over-flowing the serial buffer for tty0 and tty1 ( I opened a ticket w/ amazon regarding this and it seems they use the ttyS0 and ttyS1 for outputting console messages to the dashboard for the instance in ec2).  The issue here is that the box gets hung up if you hit it w/ enough benchmark connections because the ttyS0 gets overflowed and the kern.log will show messages like:

"kernel: [  311.699800] serial8250: too much work for irq4"

The solution is to set the ulimit for the root user to 64000 in the /etc/security/ulimit.conf file or /etc/security/limits.d/beanstalkd (i created this file with entries so that it is read).  Here is what mine looks like:

cat /etc/security/limits.d/beanstalkd.conf 

#<domain>      <type>  <item>         <value>#

beanstalkd     soft    nofile         64000

beanstalkd     hard    nofile         64000

# Adding root as it appears that it is needed to function for beanstalkd

root     soft    nofile          64000

root     hard    nofile         64000

However from what I have read here:  http://linux.die.net/man/5/limits.conf and http://linux.die.net/man/8/pam_limits , the limit.conf applies to pam.  I set the common-session and common-session-noninteractive to both honor the pam_limits.so so that when you auth using pam in a shell that the ulimits reflect appropriately.  I gave beanstalkd a shell so I could test the ulimit by doing a "sudo su - beanstalkd && ulimit -Sa" and see the correct openfiles.  This works fine.  The ulimits are reflected for both root and beanstalkd when using pam.  I however noticed that upon reboot that during the init process the beanstalkd daemon boots up but the beanstalkd doesn't use pam apparently as the ulimits don't take effect and when benchmarking, it crashes the machine because the output of the console messages is too great for the ttyS0 (location where they are going w/o log redirection and default setup) and hangs the box.  So my main issue is how to set the ulimit for the root user during initial boot?  

I know that you can set it in upstart, mongodb comes with this and here is an example:  http://docs.mongodb.org/manual/reference/ulimit/ for the user that the script runs as.  Right now it is not an upstart script, do you guys have one?  I have seen that the github project has a simple one, but the users are complaining it doesn't work in this list.  The main issue is that even with an upstart script, you cannot set the ulimit for the root user, since it appears that the ulimit is necessary for root in order to open up enough sockets (open files).  It will be set to whoever runs the process and the init script, that initially is beanstalkd.

The reason why this is a big issue is that we are currently looking to use the persistence for our implementation and what I discovered is that you need an equal amount of RAM for the amount of persistant data on disk in the binlog files that are created in /var/lib/beanstalkd.  When the beanstalk comes up, if there is a huge amount of persistent messages it takes time for it to get into memory so it appears the hang out for a while, but in actuality it's loading the bin data into memory.  We have instances setup to have the sufficient amount of memory for the total messages that need to persist for us.  Right now we need just one day, so 16 gig on an EC2 M3 XL instance is sufficient.  I noticed that you cannot free up the memory if using persistence by just blowing away the files, you actually have to restart the beanstalkd in order for the process to free up the memory.  So for us we are planning to crontab something to make it so that we reboot the instance once a day and delete the bin logs and free up the memory.  But if our system crashes during the day we are fine since we have processed the messages we needed to.  We are going to make the clients "connection smart" so that they will re-connect to beanstalkd if the process goes away.  In order to implement this we want the beanstalkd to setup the ulimits on boot up through the init process, how can we go about this if root needs to be setup?  The other question is why is root needed, why can' t the app run purely under beanstalkd user?   Right now I have to restart the application in a shell and because the shell is using pam and the process starts with the ulimit being acknowledged through pam, things work.  But this won't work for the cron restart to free up memory.  I need the ulimit to be honored during init for root.  

Suggestions on how I can accomplish this?

Another question I have is:  "Is the application running under Pam?"  It doesn't seem to be by what I have seen and implied above.

I can share the modified init script that contains the log settings and also the /etc/default/beanstalkd if others need an example of getting verbose logging to work.  A note here, is that when the logging is set the crash in amazon doesn't happen as the messages are output to the file and not the serial.  I can also show examples of how I setup benchmarking.  I used the following:  https://github.com/fangli/beanstalkd-benchmark

The throughput on beanstalk is pretty amazing... thanks.

Best Regards,

Collin

[Collin Hayden]

I found out that if I add something like the following to the init script, that this in turn fixes the issue I was having on initial reboot of the open files not being honored:

## Added by chayden on Oct 06, 2014

#Set the ulimit since beanstalkd doesn't use PAM and the limit.conf in /etc/security uses pam_limits.  We need a way to set the ulimit

# on boot up.

ulimit -n 64000

What I still don't understand then is, how does this not require that the root user have the ulimit set like prior.  For some reason when started in a shell and using pam the root user must be set to 64000 otherwise even if beanstalkd is set to 64000 it complains of too many open files.  I guess I don't understand the difference in init of beanstalkd and of shell start of beanstalkd and how beanstalk works with the 2.  

Regards,

Collin

[Shendy Kurnia]

Hi Collin,

I happen to be in the same situation, ie seeing 'kernel serial8250 too much work for irq4' in /var/log/syslog, on Ubuntu 14.04, in Amazon EC2, with beanstalkd running.

However, after following your solution, ie creating /etc/security/limits.d/beanstalkd.conf with the same content per your post and put 'ulimit -n 64000' in /etc/init.d/beanstalkd, I still see the problem.

Please advise.