Beanstalkd Timeout on Spike Traffic

[Jonathan Martins]

Hello Folks.

I have a problem quite a time:

First I will explain my infrastructure a bit:

 All instances are running Ubuntu on AWS Services. 
- One ELB with multiple instances attached, that I call: API Servers .
- One single instance running only beanstalkd.
- One single instance running the workers.

The API receives requests, put them into beanstalkd, next the worker get the jobs and process it. Basic stuff...

After pre warn my ELB with some front servers I wait for the incoming traffic. When I have a spike traffic in a short period of time beanstalkd cannot handle all requests. It gives me a timeout message. Sometimes it happened only a few seconds, but it is enough to lose some 100k+ requests

Notes
- All API servers that run during the spike time, threshold a maximum of 50% CPU, no problem to receive traffic.

- My beanstalkd server thresholds a maximum of 15% CPU, and every time I have a lot of free RAM left. I never ever have more that 20% CPU usage on this server. Beanstalkd seems to be very efficient.

- During spike time the worker server get at 100% CPU, like is expected.

I have tried
- Scale the beanstalkd server UP, (more CPU, more RAM, best Network): no improvement (a micro instance gives me the same results of a xlarge) 
- Next I try to run more that one process in different ports: was worst! :(, why ?
- I Increased the connection timeout, but I cannot run under 30s max. normally, I run my services in much lower timeouts.

Lets go to questions:

Does Ubuntu Kernel limits influence in some case? Does beanstalkd open a file descriptor, or it runs only in memory?
If I use another OS, like CentoOS, or the Amazon Linux AMI, can I get better results with beanstalkd?

Another thing that I guess is struggling the hole process are my workers, can a server consuming and processing lots of jobs from beanstalkd interfere in the connection of my API servers?

Should I open another internal ELB and scale beanstalkd servers horizontally? 

For now every time that I get a spike traffic, I save the failing requests to another DB, so later on, I can put all back in beanstalkd.

[Anthony Hobbs]

This reply probably comes too late for you but, did you try changing your file/socket config within the os?
things like somaxconn and ephemeral port reuse are important if you are opening and closing a lot of connections.
The best guide I have come across is here: https://tweaked.io/guide/kernel/