Need help with max connections settings / Load testing

[Mahesh Vernekar]

We are hosting a web application with configuration as below:

Application: Apache 2.4 / Python 3.4 / Django 2.0

Database servers x 6 : 32 GB / 8 cores behind load balancer

Web servers x 6 : 4 GB / 2 cores behind load balancer

Setup is in AWS so we are using AWS-Elastic load balancer. 

The application supports 2000 concurrent connections but failing for 2500 and beyond.

In load testing around 1800 requests are failing out of 3.5 lakh total requests.

In the apache error log we are seeing the error message as below ?

[Wed Jan 02 14:05:53.349209 2019] [wsgi:error] [pid 21559:tid 139722408036096] (11)Resource temporarily unavailable: [client 172.31.12.61:51748] mod_wsgi (pid=21559): Unable to connect to WSGI daemon process 'pe-ta-dev.knowdl.com' on '/run/httpd/wsgi.21472.0.1.sock' after multiple attempts as listener backlog limit was exceeded or the socket does not exist.

Any idea how we can resolve the issue ?

Regards

Mahesh

[Graham Dumpleton]

What is your current mod_wsgi configuration?

How long is typical request response time?

Are the requests high CPU activities, or I/O bound waiting on a backend service like a database?

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Mahesh Vernekar]

Hi Graham,

Thanks for your response. Find my answers as below:

What is your current mod_wsgi configuration? We are using deamon process

How long is typical request response time? Min : 2 secs / Max : 15 secs

Are the requests high CPU activities, or I/O bound waiting on a backend service like a database? Though the application is connected to a backend mariadb database there are no high cpu activities. The CPU does reaches 50% during the tests.

Regards

Mahesh

You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/aIcET2VqJbA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.

[Graham Dumpleton]

On 3 Jan 2019, at 5:26 pm, Mahesh Vernekar <mahesh....@gmail.com> wrote:

Hi Graham,

Thanks for your response. Find my answers as below:

What is your current mod_wsgi configuration? We are using deamon process

I need to see the actual configuration, just saying you are using daemon mode is not enough as means I have to guess everything, which makes it impossible to recommend anything.

That is, I need to see the directives you put in the Apache configuration file related to mod_wsgi. This is so I can see if embedded mode is disabled, that daemon mode is configured appropriate for processes/threads. Whether you set all the recommended timeouts etc etc.

For a bit of background also suggest you watch:

https://www.youtube.com/watch?v=CPz0s1CQsTE

This blog post may also be relevant in upcoming discussion:

http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html

How long is typical request response time? Min : 2 secs / Max : 15 secs

That is a very large value for a web server that is dependent on processes/threads and not async.

Are the requests high CPU activities, or I/O bound waiting on a backend service like a database? Though the application is connected to a backend mariadb database there are no high cpu activities. The CPU does reaches 50% during the tests.

Also, have you added any instrumentation your web application to monitoring response times and/or mod_wsgi performance?

[Mahesh Vernekar]

Hi Graham,

I have attached the httpd.conf file contents. Will that help ?

Regards

Mahesh

[Mahesh Vernekar]

Hi Graham,

Did you get a chance to go through the apache configuration file ?

Do you need more details ?

Regards

Mahesh Vernekar

[Graham Dumpleton]

I have been busy today so far so haven't had time to reply.

Did you watch the video and read the blog post I linked?

A couple of other videos worth watching to give you better background on issues and how Apache/mod_wsgi works are:

* https://www.youtube.com/watch?v=SGleKfigMsk&t=593s

* https://www.youtube.com/watch?v=k6Erh7oHvns

Watch the first of those especially and some of the problems with your configuration should become more obvious.

Graham

[Mahesh Vernekar]

Hi Graham,

Thanks for your time.

We resolved the issue by adding the parameter connect-timeout to the deamon-process configuration as below

WSGIDaemonProcess pe-ta-dev.knowdl.com python-path=/opt/data/knauthor/knauthor:/usr/lib64/python3.4/site-packages connect-timeout=300

Since our application need to support 5000 concurrent users we kept the time as 300 secs.

Regards

Mahesh

[Graham Dumpleton]

That isn't really a solution, it is more of a hack that doesn't really solve the real problem.

If you had watched the the video:

https://www.youtube.com/watch?v=SGleKfigMsk&t=593s

I was hoping you would start to understand that the real problem is that you are using the default processes/threads configuration, which is one process with fifteen threads.

If your requests all take 2 seconds, then, that means you can only handle 7 1/2 requests per second.

What you should be doing is adding the processes option to specify many more processes, and then for those, provided you are only I/O bound and not CPU bound, increasing the number of threads.

Thus:

    WSGIDaemonProcess pe-ta-dev.knowdl.com python-path=/opt/data/knauthor/knauthor processes=10 threads=30

A large number of threads in Python is not great, but if is really I/O bound and not CPU bound, it may be okay.

With that configuration, you can know handle 150 requests per second at 2 seconds per request.

Do note that increasing the capacity in the daemon process, may mean you have to adjust the MPM settings in Apache for its workers. Also make sure you are not using prefork MPM, use event MPM (or worker MPM if no choice). If the MPM settings have less capacity that processes*threads for daemon process, then the MPM settings would need to be increased to allow more capacity.

Overall though, Apache is not going to handle 5000 concurrent connections at 2+ seconds per request. So I question that figure. What is probably really happening is that clients are stuck waiting in the Apache socket listener queue, which can cause big backlog issues when things get overloaded.

Important thing is that simply increasing timeouts generally means your configuration is still wrong and not really designed to handle the capacity properly.

BTW, your inclusion of '/usr/lib64/python3.4/site-packages' in python-path is also wrong.