Tuning apache and mod_wsgi with 200 concurrent requests and 1 thread per process

[Paul Royik]

StartServers 50
ServerLimit 200

ThreadsPerChild 1
ThreadLimit 1

MaxRequestWorkers 200
MaxConnectionsPerChild 10000

WSGIApplicationGroup %{GLOBAL}
WSGIDaemonProcess process processes=75 threads=1

Is it enough? Or can it handle only 75 concurrent requests? I don't know how to synchronize apache and mod_wsgi settings. 

[Graham Dumpleton]

What sort of application are you running?

What is your average response times?

Do you have long running requests, if yes, how long?

What Apache MPM are you actually using?

My initial impression is that is a quite poor configuration which is only going to chew up huge amounts of memory for no good reason, but I don't know your application requirements.

Also, are you even setting WSGIProcessGroup?  If it isn't set it makes the whole daemon process configuration moot as it isn't even being used.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com.

[Paul Royik]

My django app makes heavy calculations which can be infinite.

That's why, when user enters the site, i.e. makes a request, heavy calculations are wrapped into multiprocessing.Process which runs at most 7 seconds.

I can't use threads, because third-party packages are not thread-safe.

So, I have around 30 concurrent requests per second. If each request can take up to 7 seconds, then it is 30*7=210 concurrent requests in the worst case.

Each of these concurrent requests opens  multiprocessing.Process, which gives (I guess) 210*2=420 (close to 500) concurrent requests in the worst case.

That' how I got 500 requests. Possibly, my calculations are incorrect.

Average page load time (average response times) is 10 seconds. 

I use MPM worker.

I set WSGIProcessGroup

StartServers 100
ServerLimit 500

ThreadsPerChild 1
ThreadLimit 1

MaxRequestWorkers 500

MaxConnectionsPerChild 10000

WSGIApplicationGroup %{GLOBAL}

WSGIDaemonProcess django_app processes=75 threads=1 python-path='...' maximum-requests=10000 request-timeout=20
WSGIProcessGroup django_app

WSGIRestrictEmbedded On
WSGILazyInitialization On

On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton wrote:

What sort of application are you running?

What is your average response times?

Do you have long running requests, if yes, how long?

What Apache MPM are you actually using?

My initial impression is that is a quite poor configuration which is only going to chew up huge amounts of memory for no good reason, but I don't know your application requirements.

Also, are you even setting WSGIProcessGroup?  If it isn't set it makes the whole daemon process configuration moot as it isn't even being used.

On 10 Aug 2020, at 7:24 pm, Paul Royik <distan...@gmail.com> wrote:

StartServers 50
ServerLimit 200

ThreadsPerChild 1
ThreadLimit 1

MaxRequestWorkers 200
MaxConnectionsPerChild 10000

WSGIApplicationGroup %{GLOBAL}
WSGIDaemonProcess process processes=75 threads=1

Is it enough? Or can it handle only 75 concurrent requests? I don't know how to synchronize apache and mod_wsgi settings. 

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Graham Dumpleton]

Personally I would be concerned about the architecture you are using if you have long running tasks like you describe. It is not usually a good idea to use 'multiprocessing.Process' to create sub processes directly from a web server process to perform work. A better architecture would be to off load the work into a queue using something like Celery and have the separate job processing system pull the jobs from the queue and process them. You would also be better off to model the interaction from the front end as queueing the job and immediately responding with an acknowledgement to say is queued. The front end can then start polling periodically to see if the job has finished, and when it has it would get the response back. The front end can then display the data or save it locally as needed.

This model avoids the problem of requests being parked doing nothing for a long time, which with your server configuration is going to be hugely expensive on memory and not scale very well because of limitations of using WSGI process/threading model. You might even consider not using a WSGI application at all. Instead, use an async web application paired with Celery for execution of the jobs. Using an async web application means you can handle as many parked requests as you want and they can quite happily sit there waiting for Celery to finish the job and don't need to use polling. Only thing am not sure about in that is what async clients there are for Celery.

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com.

[Paul Royik]

You are absolutely correct. Need to change the architecture.

One more question. I also use subprocess.check_output from django. Is it also bad idea? I'm trying to run a script (non-python) and get it output.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com.

[Graham Dumpleton]

If it is a minor quick running script that does something simple it should be okay. It is just having long running processes would be more concerned about.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com.

[Paul Royik]

Thank you for your help.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com.

[Paul Royik]

One more question.

What is a good configuration if I have 500 concurrent requests and there are no long-running tasks?

[Graham Dumpleton]

The problem was you had other odd requirements like having non thread safe code.

All I can do is suggest you watch these videos.

https://www.youtube.com/watch?v=CPz0s1CQsTE&t=4s

https://www.youtube.com/watch?v=SGleKfigMsk&t=2s

https://www.youtube.com/watch?v=k6Erh7oHvns&t=1s

There is no simple answer as one needs to know how your application works and have access to performance metrics from an APM service in order to tune the server.

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com.

[Paul Royik]

Can you at least tell me how apache settings coordinate with mod_wsgi settings?

I set number of threads, processes in Apache and in mod_wsgi. How do they work together?

In mod_wsgi processes=5 threads=12. Does it mean 60 concurrent requests?

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com.

[Graham Dumpleton]

Did you watch the videos? That is covered in some of them.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com.

[Paul Royik]

I'll watch. Thank you!

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com.

[Paul Royik]

So, I watched the video, changed the architecture using Celery.

Now requests are processed very quickly.

Here is my configuration.

StartServers 1
ServerLimit 5
MaxRequestWorkers 125
MaxConnectionsPerChild 10000

WSGIDaemonProcess django_app processes=15 threads=5

As I understand this allows 75 concurrent requests in mod_wsgi and 125 requests in apache.

Is there any reason to make number of concurrent requests in apache higher than in mod_wsgi, i.e. 125 vs. 75? Possibly, set it 75 in both cases? I don't understand this point.

The second question. Things become really faster, but from time to time I got 502 error. It appears that apache is killed, because I need to start the server manually.

What are the possible reasons for this? RAM, too many requests? Can you give me some suggestions?

Thank you.

[Graham Dumpleton]

Setting of MaxRequestWorker (MaxClients) has connections to other MPM settings, so must see those as well.

Also usually would be no need to set MaxConnectionsPerChild either if everything else is setup correctly when using mod_wsgi daemon mode.

Since using mod_wsgi daemon mode there isn't a need to recycle Apache child worker processes, nor restrict the Apache child workers so they are single threaded

Rather than cut and paste piece meal parts of your configuration, I need to see the whole mod_wsgi parts of the configuration, plus all the Apache MPM settings.

So need to see all the settings for the following, depending on whether using worker or event MPM. The preferred these days is the "event" MPM.

<IfModule mpm_worker_module>

    StartServers             3

    MinSpareThreads         75

    MaxSpareThreads        250

    ThreadsPerChild         25

    MaxRequestWorkers      400

    MaxConnectionsPerChild   0

</IfModule>

<IfModule mpm_event_module>

    StartServers             3

    MinSpareThreads         75

    MaxSpareThreads        250

    ThreadsPerChild         25

    MaxRequestWorkers      400

    MaxConnectionsPerChild   0

</IfModule>

Also need to see what you have for:

WSGIDaemonProcess

WSGIProcessGroup

WSGIScriptAlias

WSGIRestrictEmbedded

ThreadStackSize

But I need to see where you stick the WSGI directives in relation to other stuff you have in the VirtualHost to check whether the location is correct and thus whether applied.

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com.

[Paul Royik]

That's all I have in httpd.conf:

ServerRoot "/home/user/webapps/django_app/apache2"

LoadModule alias_module modules/mod_alias.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule dir_module        modules/mod_dir.so
LoadModule env_module        modules/mod_env.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module       modules/mod_mime.so
LoadModule rewrite_module    modules/mod_rewrite.so
LoadModule setenvif_module   modules/mod_setenvif.so
LoadModule wsgi_module       modules/mod_wsgi.so
LoadModule unixd_module      modules/mod_unixd.so
LoadModule deflate_module    modules/mod_deflate.so

RewriteEngine on
RewriteCond %{HTTP_REFERER} motherboard.vice.com [NC]
RewriteRule .* - [F]

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog /home/user/logs/user/access_django_app.log combined
ErrorLog /home/user/logs/user/error_django_app.log

Listen 18136
KeepAlive Off
SetEnvIf X-Forwarded-SSL on HTTPS=1

MaxRequestWorkers 200
MaxConnectionsPerChild 10000

WSGIDaemonProcess django_app processes=15 threads=5 python-path=/home/user/webapps/django_app:/home/user/webapps/django_app/lib/python3.7 maximum-requests=10000 request-timeout=20
WSGIProcessGroup django_app
WSGIRestrictEmbedded On
WSGILazyInitialization On
WSGIScriptAlias / /path_to_wsgi.py

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com.

[Graham Dumpleton]

Is there a reason you discarded the default Apache configuration?

It is important you do not do that as it contains defaults that keep your system more secure. By deleting all the default config, you have lessoned the security of your system.

Also, when you delete the configuration, you don't end up with the same defaults for many values. This is because the compiled in defaults are often not the same as is set in the files. It is therefore going to be far from clear what configuration your system is even running with at this point.

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/09d05cdb-3b84-4720-b8b1-4804a57b9d5co%40googlegroups.com.

[Paul Royik]

I din't delete anything.

This is default Webfaction configuration.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/09d05cdb-3b84-4720-b8b1-4804a57b9d5co%40googlegroups.com.

[Graham Dumpleton]

Webfaction is a quite old hosting service now and was quite restrictive in the resources it provided from what I remember. Not sure it is the best hosting service for a service which has requirements for high capacity.

One thing with Webfaction is that they use a front end nginx proxy in front of Apache. It is quite possible you could be getting rate limit or have caps on the number of concurrent connections you can have there and you would have no control over it.

Anyway, since they unwisely throw away all the configuration, you would be using whatever is the compiled in default for Apache. If that is indeed worker MPM, that would be:

    StartServers             3

    MinSpareThreads         75

    MaxSpareThreads        250

    ThreadsPerChild         25

    MaxRequestWorkers      400

    MaxConnectionsPerChild   0

You have overriden this with:

    MaxRequestWorkers 200

    MaxConnectionsPerChild 10000

Or at least that was the last ones you quoted.

Which means in effect that MaxSpareThreads gets forced to be:

    MaxSpareThreads        200

which has consequences around being able to scale back down number of Apache child worker processes.

Anyway, end result is that Apache starts up 3 initial child worker processes. Each of these has 25 threads for accept handling requests, thus the initial capacity is 75.

When one of the threads in the Apache child worker processes accepts a request which is actually for the WSGI application, it will be proxied across to one of the mod_wsgi daemon processes. That thread is locked to that request until it completes so can't do anything else.

As soon as traffic starts, because the initial capacity is 75, it means you are already at the min spare threads threshold of 75, so almost immediately you are at risk of a 4th Apache child worker process being spawned bumping capacity up to 100.

The number of Apache child worker process will keep growing whenever the spare number of threads available to handle requests drops below 75. It can grow up to a maximum of 8 processes or 200 threads due to your MaxRequestWorkers setting of 200.

Now, since MaxSpareThreads isn't set below 200, the number of processes only ever grows, maxing out at 8. It can never reduce even when the server becomes idle.

So that is what happens on the Apache side.

On the mod_wsgi daemon side you have:

    WSGIDaemonProcess django_app processes=15 threads=5 python-path=/home/user/webapps/django_app:/home/user/webapps/django_app/lib/python3.7 maximum-requests=10000 request-timeout=20

which means capacity is 75, being 15 processes times 5 threads.

The number of mod_wsgi daemon process started initially is 15 and stays fixed, there is no dynamic scaling as for Python that is a and idea because of start up cost of Python processes. Thus application is always kept resident.

As I said before, if an Apache child worker process receives a request, it proxies it to one of the mod_wsgi daemon processes, but that thread in the Apache child worker processes is allocated to that request for the life of the request.

If for some reason you reached capacity in the mod_wsgi daemon processes, ie., 75 concurrent requests, then the accepting thread in the Apache child worker process would block trying to connect to the mod_wsgi daemon process and would only get through when another request finished. So the queuing up of requests happens within the Apache child worker process effectively, or at least until the capacity of those reaches 200, in which case requests start backing up on the client side in the socket listener queue of the operating system.

Overall, the capacity of the Apache child worker processes must be more than that for the mod_wsgi daemon processes. How much more depends on a couple of factors.

The first is how many static file requests are you also serving at the same time from the Apache server.

The second is how much capacity you want for queuing up client requests bound to the WSGI application.

This becomes very fuzzy as what you should use. Especially when the Apache server socket also has a listen queue depth which is quite large (default 500).

If you allow too much backlogging of requests, if the WSGI application bogs down and gets behind it may have trouble every catching up, worse is that after a while it could be handling requests where the user gave up anyway.

For that reason, allow backlogging if you want, but ensure quick recovery from transient overload, you may want to accept users getting errors by setting queue-timeout on WSGIDaemonProcess. That is, if they are left waiting more than that amount of time from when Apache accepted the request and it finally gets to a mod_wsgi daemon process, just error it straight away with a timeout rather than handling it in the WSGI application. This enables you to throw out backlogged requests and recover more quickly. Without such a recovery mechanism, you may never catch up and may have to restart the whole server to get things back to normal.

The socket listen queue depth makes this hard to deal with though as until the request is accepted by the Apache child worker processes you cannot start timing how long it has been in the queue. So if you make the capacity of the Apache child worker processes above that of mod_wsgi daemon processes too small, it will back up in the socket listen queue. This makes the queue timeout meaningless as that may not actually take long to be handled once accepted. So having a reasonable overhead is okay for where you are setting queue timeout. In either case though, if capacity of the Apache server processes themselves reach capacity, it is hard to track backlogging and recover automatically in some way.

So try setting queue-timeout to error requests immediately if that have been waiting too long.

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/fe0e6129-dbfb-434a-acfd-13a8ed690437o%40googlegroups.com.