Performance impact when number of wsgi apps increases?

[Johan De Taeye]

HI All,

I have a multi-tenant apache server where each tenant has its own wsgi application in its own virtual host.
The apache configuration for a tenant roughly looks like:

<VirtualHost *:443>

ServerName domain-tenant-1

WSGIDaemonProcess tenant1 user=www-data processes=4 threads=4 inactivity-timeout=7200 request-timeout=3600 display-name=apache-tenant1

WSGIProcessGroup tenant1 

WSGIScriptAlias / "/home/ubuntu/config/tenant1/wsgi.py"

</VirtualHost>

I am noticing detoriating response times when the number of tenants increases beyond a certain limit - say 150.   Neither CPU, database and memory seem to be bottleneck.

Are there any apache or mod_wsgi configurations/parameters that can be tuned/optimized for this type of deployment? 

All feedback & insight is highly appreciated!

Johan

[Graham Dumpleton]

How much traffic does each site receive?

What sort of traffic do they receive? Are they short lived requests less than 500ms, or long lived requests that could last seconds?

What Apache MPM are you using (prefork/worker/event) and what are the Apache MPM settings for that MPM?

For 150 sites I would suggest you are adding too much capacity for each. Even one process with 5 threads can handle a lot of concurrent requests if requests are short lived. If you are trying to support sites which all have long running requests for file uploads etc, you are asking for trouble. The request timeout is ridiculously large, which suggests you might be trying to handle long running file uploads, which as I said is a bad idea.

Anyway, to get started I suggest you watch:

https://www.youtube.com/watch?v=H6Q3l11fjU0

Then do the following to at least cut down on CPU and memory.

1. Outside of all VirtualHosts add:

  WSGIRestrictEmbedded On

This will ensure that the Python interpreter isn't setup in Apache child worker processes where not needed if only using mod_wsgi daemon mode.

2. In each VirtualHost add:

  WSGIApplicationGroup %{GLOBAL}

This ensures the Python application runs in the main interpreter context and not a sub interpreter. Various Python packages do not work properly in sub interpreters. Ensuring main interpreter context is used will also save a bit of memory as well since don't have the main interpreter context there doing nothing when using a sub interpreter.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/577e778b-87a5-47dd-9dd3-c7a8d40fdf73n%40googlegroups.com.

[Johan De Taeye]

Hi Graham,

Thanks already for your pearls of wsgi-wisdom!

>> How much traffic does each site receive?

A lot of these will stay inactive for a long time.   At a single time, only at most 5 to 10 of the apps will have active users.   

Ideally I'd like to launch the mod_wsgi daemon process only upon receiving the first request in an app. I managed to avoid the preload but not the daemon process itself.     

>> What sort of traffic do they receive? Are they short lived requests less than 500ms, or long lived requests that could last seconds?

On an active application, the request can last seconds indeed (due to complex sql-queries, complex data validation, ...)  

And an active user can easily generate a series of such requests. Hence the 

>> What Apache MPM are you using (prefork/worker/event) and what are the Apache MPM settings for that MPM?

We're using event MPM (which is the default on our ubuntu 18 server).   

The settings are currently:

        ServerLimit              90
        StartServers             2
        MinSpareThreads          30
        MaxSpareThreads          60
        ThreadLimit              30
        ThreadsPerChild          30
        MaxRequestWorkers        800
        MaxConnectionsPerChild   0

We're using the mod_wsgi version 4.5 packaged with ubuntu 18. After listening to your talk, I'll try out with the latest 4.9 version.

>>For 150 sites I would suggest you are adding too much capacity for each. Even one process with 5 threads can handle a lot of concurrent requests if requests are short lived. If you are trying to support sites which all have long running requests for file uploads etc, you are asking for trouble. The request timeout is ridiculously large, which suggests you might be trying to handle long running file uploads, which as I said is a bad idea.

I hear you... and will reduce that request timeout.

>>    WSGIRestrictEmbedded On

Already have that one indeed.

>> WSGIApplicationGroup %{GLOBAL}

>> This ensures the Python application runs in the main interpreter context and not a sub interpreter. Various Python packages do not work properly in sub interpreters. Ensuring main interpreter context is used will also save a bit of memory as well since don't have the main interpreter context there doing nothing when using a sub interpreter.

Don't have that one yet. 

For security reasons, I'd like to keep the python processes for each tenant 100% separated.   Does this option violate this?

Johan

Op zondag 16 januari 2022 om 22:42:42 UTC+1 schreef Graham Dumpleton:

[Graham Dumpleton]

On 19 Jan 2022, at 5:03 am, Johan De Taeye <johan.d...@gmail.com> wrote:

>> How much traffic does each site receive?

A lot of these will stay inactive for a long time.   At a single time, only at most 5 to 10 of the apps will have active users.   

Ideally I'd like to launch the mod_wsgi daemon process only upon receiving the first request in an app. I managed to avoid the preload but not the daemon process itself.     

There is no lazy starting of daemon processes themselves. Apache makes that too hard.

>> What sort of traffic do they receive? Are they short lived requests less than 500ms, or long lived requests that could last seconds?

On an active application, the request can last seconds indeed (due to complex sql-queries, complex data validation, ...) 

How much CPU works is done in the process as compare to waiting on backend database.

If most of the time is waiting on database, there should be no significant drawback from using:

   processes=1 threads=15

rather than:

  processes=4 threads=4

With the number of sites you have, keeping the number of seperate processes down is better. Will also reduce amount of memory used.

And an active user can easily generate a series of such requests. Hence the 

>> What Apache MPM are you using (prefork/worker/event) and what are the Apache MPM settings for that MPM?

We're using event MPM (which is the default on our ubuntu 18 server).   

The settings are currently:

        ServerLimit              90

        StartServers             2
        MinSpareThreads          30
        MaxSpareThreads          60
        ThreadLimit              30
        ThreadsPerChild          30
        MaxRequestWorkers        800
        MaxConnectionsPerChild   0

MaxRequestWorkers is usually a multiple of ThreadsPerChild. The maximum number of Apache child processes it can scale up to is MaxRequestWorkers / ThreadsPerChild thus why it is usually a multiple. 

We're using the mod_wsgi version 4.5 packaged with ubuntu 18. After listening to your talk, I'll try out with the latest 4.9 version.

>>For 150 sites I would suggest you are adding too much capacity for each. Even one process with 5 threads can handle a lot of concurrent requests if requests are short lived. If you are trying to support sites which all have long running requests for file uploads etc, you are asking for trouble. The request timeout is ridiculously large, which suggests you might be trying to handle long running file uploads, which as I said is a bad idea.

I hear you... and will reduce that request timeout.

Inactivity timeout could also be reduced if memory is an issue. Rather than 7200 (2 hours), 900 (15 minutes) may be more reasonable. Depends on how long time between batches of requests. 

>>    WSGIRestrictEmbedded On

Already have that one indeed.

>> WSGIApplicationGroup %{GLOBAL}

>> This ensures the Python application runs in the main interpreter context and not a sub interpreter. Various Python packages do not work properly in sub interpreters. Ensuring main interpreter context is used will also save a bit of memory as well since don't have the main interpreter context there doing nothing when using a sub interpreter.

Don't have that one yet. 

For security reasons, I'd like to keep the python processes for each tenant 100% separated.   Does this option violate this?

So long as each virtual host WSGI site uses a separate daemon process group then using WSGIApplicationGroup with that value is fine.

To make it clearer and avoid mistakes, might be better to use:

<VirtualHost *:443>
ServerName domain-tenant-1
WSGIDaemonProcess tenant1 user=www-data processes=4 threads=4 inactivity-timeout=7200 request-timeout=3600 display-name=apache-tenant1

WSGIScriptAlias / "/home/ubuntu/config/tenant1/wsgi.py" process-group= tenant1

WSGIApplicationGroup %{GLOBAL}
</VirtualHost>

In other words, set process group on WSGIScriptAlias.

I didn't set application group in same way as using option to WSGIScriptAlias because if both process group and application group are specified using options to that, it triggers script preloading which result in application reloading as soon as process is restarted due to inactivity timeout.

Graham