Correlation between mod_wsgi Daemon Processes and apache mpm_worker

[Manu Itutur]

Hello,

I'm having an hard time understanding the correlation between mod_wsgi Daemon Processes and apache mpm_worker (if there is any).

I'm running a Django application on an Ubuntu 22.04 Apache Server, and I would like to optimise the CPU / RAM usage of it, as it looks like when there is a hight number of user / requests, the CPU usage is limited to around 60%.


My Apache configuration is currently the following:

WSGIDaemonProcess ipt python-path=virtualenv/lib/python3.7/site-packages processes=7 threads=2 display-name=custom-apache
WSGIScriptAlias / conf/wsgi.preprod.py process-group=custom application-group=%{GLOBAL}
WSGIApplicationGroup %{GLOBAL}
<Directory conf>
  <Files wsgi.preprod.py>
        Require all granted
        Order allow,deny
        Allow from all
  </Files>
</Directory>

And my MPM conf file (which are the default settings of Apache2):

<IfModule mpm_prefork_module>
StartServers  5
MinSpareServers   5
MaxSpareServers  10
MaxRequestWorkers   150
MaxConnectionsPerChild   0
</IfModule>


I read multiple time this documentation https://modwsgi.readthedocs.io/en/master/user-guides/processes-and-threading.html#the-mod-wsgi-daemon-processes but I don't realy understand how the mpm_prefork_module and the WSGIDaemonProcess combine with each other and how to tweak the value "safely".

Should I only increase processes and threads at the WSGIDaemonProcess without modifying the mpm_prefork_module? Or are both conf related?

Thanks for your clarification

Emmanuel 

[Graham Dumpleton]

You are missing WSGIProcessGroup directive or process-group option to WSGIScriptAlias, so you aren't even delegating requests to be handled by the mod_wsgi daemon process group. That will be in part why things are not making sense.

For more background go watch:

https://www.youtube.com/watch?v=H6Q3l11fjU0

https://www.youtube.com/watch?v=k6Erh7oHvns

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d278edf0-017d-4a2b-bf14-98f21b21346bn%40googlegroups.com.

[Graham Dumpleton]

BTW the location given in python-path, second argument to WSGIScriptAlias and argument to Directory directive are supposed to be absolute paths, not relative. Use of a relative path might cause incorrect results.

For a virtual environment you should not use python-path like that anyway. Instead use python-home to virtual environment root. See:

On 14 Dec 2023, at 1:45 am, Manu Itutur <iturbide...@gmail.com> wrote:

[Manu Itutur]

Thanks for your answer,

In my file I use absolute path, I remove some "sensitive" data, therefore it's then a relative path, sorry for the confusion. 

I already watched both video couple of days ago, but I guess I missed something. I will check process-group option in  WSGIScriptAlias, hope things will be clarified.

[Graham Dumpleton]

The issue with the process-group option you have is that it doesn't match the name of the process group. You have "custom" instead of "ipt".

To be honest I even missed you had the process-group option in there. Since didn't match you should actually have got an error. So maybe you do have it correct.

Note another video to watch which touches on performance is:

BTW, make sure you have:

    WSGIRestrictEmbedded On

set in global Apache configuration if using daemon process mode.

Graham

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/9973f79c-6265-4cee-a876-262b09975fa4n%40googlegroups.com.

[Manu Itutur]

Hello Graham,

Thanks for your answer. Here is the full apache configuration 

<VirtualHost *:80>

Servername redacted.io

RewriteEngine On

WSGIPassAuthorization on

# Block access for all IPs in /etc/apache2/ipblacklist.conf

RewriteMap hosts-deny "txt:/etc/apache2/ipblacklist.conf"

RewriteCond "${hosts-deny:%{REMOTE_ADDR}|NOT-FOUND}" "!=NOT-FOUND" [OR]

RewriteCond "${hosts-deny:%{REMOTE_HOST}|NOT-FOUND}" "!=NOT-FOUND"

RewriteRule "^" "-" [F]

# Maintenance mode section

# Allow Individual IP stored in /etc/apache2/maintenance.exceptions addresses past maintenance page

RewriteMap exceptions /etc/apache2/maintenance.exceptions

RewriteCond ${exceptions:%{REMOTE_ADDR}} =OK

RewriteRule ^ - [L]

ErrorDocument 503 /static/html/maintenance/under_construction.html

RewriteCond /etc/apache2/MAINTENANCE_ON -f

RewriteCond expr "! %{REQUEST_URI} -strmatch '/static/html/maintenance/*'"

# Allow access for LB IPs

RewriteCond expr "! -R '10.108.0.0/14'"

RewriteRule ^ - [R=503,L]

RewriteCond /etc/apache2/MAINTENANCE_ON !-f

RewriteRule ^/static/html/under_construction.html$ / [R,L]

# BLOG Conf - Test

ProxyPreserveHost On

ProxyRequests Off

Alias /503 /home/redacted/redacted/redacted/static/html/under_construction.html

<Location /mag>

ProxyPass http://192.168.1.225/mag

ProxyPassReverse http://192.168.1.225/mag

Order allow,deny

Allow from all

ErrorDocument 503 /503

</Location>

<Location /mag/es>

ProxyPass http://192.168.1.186/mag/es

ProxyPassReverse http://192.168.1.186/mag/es

Order allow,deny

Allow from all

ErrorDocument 503 /503

</Location>

<Location ~ "/admin|/staff|/admin_action|/mailator">

Order deny,allow

Include /etc/apache2/ipwhitelist.conf

Deny from all

ErrorDocument 503 /503

</Location>

Alias /robots.txt /home/redacted/redacted/redacted/static/robots.txt

Alias /.well-known/security.txt /home/redacted/redacted/redacted/static/security.txt

Alias /favicon.ico /home/redacted/redacted/redacted/static/favicon.ico

Alias /static/ /home/redacted/redacted/redacted/static/

<Directory /home/redacted/redacted/redacted/static>

Options Includes FollowSymLinks MultiViews

Order allow,deny

Require all granted

Allow from all

</Directory>

WSGIDaemonProcess redacted python-path=/home/redacted/redacted:/home/redacted/redacted/virtualenv/lib/python3.7/site-packages processes=8 threads=2 display-name=redacted-apache

WSGIScrredactedAlias / /home/redacted/redacted/redacted/conf/wsgi.prod.py process-group=redacted application-group=%{GLOBAL}

WSGIApplicationGroup %{GLOBAL}

<Directory /home/redacted/redacted/redacted/conf>

<Files wsgi.prod.py>

Require all granted

Order allow,deny

Allow from all

</Files>

</Directory>

</VirtualHost>

<VirtualHost *:80>

ServerName cms.redacted.io

Redirect 301 / https://redacted.io/

</VirtualHost>

<VirtualHost *:80>

ServerName get.redacted.io

Redirect 301 / https://redacted.io/

</VirtualHost>

<VirtualHost *:80>

Servername help.redacted.io

ErrorLog "/var/log/apache2/help_error_log.log"

CustomLog "/var/log/apache2/help_access_log.log" common

SSLProxyEngine on

SSLProxyVerifyDepth 10

SSLProxyCheckPeerCN off

SSLProxyCheckPeerName off

ProxyPreserveHost On

ProxyPass / https://custom.intercom.help/

ProxyPassReverse / https://custom.intercom.help/

</VirtualHost>

On my serveur, from the monitoring perspective, the maximum workers is reach at 260, while the CPU / RAM usage remains pretty low.

My mpm_prefork conf is:

<IfModule mpm_prefork_module>
StartServers      20
MinSpareServers  15
MaxSpareServers  30

MaxRequestWorkers  150
MaxConnectionsPerChild   0
</IfModule>

So as the max workers does not seems to be use from mpm... I'm really confuse, I don't understand how both file are related, if they are. 

Thanks for your help.

Manu

[Manu Itutur]

Any chance to have some help on this ? 

Thanks in advance. 

Manu 

[Graham Dumpleton]

Not sure what you are asking for at this point.

The workers talked about for the MPM are the Apache child worker processes, which when using mod_wsgi daemon mode only serve as proxies for proxying requests through to the mod_wsgi daemon processes. This is explained in the videos I linked.

It would be quite normal when using mod_wsgi daemon mode that the number of Apache child worker processes wouldn't be spun up to the max.

As I probably said before or in the videos, using prefork MPM when using mod_wsgi daemon mode, if nothing else is running in Apache besides the mod_wsgi application, is not a good idea. You are better off using worker or event MPM. Only makes sense to use prefork MPM if you must run PHP in the same Apache instance.

Can you be clearer about what you problem is?

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/bba63b87-65ef-4699-b41b-fddb20084649n%40googlegroups.com.

[Manu Itutur]

Hello,

Thanks for your answer. My problem is that, for time to time, my Django application receive a lot of unwanted trafic from automatic scans. For exemple yesterday, we received a very high number of requests, the MaxRequestsWorkers set to 256 is reach, the Apache serveur stuck and legitimate traffic is blocked, blocking our client from using our App, while the CPU / Memory usage remains pretty low. 

My MPM prefork config was the following. 

<IfModule mpm_prefork_module>
StartServers      20
MinSpareServers  15
MaxSpareServers  30

MaxRequestWorkers  256
MaxConnectionsPerChild   0
</IfModule>

It seems that the MaxRequestWorkers is reach. In order to increase it after reading the Apache documentation, I have to set ServerLimit as well, which I did. My new configuration now is: 

<IfModule mpm_prefork_module>
StartServers      20
MinSpareServers  20
MaxSpareServers  30
MaxRequestWorkers  500
MaxConnectionsPerChild   0
ServerLimit          500
</IfModule>

(I adjusted as well StartServers / MinSpareServers based on your feedback on the video "Making Apache suck less for hosting Python web applications." 

With this configuration, I should be able to handle more trafic, and I will adjust the MaxRequestWorkers / ServerLimit if needed. 

My question is, what is the correlation between the WSGIDaemonProcess options "processes" and "threads"  and the MPM configuration.

What would be the impact of increasing these options without modifying the MPM prefork config? 

Regards 

Manu 

[Graham Dumpleton]

Try disabling keep alive connections.

https://httpd.apache.org/docs/2.4/mod/core.html#keepalivetimeout

Ensure you have a robots.txt file being served up to block robots such as search engines if they should not be allowed to even index your site.

Make sure you don't have the Apache Timeout directive set to some ridiculously high value.

https://httpd.apache.org/docs/2.4/mod/core.html#timeout

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/7b024a99-4500-49a6-a4c5-9b042b9a843bn%40googlegroups.com.