Apache (prefork MPM) and mod_wsgi are installed on a small VPS
(serving approx. 1000 pages/day), and mod_wsgi is used in daemon mode
to run Django applications. I tried tinkering with both the MPM
parameters
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 50
MaxRequestsPerChild 1000
and the WSGIDaemonProcess options
WSGIDaemonProcess djangoproject user=nobody group=nobody threads=10
maximum-requests=1000 deadlock-timeout=30 inactivity-timeout=100 stack-
size=524288
Switching to mod_wsgi (from mod_python) eliminated the frequent black
zone QoS alerts (othersockbuf) I experienced before. But my server
still reports hitting the yellow zone (approaching the soft limit for
numproc) a couple of times a day (and even the black zone
occasionally). I wonder if I can avoid this without impairing the
responsiveness of the server too much.
As far as I understand, the configuration above allows Apache to
create a maximum of 50 child processes (servers), each of which can
create 10 mod_wsgi threads. Based on this, I assume I could easily
avoid the numproc issue by reducing the value for MaxClients and or
threads. If I reduce the number of MaxClients too much – will the more
frequent creation of child processes affect the memory usage
adversely?
Idle threads are killed after 100 seconds of inactivity, and they are
limited to a stack size of 524288 bytes. Each thread can serve up to
1000 requests, while the child process itself is also limited to 1000
requests (so this configuration probably does not make sense).
Are my assumptions correct, and is there anything else about this
setup which could/should be optimizied? Are there other parameters I
should take into account?
Kind regards,
Jan
Do you have a corresponding WSGIProcessGroup directive for the
WSGIDaemonProcess, or tagging WSGIScriptAlias with process-group
option?
If you don't, you will be running in embedded mode, which isn't what you want.
Also, are you running PHP in the same server? If not, do not use
prefork MPM, use worker MPM instead. If you can, get rid of mod_php if
you are using it, using mod_fcgid for PHP instead.
I will explain everything properly later when have a chance.
In the mean time, please post all parts of the Apache configuration
showing how you are setting up mod_wsgi for your application so can
verify it is set up correctly.
> Switching to mod_wsgi (from mod_python) eliminated the frequent black
> zone QoS alerts (othersockbuf) I experienced before. But my server
> still reports hitting the yellow zone (approaching the soft limit for
> numproc) a couple of times a day (and even the black zone
> occasionally). I wonder if I can avoid this without impairing the
> responsiveness of the server too much.
>
> As far as I understand, the configuration above allows Apache to
> create a maximum of 50 child processes (servers), each of which can
> create 10 mod_wsgi threads. Based on this, I assume I could easily
> avoid the numproc issue by reducing the value for MaxClients and or
> threads. If I reduce the number of MaxClients too much – will the more
> frequent creation of child processes affect the memory usage
> adversely?
>
> Idle threads are killed after 100 seconds of inactivity, and they are
> limited to a stack size of 524288 bytes. Each thread can serve up to
> 1000 requests, while the child process itself is also limited to 1000
> requests (so this configuration probably does not make sense).
>
> Are my assumptions correct, and is there anything else about this
> setup which could/should be optimizied? Are there other parameters I
> should take into account?
Graham
Yes, I do. This is my complete configuration (MPM- and mod_wsgi-
related):
<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 50
MaxRequestsPerChild 1000
</IfModule>
WSGISocketPrefix /var/run/wsgi
WSGIRestrictEmbedded On
<VirtualHost 87.106.49.95:80>
ServerName janeden.net
AliasMatch ^/(css|images|files|admin_media)(.*) /var/www/
djangoproject/static/$1/$2
<Directory /var/www/djangoproject/static>
Order deny,allow
Allow from all
</Directory>
WSGIScriptAlias / /var/www/djangoproject/apache/django.wsgi
WSGIDaemonProcess djangoproject user=apache group=apache threads=10
stack-size=524288 maximum-requests=3000 deadlock-timeout=30 inactivity-
timeout=100
WSGIProcessGroup djangoproject
<Directory /var/www/djangoproject>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
There are actually three VirtualHosts like the one above, but they are
defined almost identically (the only difference being the ServerName,
WSGIDaemonProcess name, WSGIProcessGroup name and directories).
> Also, are you running PHP in the same server? If not, do not use
> prefork MPM, use worker MPM instead. If you can, get rid of mod_php if
> you are using it, using mod_fcgid for PHP instead.
I don't use PHP, so I can switch to the worker MPM. Is there anything
in the above configuration (or in other areas of httpd.conf) which
should be reviewed prior to switching the MPM? The default worker
configuration currently looks like this:
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
> I will explain everything properly later when have a chance.
>
> In the mean time, please post all parts of the Apache configuration
> showing how you are setting up mod_wsgi for your application so can
> verify it is set up correctly.
Thanks a lot for your help! This is by far the most responsive and
professional 'product support' I have seen for an open-source package
so far.
- Jan
Hmmm, that is rather amusing considering how slack I have been of late
in replying in a prompt matter. Haven't had much time of late so have
been cherry picking questions which I can answer quickly and leaving
those questions or parts of questions which need a more thorough
response to when I can find some time, which isn't too often.
Graham
Cannot allocate memory: apr_thread_create: unable to create worker
thread
- Jan
On Jan 29, 10:12 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> 2010/1/29 janedenone <janeden...@googlemail.com>:
>
So, send the full messages and see if Apache works without mod_wsgi
but with worker MPM just for static files.
Graham
2010/1/30 janedenone <janed...@googlemail.com>:
> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>
Also might be an idea to say whether you are installing binary
packages for Apache and mod_wsgi or whether compiling from source code
and what versions of Apache, APR and mod_wsgi.
Graham
2010/1/30 Graham Dumpleton <graham.d...@gmail.com>:
On Jan 30, 3:14 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> BTW, what operating system is this? Don't think you have said what it was.
Sorry: CentOS 5.2
> Also might be an idea to say whether you are installing binary
> packages for Apache and mod_wsgi or whether compiling from source code
> and what versions of Apache, APR and mod_wsgi.
Apache 2.2.3/APR 1.2.7, mod_wsgi 3.1.
> > Can you send the complete messages from before and after that. That
> > message doesn't look like anything mod_wsgi produces which means you
> > likely have some combination of Apache and APR libraries which is
> > incompatible and is thus breaking things even for normal Apache static
> > file serving. You could check that by simply disabling mod_wsgi and
> > trying to restart Apache and see if problem still persists.
I suspected a general memory issue first, so I changed the default
worker setup to (following http://blog.zakame.net/tips/apache2-worker-lowmem):
<IfModule mpm_worker_module>
MaxClients 10
MinSpareThreads 3
MaxSpareThreads 7
ThreadsPerChild 3
MaxRequestsPerChild 200
</IfModule>
With mod_wsgi and LogLevel debug, I get:
[Sat Jan 30 08:58:06 2010] [debug] proxy_util.c(1854): proxy: grabbed
scoreboard slot 0 in child 5674 for worker proxy:reverse
[Sat Jan 30 08:58:06 2010] [debug] proxy_util.c(1950): proxy:
initialized worker 0 in child 5674 for (*) min=0 max=25 smax=25
[Sat Jan 30 08:58:06 2010] [notice] Apache/2.2.3 (CentOS) configured
-- resuming normal operations
[Sat Jan 30 08:58:06 2010] [info] Server built: Nov 12 2009 18:46:18
[Sat Jan 30 08:58:06 2010] [debug] worker.c(1740): AcceptMutex:
sysvsem (default: sysvsem)
[Sat Jan 30 08:58:06 2010] [alert] (12)Cannot allocate memory:
apr_thread_create: unable to create worker thread
[Sat Jan 30 08:58:08 2010] [alert] No active workers found... Apache
is exiting!
Without mod_wsgi:
[Sat Jan 30 08:51:40 2010] [debug] proxy_util.c(1854): proxy: grabbed
scoreboard slot 0 in child 26443 for worker proxy:reverse
[Sat Jan 30 08:51:40 2010] [debug] proxy_util.c(1950): proxy:
initialized worker 0 in child 26443 for (*) min=0 max=25 smax=25
[Sat Jan 30 08:51:40 2010] [notice] Apache/2.2.3 (CentOS) configured
-- resuming normal operations
[Sat Jan 30 08:51:40 2010] [info] Server built: Nov 12 2009 18:46:18
[Sat Jan 30 08:51:40 2010] [debug] worker.c(1740): AcceptMutex:
sysvsem (default: sysvsem)
I should add that my VPS only has 256 MB RAM guaranteed (1024 MB
dynamic). Maybe it's pointless to try to optimize with so little RAM?
On the other hand, the machine manages to serve all three sites (via
Django) with prefork MPM.
- Jan
Are though these being installed from binary RPM packages or all
compiled from source code?
If I had to guess, the problem may be that you are using a binary RPM
package for mod_wsgi which is compiled against a newer version of
Apache and APR libraries than what you are running.
Although there is some measure of guaranteed compatibility if using an
Apache module compiled against older Apache than what it is being used
with, the reverse isn't necessarily true.
If you are compiling everything from source code and even recompiled
mod_wsgi from source code when you switched from prefork to worker MPM
and still had the problem, then I would be worried. Although should
point out that mod_wsgi module doesn't use anything which is MPM
specific, so technically even if compiled against prefork MPM, same
.so should work fine with worker MPM. This presumes though that a
specific Linux distribution isn't using quite radically different
compile time options for Apache for different MPMs such that modules
not dependent on the MPM cant be used against either.
For a freshly started Apache 256 MB is more than enough. Have seen
people run Apache/mod_wsgi and Django on a 64MB VM on SliceHost. The
trick there is using nginx proxy front end to handle static files,
worker MPM in Apache and mod_wsgi daemon mode to get tight control on
number of fat Python web processes and thread counts.
Graham
Apache/APR were provided with CentOS, mod_wsgi was compiled from
source.
> If I had to guess, the problem may be that you are using a binary RPM
> package for mod_wsgi which is compiled against a newer version of
> Apache and APR libraries than what you are running.
>
> Although there is some measure of guaranteed compatibility if using an
> Apache module compiled against older Apache than what it is being used
> with, the reverse isn't necessarily true.
>
> If you are compiling everything from source code and even recompiled
> mod_wsgi from source code when you switched from prefork to worker MPM
> and still had the problem, then I would be worried.
Just did that: Switched to worker MPM and recompiled/installed
mod_wsgi – same result.
> > I should add that my VPS only has 256 MB RAM guaranteed (1024 MB
> > dynamic). Maybe it's pointless to try to optimize with so little RAM?
> > On the other hand, the machine manages to serve all three sites (via
> > Django) with prefork MPM.
>
> For a freshly started Apache 256 MB is more than enough. Have seen
> people run Apache/mod_wsgi and Django on a 64MB VM on SliceHost. The
> trick there is using nginx proxy front end to handle static files,
> worker MPM in Apache and mod_wsgi daemon mode to get tight control on
> number of fat Python web processes and thread counts.
Ok. Is there anything in my Apache worker configuration, or in the
mod_wsgi daemon config which could be optimized? If the worker MPM
should use less memory compared to prefork in theory, I must be doing
something wrong.
Thanks again for all your help,
Jan
Can you provide output for:
egrep -h '^ *LoadModule' /etc/httpd/conf/httpd.conf /etc/httpd/conf.d/*.conf
I had no issue with switching to use worker MPM for precompiled mod_wsgi rpm.
I wish to compare what other Apache modules you have loaded to basic
set as well as to what other person with CentOS issues is using.
My CentOS installation is virgin install from 5.4 DVD with only
addition being mod_wsgi.
Graham
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule status_module modules/mod_status.so
LoadModule info_module modules/mod_info.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule wsgi_module modules/mod_wsgi.so
With this configuration, Apache is able to create threads for two of
my Django applications, but not for the third:
[Wed Feb 03 11:23:32 2010] [alert] (12)Cannot allocate memory:
mod_wsgi (pid=13314): Couldn't create reaper thread in daemon process
'djangoanke'.
[Wed Feb 03 11:23:33 2010] [alert] (12)Cannot allocate memory:
mod_wsgi (pid=13401): Couldn't create worker thread 5 in daemon
process 'djangoanke'.
[Wed Feb 03 11:23:39 2010] [alert] (12)Cannot allocate memory:
mod_wsgi (pid=13429): Couldn't create worker thread 5 in daemon
process 'djangoanke'.
[Wed Feb 03 11:23:44 2010] [alert] (12)Cannot allocate memory:
mod_wsgi (pid=13429): Couldn't create worker thread 7 in daemon
process 'djangoanke'.
[Wed Feb 03 11:23:49 2010] [alert] (12)Cannot allocate memory:
mod_wsgi (pid=13429): Couldn't create worker thread 9 in daemon
process 'djangoanke'.
[Wed Feb 03 11:23:55 2010] [alert] (12)Cannot allocate memory:
mod_wsgi (pid=13469): Couldn't create worker thread 5 in daemon
process 'djangoanke'.
[Wed Feb 03 11:24:01 2010] [alert] (12)Cannot allocate memory:
mod_wsgi (pid=13517): Couldn't create worker thread 5 in daemon
process 'djangoanke'.
So it does look like a memory issue which affects the worker MPM more
than the prefork MPM (with which I can run all three apps, although
with frequent QoS alerts).
Kind regards,
Jan
On Feb 2, 2:59 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
<IfModule worker.c>
StartServers 1
MaxClients 10
MinSpareThreads 1
MaxSpareThreads 4
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
to this
<IfModule worker.c>
StartServers 1
MaxClients 9
MinSpareThreads 3
MaxSpareThreads 9
ThreadsPerChild 3
MaxRequestsPerChild 0
</IfModule>
and Apache starts with worker MPM and mod_wsgi. Now I have a
working configuration for three apps set up like this:
WSGIDaemonProcess djangolit user=apache group=apache threads=10
stack-
size=524288 maximum-requests=3000 deadlock-timeout=30 inactivity-
timeout=100
WSGIProcessGroup djangolit
Unfortunately, there are even more QoS alerts than before (with
prefork MPM). How do the process/thread related directives of the
Apache configuration to the process/thread related option of
WSGIDaemonProcess?
Kind regards,
Jan
I'm having similar issues, but unfortunately, this thread hasn't helped me sort them out.I am trying to run moinmoin wiki with mod_wsgi, but when I attempt to do so, my apache processes just start eating all the available memory (4 x 128Mb), and the error log shows
(12)Cannot allocate memory: mod_wsgi (pid=13356): Couldn't create worker thread 7 in daemon process 'moin'.
Top tells me12257 apache 20 0 123m 4748 2000 S 0.0 0.9 0:00.00 httpd12130 apache 23 0 133m 4628 1644 S 0.0 0.9 0:00.01 httpd12128 root 18 0 12556 4392 2396 S 0.0 0.8 0:00.07 httpd12131 apache 24 0 133m 4168 1436 S 0.0 0.8 0:00.00 httpd12132 apache 25 0 133m 4164 1432 S 0.0 0.8 0:00.00 httpdI don't know if that's just what I should expect of mod_wsgi (big memory footprint) or if something's going wrong. I installed mod_wsgi 3.4 and moinmoin 1.9.5 from the latest sources.
I'd be very grateful for any help I can get on this matter.ArthurMy system info follows underneath. I've tried to be thorough so as to avoid back and forth...[root@longcat arthurb]# pythonPython 2.4.3 (#1, Jun 18 2012, 08:55:31)[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2[root@longcat arthurb]# uname -aLinux longcat 2.6.18-194.26.1.el5.028stab079.2 #1 SMP Fri Dec 17 19:25:15 MSK 2010 i686 i686 i386 GNU/Linux[root@longcat arthurb]# cat /proc/meminfoMemTotal: 524288 kBMemFree: 498624 kB...[root@longcat arthurb]# cat /selinux/enforce0Modules[root@longcat arthurb]# grep "LoadModule" /etc/httpd/conf/httpd.conf# have to place corresponding `LoadModule' lines at this location so the# LoadModule foo_module modules/mod_foo.so
LoadModule auth_basic_module modules/mod_auth_basic.soLoadModule auth_digest_module modules/mod_auth_digest.soLoadModule authn_file_module modules/mod_authn_file.soLoadModule authn_alias_module modules/mod_authn_alias.soLoadModule authn_anon_module modules/mod_authn_anon.soLoadModule authn_dbm_module modules/mod_authn_dbm.soLoadModule authn_default_module modules/mod_authn_default.soLoadModule authz_host_module modules/mod_authz_host.soLoadModule authz_user_module modules/mod_authz_user.soLoadModule authz_owner_module modules/mod_authz_owner.soLoadModule authz_groupfile_module modules/mod_authz_groupfile.soLoadModule authz_dbm_module modules/mod_authz_dbm.soLoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.soLoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.soLoadModule log_config_module modules/mod_log_config.soLoadModule logio_module modules/mod_logio.soLoadModule env_module modules/mod_env.soLoadModule ext_filter_module modules/mod_ext_filter.soLoadModule mime_magic_module modules/mod_mime_magic.soLoadModule expires_module modules/mod_expires.soLoadModule deflate_module modules/mod_deflate.soLoadModule headers_module modules/mod_headers.soLoadModule usertrack_module modules/mod_usertrack.soLoadModule setenvif_module modules/mod_setenvif.soLoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.soLoadModule status_module modules/mod_status.soLoadModule autoindex_module modules/mod_autoindex.soLoadModule info_module modules/mod_info.soLoadModule dav_fs_module modules/mod_dav_fs.soLoadModule vhost_alias_module modules/mod_vhost_alias.soLoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.soLoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.soLoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.soLoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.soLoadModule proxy_balancer_module modules/mod_proxy_balancer.soLoadModule proxy_ftp_module modules/mod_proxy_ftp.soLoadModule proxy_http_module modules/mod_proxy_http.soLoadModule proxy_connect_module modules/mod_proxy_connect.soLoadModule cache_module modules/mod_cache.soLoadModule suexec_module modules/mod_suexec.soLoadModule disk_cache_module modules/mod_disk_cache.soLoadModule file_cache_module modules/mod_file_cache.soLoadModule mem_cache_module modules/mod_mem_cache.soLoadModule cgi_module modules/mod_cgi.soLoadModule version_module modules/mod_version.somod_wsgi is loaded in a sub directory[root@longcat arthurb]# grep "LoadModule" /etc/httpd/conf.d/wsgi.confLoadModule wsgi_module modules/mod_wsgi.soI call mod_wsgi like this in my configWSGIScriptAlias /thewiki /usr/local/share/moin/moin.wsgiWSGIDaemonProcess moin user=apache group=apache processes=5 threads=10 maximum-requests=1000 umask=0007WSGIProcessGroup moinWSGIRestrictEmbedded OnWSGISocketPrefix /var/run/moin-wsgi
This is swtune.conf, which I've tried to tweak according to this thread
<IfModule prefork.c>StartServers 1MinSpareServers 1MaxSpareServers 5
ServerLimit 10MaxClients 10MaxRequestsPerChild 4000</IfModule>
--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To view this discussion on the web visit https://groups.google.com/d/msg/modwsgi/-/lDLZOJqVHEAJ.