Apache stopped working after error "Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting"

[Leonardo Almeida]

Hi.

I have been working with Apache for more than 8 years, but now I'm facing a severe problem and I couldn't find a solution on internet so far.

I'm running Apache 2.2.15 on a Windows 2008 R2.

Since the installation many months ago, everything was running correctly, but now my Apache stopped working and I don't know how to fix this problem.

It has been many weeks I don't change any configuration, but suddenly the Apache stopped serving the HTTP responses. All requests get timed out. In a practical sense, my site is offline.


By using Windows Event Viewer, I have found an error log when Apache stopped working. It's the error event:

---------------------------------------------------------------------------
Log Name: Application
Source: Application Error
Date: 6/28/2016 8:08:33 PM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: PictureToPeople
Description:
Faulting application name: httpd.exe, version: 2.2.15.0, time stamp: 0x4b8fed95
Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58
Exception code: 0xc0000005
Fault offset: 0x00037188
Faulting process id: 0xbbc
Faulting application start time: 0x01d1cdfef0f8148c
Faulting application path: C:\Apache2.2\bin\httpd.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: 3c1e0a43-3d85-11e6-aad3-001e8cf18221
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-06-28T23:08:33.000000000Z" />
<EventRecordID>693</EventRecordID>
<Channel>Application</Channel>
<Computer>PictureToPeople</Computer>
<Security />
</System>
<EventData>
<Data>httpd.exe</Data>
<Data>2.2.15.0</Data>
<Data>4b8fed95</Data>
<Data>ntdll.dll</Data>
<Data>6.1.7601.17514</Data>
<Data>4ce7ba58</Data>
<Data>c0000005</Data>
<Data>00037188</Data>
<Data>bbc</Data>
<Data>01d1cdfef0f8148c</Data>
<Data>C:\Apache2.2\bin\httpd.exe</Data>
<Data>C:\Windows\SysWOW64\ntdll.dll</Data>
<Data>3c1e0a43-3d85-11e6-aad3-001e8cf18221</Data>
</EventData>
</Event>
---------------------------------------------------------------------------



It's what Apache error log recorded near to the time of error above (6/28/2016 8:08:33). It's possible to see the
error message "Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting".
After this message, Apache never worked again.
(Previous Apache error log messages were recorded very earlier and are not related.)

---------------------------------------------------------------------------
[Tue Jun 28 20:08:30 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 20:08:31 2016] [notice] Child 3004: Exit event signaled. Child process is ending.
[Tue Jun 28 20:08:32 2016] [info] Child 3004: Accept thread exiting.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Released the start mutex
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Terminating 99 threads that failed to exit.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: All worker threads have exited.
[Tue Jun 28 20:08:32 2016] [info] [client 1.23.96.161] (OS 10038)An operation was attempted on something that is not a socket. : core_output_filter: writing data to the network
[Tue Jun 28 20:08:34 2016] [notice] Parent: Child process exited successfully.
[Tue Jun 28 20:08:34 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=284)
[Tue Jun 28 20:08:54 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 20:08:54 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 20:08:54 2016] [notice] Parent: Created child process 2468
[Tue Jun 28 20:08:54 2016] [info] Parent: Duplicating socket 308 and sending it to child process 2468
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Child process is running
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Acquired the start mutex.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting 100 worker threads.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting thread to listen on port 80.
[Tue Jun 28 20:08:56 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
[Tue Jun 28 20:12:29 2016] [warn] pid file C:/Apache2.2/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Tue Jun 28 20:12:29 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 20:12:29 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 20:12:29 2016] [notice] Parent: Created child process 1236
[Tue Jun 28 20:12:29 2016] [info] Parent: Duplicating socket 308 and sending it to child process 1236
[Tue Jun 28 20:12:29 2016] [notice] Child 1236: Child process is running
[Tue Jun 28 20:12:29 2016] [notice] Child 1236: Acquired the start mutex.
[Tue Jun 28 20:12:29 2016] [notice] Child 1236: Starting 100 worker threads.
[Tue Jun 28 20:12:29 2016] [notice] Child 1236: Starting thread to listen on port 80.
[Tue Jun 28 20:12:30 2016] [info] [client 101.182.34.93] (OS 10054)An existing connection was forcibly closed by the remote host. : core_output_filter: writing data to the network
[Tue Jun 28 20:12:31 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
[Tue Jun 28 20:12:34 2016] [info] [client 68.116.88.14] (OS 10054)An existing connection was forcibly closed by the remote host. : core_output_filter: writing data to the network
---------------------------------------------------------------------------



After the error above, restarting Windows and Apache service didn't solve the problem (same error message).
As you can see below, something is vey wrong, because 100 worker threads were consumed in 2 seconds. It's not normal at all.

---------------------------------------------------------------------------
[Tue Jun 28 20:25:04 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 20:25:04 2016] [notice] Child 1236: Exit event signaled. Child process is ending.
[Tue Jun 28 20:25:04 2016] [info] Child 1236: Accept thread exiting.
[Tue Jun 28 20:25:05 2016] [notice] Child 1236: Released the start mutex
[Tue Jun 28 20:25:34 2016] [notice] Parent: Forcing termination of child process 384
[Tue Jun 28 20:25:34 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=1028)
[Tue Jun 28 20:29:38 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 20:29:38 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 20:29:38 2016] [notice] Parent: Created child process 1760
[Tue Jun 28 20:29:38 2016] [info] Parent: Duplicating socket 308 and sending it to child process 1760
[Tue Jun 28 20:29:38 2016] [notice] Child 1760: Child process is running
[Tue Jun 28 20:29:38 2016] [notice] Child 1760: Acquired the start mutex.
[Tue Jun 28 20:29:38 2016] [notice] Child 1760: Starting 100 worker threads.
[Tue Jun 28 20:29:38 2016] [notice] Child 1760: Starting thread to listen on port 80.
[Tue Jun 28 20:29:40 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
---------------------------------------------------------------------------



After reinstalling Apache and booting the machine, the problem was still there (same error message, so I omitted such log lines here).
So I increased the "MaxRequestsPerChild" parameter to 300, but all these worker threads were consumed in 12 seconds as you can see below.

---------------------------------------------------------------------------
[Tue Jun 28 21:49:18 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 21:49:18 2016] [notice] Child 1136: Exit event signaled. Child process is ending.
[Tue Jun 28 21:49:18 2016] [info] Child 1136: Accept thread exiting.
[Tue Jun 28 21:49:19 2016] [notice] Child 1136: Released the start mutex
[Tue Jun 28 21:49:48 2016] [notice] Parent: Forcing termination of child process 368
[Tue Jun 28 21:49:48 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=1028)
[Tue Jun 28 21:49:48 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 21:49:48 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 21:49:48 2016] [notice] Parent: Created child process 864
[Tue Jun 28 21:49:48 2016] [notice] Child 864: Child process is running
[Tue Jun 28 21:49:48 2016] [info] Parent: Duplicating socket 308 and sending it to child process 864
[Tue Jun 28 21:49:48 2016] [notice] Child 864: Acquired the start mutex.
[Tue Jun 28 21:49:48 2016] [notice] Child 864: Starting 300 worker threads.
[Tue Jun 28 21:49:48 2016] [notice] Child 864: Starting thread to listen on port 80.
[Tue Jun 28 21:49:53 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
---------------------------------------------------------------------------



So, I increased the "MaxRequestsPerChild" parameter to 500, but all these worker threads were consumed in 9 seconds as shown below.

---------------------------------------------------------------------------
[Tue Jun 28 21:55:01 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 21:55:01 2016] [notice] Child 864: Exit event signaled. Child process is ending.
[Tue Jun 28 21:55:02 2016] [info] Child 864: Accept thread exiting.
[Tue Jun 28 21:55:02 2016] [notice] Child 864: Released the start mutex
[Tue Jun 28 21:55:20 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [notice] Parent: Forcing termination of child process 368
[Tue Jun 28 21:55:31 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=2252)
[Tue Jun 28 22:10:13 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 22:10:13 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 22:10:13 2016] [notice] Parent: Created child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Child process is running
[Tue Jun 28 22:10:13 2016] [info] Parent: Duplicating socket 308 and sending it to child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Acquired the start mutex.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting 500 worker threads.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting thread to listen on port 80.
[Tue Jun 28 22:10:22 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
---------------------------------------------------------------------------




Below you can see my complete original httpd.conf. It had been working correctly for months.
(I removed just all the comment lines to make it smaller and more readable, and also masked some minimal sensible data):

---------------------------------------------------------------------------
ServerRoot "C:/Apache2.2"

Listen 80

LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so

<IfModule !mpm_netware_module>
<IfModule !mpm_winnt_module>
User daemon
Group daemon

</IfModule>
</IfModule>


ServerAdmin xxxxxxxxxxxxxxxxxxxxxxxx

ServerName xxxxxxxxxxxxxxxxxxxxxxxx:80


DocumentRoot "xxxxxxxxxxxxxxxxxxxxxxxx"

<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
</Directory>


<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
Options FollowSymLinks
Options +Includes
Options -Indexes

AllowOverride None

Order allow,deny
Allow from all

LimitRequestBody 10485760

SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip

</Directory>

<IfModule dir_module>
DirectoryIndex index.html
</IfModule>

<FilesMatch "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</FilesMatch>

<IfModule mod_headers.c>
Header always append X-Frame-Options SAMEORIGIN
</IfModule>

<IfModule mod_expires.c>

ExpiresActive on
ExpiresDefault "access plus 1 week"

ExpiresByType text/css "access plus 1 week"

ExpiresByType application/atom+xml "access plus 1 hour"
ExpiresByType application/rdf+xml "access plus 1 hour"
ExpiresByType application/rss+xml "access plus 1 hour"

ExpiresByType application/json "access plus 0 seconds"
ExpiresByType application/ld+json "access plus 0 seconds"
ExpiresByType application/schema+json "access plus 0 seconds"
ExpiresByType application/vnd.geo+json "access plus 0 seconds"
ExpiresByType application/xml "access plus 0 seconds"
ExpiresByType text/xml "access plus 0 seconds"

ExpiresByType image/vnd.microsoft.icon "access plus 1 week"
ExpiresByType image/x-icon "access plus 1 week"

ExpiresByType text/html "access plus 1 day"

ExpiresByType application/javascript "access plus 1 week"
ExpiresByType application/x-javascript "access plus 1 week"
ExpiresByType text/javascript "access plus 1 week"

ExpiresByType application/manifest+json "access plus 1 year"

ExpiresByType application/x-web-app-manifest+json "access plus 0 seconds"
ExpiresByType text/cache-manifest "access plus 0 seconds"

ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType image/bmp "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"

<Directory xxxxxxxxxxxxxxxxxxxxxxxx>
ExpiresByType image/gif "access plus 0 seconds"
ExpiresByType image/jpeg "access plus 0 seconds"
ExpiresByType image/png "access plus 0 seconds"
</Directory>



ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType font/eot "access plus 1 month"

ExpiresByType font/opentype "access plus 1 month"

ExpiresByType application/x-font-ttf "access plus 1 month"

ExpiresByType application/font-woff "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"
ExpiresByType font/woff "access plus 1 month"

ExpiresByType application/font-woff2 "access plus 1 month"

ExpiresByType text/x-cross-domain-policy "access plus 1 week"

</IfModule>


ErrorLog "|bin/rotatelogs logs/error.%Y-%m-%d.log 50M"

LogLevel info

<IfModule log_config_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common

<IfModule logio_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>

CustomLog "|bin/rotatelogs logs/access.%Y-%m-%d---%H.log 300M" common

</IfModule>

<IfModule alias_module>


ScriptAlias /p2p/ "xxxxxxxxxxxxxxxxxxxxxxxx"

</IfModule>

<IfModule cgid_module>
</IfModule>

<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>

DefaultType text/plain

<IfModule mime_module>
TypesConfig conf/mime.types

AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz

AddType text/html .html
AddHandler server-parsed .html
</IfModule>


<IfModule mpm_winnt_module>
ThreadsPerChild 100
MaxMemFree 100
MaxRequestsPerChild 5000
</IfModule>


<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>


RewriteEngine On
RewriteCond %{HTTP_HOST} !^www
RewriteRule (.*) http://www.%{HTTP_HOST}$1 [L,R]
---------------------------------------------------------------------------


My site doesn't connect to databases. It just serves dynamic pages by using CGI.

Am I missing any useful information?

Is there a way to clean up the system to make Apache work again as it used to work before this mysterious crash?

Could anyone give me any advice to solve this issue or at least to get a workaround?

Thanks in advance,

Leonardo.