0 idle workers

[Lewis Rosenthal]

I finally caught the server in the process of falling over. The daemon
did restart, but the server-status showed some very interesting stuff
just prior:


Current Time: Saturday, 01-Aug-2015 18:25:20 EDT
Restart Time: Saturday, 01-Aug-2015 16:00:10 EDT
Parent Server Generation: 0
Server uptime: 2 hours 25 minutes 9 seconds
Total accesses: 2511 - Total Traffic: 9.3 MB
CPU Usage: u3406.38 s0 cu0 cs0 - 39.1% CPU load
.288 requests/sec - 1119 B/second - 3882 B/request
508 requests currently being processed, 0 idle workers

WWWWWWWWWWWWWWKWWWWWWWWKWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWWW
WKWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW..
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWWWWWWWW
KWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW..
................................................................

The vast majority of those connections being written were returns for a
Google maps plugin for a Joomla site. However, that's not what's really
interesting. We should not have run out of idle workers. The MPM says:

ThreadStackSize 262144
StartServers 2
MinSpareThreads 50
MaxSpareThreads 60
MaxRequestsPerChild 0

We should have continued to spawn threads until we ran out of resources
(memory fragmentation might be a possibility, but the box had not been
up that long for me to expect this to be an issue, and there was plenty
of available RAM - over a gig).

server-config clearly shows that the MPM settings are being read.
MaxKeepAliveRequests? That is set for 500, with a higher keepalive
timeout than I remembered (300 seconds, which is probably way too high).

THREADS in CONFIG.SYS is set for 1024, so it is conceivable that we
actually did run out of threads. Checking now (system relatively idle),
there are 299 non-httpd threads in use. It is possible, that with the
system under load and responding more slowly (Communigate is currently
using 98 threads), other things could have spawned more threads leaving
only 508 threads available for all httpd processes. The problem for me
is that I saw us hit that number *exactly* on multiple occasions with
Apache restarts in between and even server reboots. Seems a strange
coincidence.

So, in addition to lowering the Apache timeout to 3 and keepalive
timeout to 10 seconds, I bumped StartServers to 5 (from 2) and increased
THREADS in CONFIG.SYS to 2048.

I did see the system get to almost 1,000 current requests, but idle
workers stayed above 0 (though still lower than I would have expected
for 5 processes which should maintain a minimum of 50 spares).

Obviously, more monitoring is required.

Oh, and I fixed the (old) broken Google maps plugin which was
functioning as an open web proxy (ugh...), and have a redirect rule in
place to immediately trash requests for the offending php script.

Anyone else ever run out of idle workers?

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS, EA
Rosenthal & Rosenthal, LLC www.2rosenthals.com
visit my IT blog www.2rosenthals.net/wordpress
IRS Circular 230 Disclosure applies see www.2rosenthals.com
-------------------------------------------------------------


--
This email was Anti Virus checked by Astaro Security Gateway. http://www.astaro.com

[Steven Levine]

In <55BDA9A...@2rosenthals.com>, on 08/02/15
at 01:24 AM, Lewis Rosenthal <lgros...@2rosenthals.com> said:

Hi,

>508 requests currently being processed, 0 idle workers

> ThreadStackSize 262144

>We should have continued to spawn threads until we ran out of resources

Which is probalby what happened.

508 * 256K is something like 135M. That does not leave a lot of lower
private memory for other purposes, especially if there is moderate
fragmentation.

>up that long for me to expect this to be an issue, and there was plenty
>of available RAM - over a gig).

As I said many times. There is available memory and there is available
memory. It's not all the same.

>THREADS in CONFIG.SYS is set for 1024, so it is conceivable that we
>actually did run out of threads.

Recall also the recent httpd builds say

#ifndef HARD_THREAD_LIMIT
#define HARD_THREAD_LIMIT 254 // 2014-11-30 SHL avoid VIO per
session thread limit #endif

>timeout to 10 seconds, I bumped StartServers to 5 (from 2) and increased
>THREADS in CONFIG.SYS to 2048.

The StartServers bump might be useful, assuming the CPU has cycles left to
get any work done.

Steven

--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> Warp/DIY/eComStation etc.
www.scoug.com
----------------------------------------------------------------------

[Lewis Rosenthal]

Hi, Steve...

On 08/02/2015 01:51 AM, Steven Levine wrote:
> In <55BDA9A...@2rosenthals.com>, on 08/02/15
> at 01:24 AM, Lewis Rosenthal <lgros...@2rosenthals.com> said:
>
> Hi,
>
>> 508 requests currently being processed, 0 idle workers
>> ThreadStackSize 262144
>> We should have continued to spawn threads until we ran out of resources
> Which is probalby what happened.
>

Not quite, as proven by the fact that I was able to spawn almost 1,000
threads when I increased the spare servers (processes).

> 508 * 256K is something like 135M. That does not leave a lot of lower
> private memory for other purposes, especially if there is moderate
> fragmentation.
>

True, but I wouldn't even expect moderate fragmentation when the system
was up for 10 minutes (unless something was really having a problem). ;-)

>> up that long for me to expect this to be an issue, and there was plenty
>> of available RAM - over a gig).
> As I said many times. There is available memory and there is available
> memory. It's not all the same.
>
>> THREADS in CONFIG.SYS is set for 1024, so it is conceivable that we
>> actually did run out of threads.
> Recall also the recent httpd builds say
>
> #ifndef HARD_THREAD_LIMIT
> #define HARD_THREAD_LIMIT 254 // 2014-11-30 SHL avoid VIO per
> session thread limit #endif
>

Yep. That's the one. I forgot about that. Thus, more spare servers will
allow for more overall threads (254 * x spare servers).

>> timeout to 10 seconds, I bumped StartServers to 5 (from 2) and increased
>> THREADS in CONFIG.SYS to 2048.
> The StartServers bump might be useful, assuming the CPU has cycles left to
> get any work done.
>

CPUs were not overtaxed. the system was still responsive, and I could
even kill Apache - if I did it soon enough (5-10 minutes was a definite
unkillable process, however, but not due to CPU contention).

It was indeed the hard limit, and I did forget about that safeguard.
Thanks for reminding me and solving the mystery. Let's see how these new
timeouts and additional spare servers work for us.

Cheers

[Steven Levine]

In <55BDBC2B...@2rosenthals.com>, on 08/02/15

at 02:43 AM, Lewis Rosenthal <lgros...@2rosenthals.com> said:

Hi,

>Not quite,

My bad. It more like 70MB or so per process given the 254 thread limit.

>as proven by the fact that I was able to spawn almost 1,000
>threads when I increased the spare servers (processes).

This is controlled by the system thread limit.

>True, but I wouldn't even expect moderate fragmentation when the system
>was up for 10 minutes (unless something was really having a problem). ;-)

One would think so, but measurements might prove otherwise. The Thesues
linear memory map is good at showing fragmentation. My linear_memory_map
script script is my preferred for capturing these maps. The resulting
text file is easier to analyze than the GUI window which can overflow.

>CPUs were not overtaxed.

Once you hit the thread limit, it likely that the CPUs are not overtaxed
because all the threads are doing nothing other than consuming scarce
linear memory resources.

My comment applies to an otherwise stable system that is processing
requests.

>Let's see how these new
>timeouts and additional spare servers work for us.

Given the stability issues, the additional servers should help.

[Massimo S.]

Il 02/08/2015 09:00, Steven Levine ha scritto:
> In<55BDBC2B...@2rosenthals.com>, on 08/02/15
> at 02:43 AM, Lewis Rosenthal<lgros...@2rosenthals.com> said:
>
> Hi,
>
>> >Not quite,
> My bad. It more like 70MB or so per process given the 254 thread limit.
>
>> >as proven by the fact that I was able to spawn almost 1,000
>> >threads when I increased the spare servers (processes).
> This is controlled by the system thread limit.
>
>> >True, but I wouldn't even expect moderate fragmentation when the system
>> >was up for 10 minutes (unless something was really having a problem).;-)
> One would think so, but measurements might prove otherwise. The Thesues
> linear memory map is good at showing fragmentation. My linear_memory_map
> script script is my preferred for capturing these maps. The resulting
> text file is easier to analyze than the GUI window which can overflow.
>
>> >CPUs were not overtaxed.
> Once you hit the thread limit, it likely that the CPUs are not overtaxed
> because all the threads are doing nothing other than consuming scarce
> linear memory resources.
>
> My comment applies to an otherwise stable system that is processing
> requests.
>
>> >Let's see how these new
>> >timeouts and additional spare servers work for us.
> Given the stability issues, the additional servers should help.

if it's about of any help, here i have 6 start servers (1 father with 6 childs httpds)
for about 70 vhosts

massimo