apache 2.2.34 php 5.4.45 mod_ssl/2.2.34 openssl/1.0.2k hangs

[Massimo S.]

Hi all,

in the last months (since the end of october 2019) i've a number of hangs
of the web server due to attacks (ddos or other) that comes on apache.

Server Version: Apache/2.2.34 (OS/2) PHP/5.4.45 mod_ssl/2.2.34 OpenSSL/1.0.2k
Server Built: Feb 3 2019 17:11:06

I've Max Request per child active and set to 99.
It help a bit, but don't resolve the issue.

Often, sometimes 2 times per day i receive some kind of attacks on http or https, some kind of
them i can filter with Injoy Firewall, some of them
i still have to understand how they act.

Attacks like the slow loris, syn rotating packets attack etc..

I also put a limit to 29 max tcp/ip connections from the same ip on http and https.

All theese protections seems not to be sufficient.

Any idea?
Any help?

on the same server i've also a mail server with 11 mail domains and the database mysql, so when
the server hangs also webmail (the frontend is
on another server) and 11 mail domanins sto working, that's a problem.

I'm also thinking of virtualizing the server2 and do a guest VM
only for apache+php
and another guest VM only for mysql+mail server

massimo

[Massimo S.]

hi all,

about MPM_MPMT

i have:

MinSpareThreads 29
MaxSpareThreads 33

but i see this:

P-ID PPID Session Thr Prio CPU Time Name

2954 2947 01F VIO 33 0200 0:00:08.43 HTTPD.EXE 2953 2947 01F
VIO 34 0200 0:00:10.25 HTTPD.EXE 2952 2947 01F VIO 35 0200
0:00:08.06 HTTPD.EXE 2951 2947 01F VIO 33 0200 0:00:11.21
HTTPD.EXE 2950 2947 01F VIO 34 0200 0:00:09.15 HTTPD.EXE

after some time..

3790 3785 020 VIO 36 0200 0:00:04.65 HTTPD.EXE

where 3, 35, 33 and 34 are threads, is this normal?

thanks

massimo

[Massimo S.]

> after some time..
>
> 3790 3785 020 VIO  36 0200    0:00:04.65        HTTPD.EXE
>
> where 3, 35, 33 and 34 are threads, is this normal?
>
> thanks
>
> massimo
>

where 36, 35, 33 and 34 are threads
sorry the typo and the bad copy&paste

massimo

[Lewis G Rosenthal]

On 01/21/20 04:52 pm, Massimo S. wrote:
>
>
> Il 21/01/2020 22:46, Massimo S. ha scritto:
>>
>>
>> Il 20/01/2020 10:54, Massimo S. ha scritto:
>>> Hi all,
>>>
>>> in the last months (since the end of october 2019) i've a number of hangs
>>> of the web server due to attacks (ddos or other) that comes on apache.
>>>

How do you know these are "attacks"? Often, legitimate traffic may result in
the same type of overload as a deliberate attempt to bring the server
offline. Without examining the log, it's hard to say.

>>> Server Version: Apache/2.2.34 (OS/2) PHP/5.4.45 mod_ssl/2.2.34
>>> OpenSSL/1.0.2k
>>> Server Built: Feb 3 2019 17:11:06
>>>
>>> I've Max Request per child active and set to 99.
>>> It help a bit, but don't resolve the issue.
>>>

Okay.

>>> Often, sometimes 2 times per day i receive some kind of attacks on http
>>> or https, some kind of them i can filter with Injoy Firewall, some of them
>>> i still have to understand how they act.
>>>
>>> Attacks like the slow loris, syn rotating packets attack etc..
>>>
>>> I also put a limit to 29 max tcp/ip connections from the same ip on http
>>> and https.
>>>
>>> All theese protections seems not to be sufficient.
>>>
>>> Any idea?
>>> Any help?
>>>

Not enough info. No idea what modules you have loaded, or anything else, let
alone what the server is serving. PHP is likely the biggest drag on the
system, and that's a whole other set of configurations (php.ini and the
settings for the PHP apps themselves).

>>> on the same server i've also a mail server with 11 mail domains and the
>>> database mysql, so when the server hangs also webmail (the frontend is
>>> on another server) and 11 mail domanins sto working, that's a problem.
>>>

Yes, I would think it would be.

>>> I'm also thinking of virtualizing the server2 and do a guest VM
>>> only for apache+php
>>> and another guest VM only for mysql+mail server
>>>
>>> massimo
>>
>> hi all,
>>
>> about MPM_MPMT
>>
>> i have:
>>
>> MinSpareThreads 29
>> MaxSpareThreads 33
>>
>> but i see this:
>>
>> P-ID PPID Session Thr Prio CPU Time Name
>
>> after some time..
>>
>> 3790 3785 020 VIO 36 0200 0:00:04.65 HTTPD.EXE
>>
>> where 3, 35, 33 and 34 are threads, is this normal?
>>
>> thanks
>>
>> massimo
>>
>
> where 36, 35, 33 and 34 are threads
> sorry the typo and the bad copy&paste
>

No idea. How would I know, I wonder?

As posted a gazillion times on this list already:

<IfModule mpm_mpmt_os2_module>
ThreadStackSize 262144
StartServers 3
MinSpareThreads 50
MaxSpareThreads 60
MaxRequestsPerChild 1000
</IfModule>

Been that way for ages. If your min/max values are too close, the server
will labor to constantly adjust the amount of spare threads. Note I said
*spare* threads, as in, idle threads available for work but which are *not*
in use. Without knowing what your overall thread count is in the box, your
THREADS= in CONFIG.SYS, how could anyone possibly know whether your thread
settings for the MPM are sane?

Sorry to sound out of sorts, here, but please do some research on these
values and what they mean before suspecting anything is wrong. If you feel
that the server performance is off, switch back to defaults and start tuning
again. How you tune depends upon what you're serving. In fact, how you have
your SSL configured makes a big difference as well, as there is a
significant amount of overhead which may be generated when establishing
connections. There is no magic pill for all of this.

If you are seeing specific hangs, read your logs and try to determine common
factors, such as source IP and target vhost/page. Work from that to
determine what is happening before you start making massive changes to
configuration, because all you'll end up doing is making the real problem
harder to find.

Good luck.

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS, EA
Rosenthal & Rosenthal, LLC www.2rosenthals.com
visit my IT blog www.2rosenthals.net/wordpress
-------------------------------------------------------------

[Massimo S.]

Il 22/01/2020 22:59, Lewis G Rosenthal ha scritto:
> On 01/21/20 04:52 pm, Massimo S. wrote:
>>
>>
>> Il 21/01/2020 22:46, Massimo S. ha scritto:
>>>
>>>
>>> Il 20/01/2020 10:54, Massimo S. ha scritto:
>>>> Hi all,
>>>>
>>>> in the last months (since the end of october 2019) i've a number of hangs
>>>> of the web server due to attacks (ddos or other) that comes on apache.
>>>>
>
> How do you know these are "attacks"? Often, legitimate traffic may result in the same type of
> overload as a deliberate attempt to bring the server offline. Without examining the log, it's
> hard to say.
>
>>>> Server Version: Apache/2.2.34 (OS/2) PHP/5.4.45 mod_ssl/2.2.34 OpenSSL/1.0.2k
>>>> Server Built: Feb 3 2019 17:11:06
>>>>
>>>> I've Max Request per child active and set to 99.
>>>> It help a bit, but don't resolve the issue.
>>>>
>
> Okay.
>
>>>> Often, sometimes 2 times per day i receive some kind of attacks on http or https, some kind
>>>> of them i can filter with Injoy Firewall, some of them
>>>> i still have to understand how they act.
>>>>
>>>> Attacks like the slow loris, syn rotating packets attack etc..
>>>>
>>>> I also put a limit to 29 max tcp/ip connections from the same ip on http and https.
>>>>
>>>> All theese protections seems not to be sufficient.
>>>>
>>>> Any idea?
>>>> Any help?
>>>>
>
> Not enough info. No idea what modules you have loaded,

loadModule authn_file_module modules/authn_fi.dll
loadmodule authn_default_module modules/authn_de.dll
loadmodule authz_host_module modules/authz_ho.dll
loadmodule authz_groupfile_module modules/authz_gr.dll
loadmodule authz_user_module modules/authz_us.dll
loadmodule authz_default_module modules/authz_de.dll
loadmodule auth_basic_module modules/auth_bas.dll
loadmodule auth_digest_module modules/auth_dig.dll
loadmodule dumpio_module modules/dumpio.dll
loadmodule echo_module modules/echo.dll
loadmodule ext_filter_module modules/ext_filt.dll
loadmodule include_module modules/include.dll
loadmodule filter_module modules/filter.dll
loadmodule deflate_module modules/deflate.dll
loadmodule log_config_module modules/log_conf.dll
loadmodule log_forensic_module modules/log_fore.dll
loadmodule env_module modules/env.dll
loadmodule mime_magic_module modules/mime_mag.dll
loadmodule expires_module modules/expires.dll
loadmodule headers_module modules/headers.dll
loadmodule ident_module modules/ident.dll
loadmodule usertrack_module modules/usertrac.dll
loadmodule setenvif_module modules/setenvif.dll
loadmodule version_module modules/version.dll
loadmodule ssl_module modules/ssl.dll
loadmodule mime_module modules/mime.dll
loadmodule status_module modules/status.dll
loadmodule autoindex_module modules/autoinde.dll
loadmodule asis_module modules/asis.dll
loadmodule info_module modules/info.dll
loadmodule vhost_alias_module modules/vhost_al.dll
loadmodule negotiation_module modules/negotiat.dll
loadmodule dir_module modules/dir.dll
loadmodule imagemap_module modules/imagemap.dll
loadmodule alias_module modules/alias.dll
loadmodule rewrite_module modules/rewrite.dll
loadModule php5_module modules/modphp5.dll

>or anything else, let alone what the
> server is serving. PHP is likely the biggest drag on the system,

i know this i've used apache with only html websites without php
and it's rock solid stable

> and that's a whole other set
> of configurations (php.ini and the settings for the PHP apps themselves).

php.ini (resource limit part)

max_execution_time = 180
max_input_time = 600
max_input_time = 180
memory_limit = 256M

>>>> on the same server i've also a mail server with 11 mail domains and the database mysql, so
>>>> when the server hangs also webmail (the frontend is
>>>> on another server) and 11 mail domanins sto working, that's a problem.
>>>>
>
> Yes, I would think it would be.
>
>>>> I'm also thinking of virtualizing the server2 and do a guest VM
>>>> only for apache+php
>>>> and another guest VM only for mysql+mail server
>>>>
>>>> massimo
>>>
>>> hi all,
>>>
>>> about MPM_MPMT
>>>
>>> i have:
>>>
>>> MinSpareThreads       29
>>> MaxSpareThreads       33
>>>
>>> but i see this:
>>>
>>> P-ID PPID Session Thr Prio    CPU Time                    Name
>>
>>> after some time..
>>>
>>> 3790 3785 020 VIO  36 0200    0:00:04.65        HTTPD.EXE
>>>
>>> where 3, 35, 33 and 34 are threads, is this normal?
>>>
>>> thanks
>>>
>>> massimo
>>>
>>
>> where 36, 35, 33 and 34 are threads
>> sorry the typo and the bad copy&paste
>>
>
> No idea. How would I know, I wonder?

it sound to me strange as MaxSpareThreads were 33
and the threads of the childs show 36..

> As posted a gazillion times on this list already:
>
> <IfModule mpm_mpmt_os2_module>
>     ThreadStackSize    262144
>     StartServers            3
>     MinSpareThreads        50
>     MaxSpareThreads        60
>     MaxRequestsPerChild  1000
> </IfModule>
>
> Been that way for ages. If your min/max values are too close, the server will labor to
> constantly adjust the amount of spare threads. Note I said *spare* threads, as in, idle threads
> available for work but which are *not* in use. Without knowing what your overall thread count

There are 32 Processes with 305 Threads.
This machine's uptime is 0d 2h 40m 25s 949ms.

apache:

P-ID PPID Session Thr Prio CPU Time Name

4440 16 020 VIO 1 0200 0:00:05.12 HTTPD.EXE
4974 4440 020 VIO 36 0200 0:00:02.31 HTTPD.EXE
4973 4440 020 VIO 36 0200 0:00:07.87 HTTPD.EXE
4972 4440 020 VIO 34 0200 0:00:06.90 HTTPD.EXE
4971 4440 020 VIO 33 0200 0:00:06.75 HTTPD.EXE

> is in the box, your THREADS= in CONFIG.SYS, how could anyone possibly know whether your thread
> settings for the MPM are sane?

threads in config.sys are

THREADS=1024

i've the same value on my 2 server, but also on the new AOS 503 on the
test virtual machine that in the next monts will handle only apache+php
i've seen that AOS put this value as default

other values in httpd.conf that may be of interested are set to:

Timeout 60
KeepAlive On
MaxKeepAliveRequests 45
KeepAliveTimeout 5


about MinSpareTd and MaxSpareTd i've right now modified the apache configuration and now i have:

<IfModule mpm_mpmt_os2_module>
StartServers 4
MinSpareThreads 29
MaxSpareThreads 39
MaxRequestsPerChild 95
</IfModule>


..ThreadStackSize 262144
what this value?
i don't know much about this function is this value the one to use on mpmt_os2 or is a generic one?

> Sorry to sound out of sorts, here, but please do some research on these values and what they

i've read and tried a lot in the past
in the latest years that web server was more stable than in the last months
after october 2019 i've seen an increase in the number of DDOS attacks
and that kept the server more unstable (or at least i suppose this)

> mean before suspecting anything is wrong. If you feel that the server performance is off,
> switch back to defaults and start tuning again. How you tune depends upon what you're serving.
> In fact, how you have your SSL configured makes a big difference as well, as there is a
> significant amount of overhead which may be generated when establishing connections. There is
> no magic pill for all of this.

slowly i'm moving websites to https

thanks a lot for the help


massimo

[wkit...@windstream.net]

On 1/22/20 5:46 PM, Massimo S. wrote:
> php.ini (resource limit part)
>
> max_execution_time = 180
> max_input_time = 600
> max_input_time = 180
> memory_limit = 256M

is there a typo above? the second and third lines define the same thing but with
different values...


--
NOTE: No off-list assistance is given without prior approval.
*Please keep mailing list traffic on the list where it belongs!*

[Lewis G Rosenthal]

On 01/22/20 05:46 pm, Massimo S. wrote:
>
>
> Il 22/01/2020 22:59, Lewis G Rosenthal ha scritto:
>> On 01/21/20 04:52 pm, Massimo S. wrote:
>>>
>>>
>>> Il 21/01/2020 22:46, Massimo S. ha scritto:
>>>>
>>>>
>>>> Il 20/01/2020 10:54, Massimo S. ha scritto:
>>>>> Hi all,
>>>>>
>>>>> in the last months (since the end of october 2019) i've a number of hangs
>>>>> of the web server due to attacks (ddos or other) that comes on apache.
>>>>>
>>

<snip>

Suggestion #1: comment any modules you don't need. I doubt that you need
*all* those auth modules (authz_groupfile? really?). Do you really need
echo? usertrack consumes considerable overhead. Do you really track your
users via cookies? If not, you don't need this module.

>> or anything else, let alone what the server is serving. PHP is likely the
>> biggest drag on the system,
>
> i know this i've used apache with only html websites without php
> and it's rock solid stable
>

All well and good. When using php as module, there is a considerable amount
of interaction between httpd and php resources (php is loaded *all the
time*). You need to balance http performance against php performance
(sometimes), and you have to remember that you are essentially sharing
resources between the two environments (shared memory resources, that is).

>> and that's a whole other set of configurations (php.ini and the settings
>> for the PHP apps themselves).
>
> php.ini (resource limit part)
>
> max_execution_time = 180
> max_input_time = 600
> max_input_time = 180
> memory_limit = 256M
>

As Waldo has rightly point out, you have duplicate entries here. php.ini is
read and processed top down, so the last setting takes precedence. That
said, a production value for max_input_time is 60 seconds, not 10(!) or even
3 minutes (holy cow!), and production setting for max_execution_time is 30
seconds, not 3 minutes. the longer your php connections remain open, the
more you expose the server to getting "stuck." These aren't attacks, but
just lingering connections which shouldn't be left that way. Most PHP code -
for better or worse - leaves this up to the server setting (or
application-specific setting) to address.

memory_limit at 256M is fine (I used to be able to run at a fraction of this
value, but PHP hackers - not malicious hackers or crackers, but people
writing PHP scripts - being what they are, much code has just gotten sloppy
in recent years, requiring more memory due to shoddy practices). Still, it;s
nice to be able to keep this low overall and just set it higher in the
config for specific apps which require it.

<snip>

>>>> about MPM_MPMT
>>>>
>>>> i have:
>>>>
>>>> MinSpareThreads 29
>>>> MaxSpareThreads 33
>>>>
>>>> but i see this:
>>>>
>>>> P-ID PPID Session Thr Prio CPU Time Name
>>>
>>>> after some time..
>>>>
>>>> 3790 3785 020 VIO 36 0200 0:00:04.65 HTTPD.EXE
>>>>
>>>> where 3, 35, 33 and 34 are threads, is this normal?
>>>>
>>>> thanks
>>>>
>>>> massimo
>>>>
>>>
>>> where 36, 35, 33 and 34 are threads
>>> sorry the typo and the bad copy&paste
>>>
>>
>> No idea. How would I know, I wonder?
>
> it sound to me strange as MaxSpareThreads were 33
> and the threads of the childs show 36..
>

3 threads in use and 33 spares were available. The point of setting min/max
values is that these are created and idle (the min value) at daemon start
and will be added (put to use and the number of spares possibly increased)
as demanded. The numbers should bounce between the in-use threads + MinSpare
and in-use + MaxSpare.

>> As posted a gazillion times on this list already:
>>
>> <IfModule mpm_mpmt_os2_module>
>> ThreadStackSize 262144
>> StartServers 3
>> MinSpareThreads 50
>> MaxSpareThreads 60
>> MaxRequestsPerChild 1000
>> </IfModule>
>>
>> Been that way for ages. If your min/max values are too close, the server
>> will labor to constantly adjust the amount of spare threads. Note I said
>> *spare* threads, as in, idle threads available for work but which are
>> *not* in use. Without knowing what your overall thread count
>
> There are 32 Processes with 305 Threads.
> This machine's uptime is 0d 2h 40m 25s 949ms.
>

You need to monitor your threads over time to see what happens to this
value. Also, 2 hours is a vry short time to get an impression. Check it at
random intervals or when the system seems to be loaded and over a few days
of uptime. That should give you a better picture.

> apache:
>
> P-ID PPID Session Thr Prio CPU Time Name
>
> 4440 16 020 VIO 1 0200 0:00:05.12 HTTPD.EXE
> 4974 4440 020 VIO 36 0200 0:00:02.31 HTTPD.EXE
> 4973 4440 020 VIO 36 0200 0:00:07.87 HTTPD.EXE
> 4972 4440 020 VIO 34 0200 0:00:06.90 HTTPD.EXE
> 4971 4440 020 VIO 33 0200 0:00:06.75 HTTPD.EXE
>
>
>> is in the box, your THREADS= in CONFIG.SYS, how could anyone possibly
>> know whether your thread settings for the MPM are sane?
>
> threads in config.sys are
>
> THREADS=1024
>
> i've the same value on my 2 server, but also on the new AOS 503 on the
> test virtual machine that in the next monts will handle only apache+php
> i've seen that AOS put this value as default
>

Yes, it's a sane default figure. For a server, I wouldn't recommend anything
lower.

> other values in httpd.conf that may be of interested are set to:
>
> Timeout 60
> KeepAlive On
> MaxKeepAliveRequests 45
> KeepAliveTimeout 5
>

I would cut timeout to 30 (again, the longer you sit around waiting, the
more resources you're going to waste in the process). In the days of 56Kbps
connections, longer timeouts were necessary; today, not so much.

MaxKeepAliveRequests should be a higher value (Apache docs recommend 500 or
so; I use 500). The server will labor when this is set lower, as connections
will have to be reestablished. Reestablishing a connection is an expensive
proposition. There are conflicting user reports about this, with some
recommending very low values. I've *never* had good luck on OS/2 with this
set low (on NetWare, I've not found it to make much difference; again, it
all depends upon the memory model and how expensive these operations are on
a given platform).

KeepAliveTimeout 5 is a reasonable number. I would *not* recommend setting
this higher.

>
> about MinSpareTd and MaxSpareTd i've right now modified the apache
> configuration and now i have:
>
> <IfModule mpm_mpmt_os2_module>
> StartServers 4
> MinSpareThreads 29
> MaxSpareThreads 39
> MaxRequestsPerChild 95
> </IfModule>
>
>
> ..ThreadStackSize 262144
> what this value?
> i don't know much about this function is this value the one to use on
> mpmt_os2 or is a generic one?
>

If you are *not* getting crashes with Apache blowing the stack, you probably
don't need to adjust this. Various MPMs use this directive to set the stack
size allocated for each worker thread. The default on OS/2 is (still, IIRC)
65536 (see: https://mantis.smedley.id.au/view.php?id=558 and
https://mantis.smedley.id.au/view.php?id=637 for more).

(That second link contains a lot of research and testing we did back in
2014/2015.)

>> Sorry to sound out of sorts, here, but please do some research on these
>> values and what they
>
> i've read and tried a lot in the past
> in the latest years that web server was more stable than in the last months
> after october 2019 i've seen an increase in the number of DDOS attacks
> and that kept the server more unstable (or at least i suppose this)
>

We all go through rough periods, and stopping all bad traffic is simply not
possible, even with the best firewalls. Still, not everything which appears
to be a DOS is really a DOS. Your server may be busy enough with something
else that clients get impatient and connections start piling up. It's a
balancing act, always, and it gets more difficult every year.

>> mean before suspecting anything is wrong. If you feel that the server
>> performance is off, switch back to defaults and start tuning again. How
>> you tune depends upon what you're serving. In fact, how you have your SSL
>> configured makes a big difference as well, as there is a significant
>> amount of overhead which may be generated when establishing connections.
>> There is no magic pill for all of this.
>
> slowly i'm moving websites to https
>

Best recommendation:

https://www.ssllabs.com/ssltest/

Test your server at the above. While you're there, try testing
www.arcanoae.com or www.2rosenthals.com (A+ on both of those). Don't
discount the server load for poor SSL configuration.

Full disclosure: I own stock in Qualys. The above test is completely free,
however, and Qualys does not use the ssllabs.com site to solicit business.
There is a lot of good information in the Qualys blog, too (though I miss
Ivan Ristić there; he's quite a wealth of information and has gone on to
form Hardenize: https://www.hardenize.com).

> thanks a lot for the help
>

You bet.

[Massimo S.]

removed echo_module and authz_groupfile
less modules -> less resources used, less potential security flaws

>usertrack consumes considerable
> overhead. Do you really track your users via cookies? If not, you don't >need this module.

dunno exactly, but i know that some websites use cookies
i've here also CMS joomla and wp CMSs (so cookie's banners etc.)

>>> or anything else, let alone what the server is serving. PHP is likely the biggest drag on
>>> the system,
>>
>> i know this i've used apache with only html websites without php
>> and it's rock solid stable
>>
>
> All well and good. When using php as module, there is a considerable amount of interaction
> between httpd and php resources (php is loaded *all the time*). You need to balance http
> performance against php performance (sometimes), and you have to remember that you are
> essentially sharing resources between the two environments (shared memory resources, that is).
>
>>> and that's a whole other set of configurations (php.ini and the settings for the PHP apps
>>> themselves).
>>
>> php.ini (resource limit part)
>>
>> max_execution_time = 180
>> max_input_time = 600
>> max_input_time = 180
>> memory_limit = 256M

max_input_time = 600 this is a typo (since it's commented in php.ini)

others now reduced to theese:

max_execution_time = 60
max_input_time = 120

i raised up them in the past years for Prestashop CMS, but now i've no prestashop websites, so....

i use MRTG

>Also, 2 hours is
> a vry short time to get an impression. Check it at random intervals or when the system seems to
> be loaded and over a few days of uptime. That should give you a better picture.

this server do 2 reboots (setboot/b) per days in the night
and since 18 jen 2020 i've 2 hang/freeze per day
i've moved here from another webserver a terrible WP website
but now i've moved back to the external webserver

>> apache:
>>
>> P-ID PPID Session Thr Prio    CPU Time                    Name
>>
>> 4440   16 020 VIO   1 0200    0:00:05.12      HTTPD.EXE
>> 4974 4440 020 VIO  36 0200    0:00:02.31        HTTPD.EXE
>> 4973 4440 020 VIO  36 0200    0:00:07.87        HTTPD.EXE
>> 4972 4440 020 VIO  34 0200    0:00:06.90        HTTPD.EXE
>> 4971 4440 020 VIO  33 0200    0:00:06.75        HTTPD.EXE
>>
>>
>>> is in the box, your THREADS= in CONFIG.SYS, how could anyone possibly know whether your
>>> thread settings for the MPM are sane?
>>
>> threads in config.sys are
>>
>> THREADS=1024
>>
>> i've the same value on my 2 server, but also on the new AOS 503 on the
>> test virtual machine that in the next monts will handle only apache+php
>> i've seen that AOS put this value as default
>>
>
> Yes, it's a sane default figure. For a server, I wouldn't recommend anything lower.
>
>> other values in httpd.conf that may be of interested are set to:
>>
>> Timeout 60
>> KeepAlive On
>> MaxKeepAliveRequests 45
>> KeepAliveTimeout 5
>>
>
> I would cut timeout to 30 (again, the longer you sit around waiting, the more resources you're
> going to waste in the process). In the days of 56Kbps connections, longer timeouts were
> necessary; today, not so much.

will try 45 then 30 in the next days today i've moved too much parameters :)

> MaxKeepAliveRequests should be a higher value (Apache docs recommend 500 or so; I use 500). The
> server will labor when this is set lower, as connections will have to be reestablished.
> Reestablishing a connection is an expensive proposition. There are conflicting user reports
> about this, with some recommending very low values. I've *never* had good luck on OS/2 with
> this set low (on NetWare, I've not found it to make much difference; again, it all depends upon
> the memory model and how expensive these operations are on a given platform).

ok raised from 45 to 100
i've to do all theese tests slowly to understand the behaviour

> KeepAliveTimeout 5 is a reasonable number. I would *not* recommend setting this higher.
>
>>
>> about MinSpareTd and MaxSpareTd i've right now modified the apache configuration and now i have:
>>
>> <IfModule mpm_mpmt_os2_module>
>>     StartServers           4
>>     MinSpareThreads       29
>>     MaxSpareThreads       39
>>     MaxRequestsPerChild   95
>> </IfModule>
>>
>>
>> ..ThreadStackSize    262144
>> what this value?
>> i don't know much about this function is this value the one to use on mpmt_os2 or is a
>> generic one?
>>

i will try also this option
since the default i read i 65536 and it seems that's too low

tried and i get B due to this:

This server's certificate chain is incomplete. Grade capped to B.
since i don't have this entry in the dns zone:

mydomain.org. CAA 128 issue "letsencrypt.org"

i'm running bind 9.6.0pl1 and it don't support CAA function
i've to upgrade at least to 9.9.6

i've tried newer binds in the last months, but it seems to me that all were
giving issues, so i sticket to 9.6.0

will see what to do thanks

for now i've moved too much parameters so i will wait 1 or 2 days to see how the things goes on


massimo

[wkit...@windstream.net]

thanks for that ssllabs link... it looks to be quite handy and provides a lot of
good information on a site's https implementation... i'd quote the link here but
for some reason, my ISP's SMTP server thought it was a spam message just now
when i tried to respond quoting only that link and writing a sentence or two so
i've replied like this without any quotes ;)

[Massimo S.]

Il 23/01/2020 13:09, Massimo S. ha scritto:
>>>
>>
>> Suggestion #1: comment any modules you don't need. I doubt that you need *all* those auth
>> modules (authz_groupfile? really?). Do you really need echo?
> removed echo_module and authz_groupfile
> less modules -> less resources used, less potential security flaws

doh :-(

after having removed authz_groupfile....
a phone call arrived :D

i had to re-enable it due to a php webapp that use it

ok, but at least i've turned off echo_module :)

massimo

[Lewis G Rosenthal]

Hi, Waldo...

On 01/23/20 08:07 am, wkit...@windstream.net wrote:
>
> thanks for that ssllabs link... it looks to be quite handy and provides a
> lot of good information on a site's https implementation... i'd quote the
> link here but for some reason, my ISP's SMTP server thought it was a spam
> message just now when i tried to respond quoting only that link and
> writing a sentence or two so i've replied like this without any quotes ;)
>

Gotta commend that AI concerning potential spam you might be trying to
relay. LOL

Yes, it's a great site to get a feel for how the server looks and behaves to
the outside world. Sometimes when errors are reported, they will link to
substantive suggestions for improvement. I learned an awful lot about how
all of these pieces fit together and how different cipher strings work just
from reading the comments there. Invaluable information, truly.

Cheers

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS, EA

[Lewis G Rosenthal]

Hi, Max...

(Keeping most of this for context; apologies to all for the absurdly long
quoting.)

You don't need to have the *server* instance track cookies just becuase
various applications running on the server use cookies. These are different
issues.

I see tha you have determined that you really do need authz_groupfile.
That's an interesting one. I haven't seen that on a rooted Apache
installation on a single-user OS before. The idea is that you should be
aware of what modules you actually use and try to avoid loading the ones you
don't, as they are just so much extra baggage to take along with you.

>>>> or anything else, let alone what the server is serving. PHP is likely
>>>> the biggest drag on the system,
>>>
>>> i know this i've used apache with only html websites without php
>>> and it's rock solid stable
>>>
>>
>> All well and good. When using php as module, there is a considerable
>> amount of interaction between httpd and php resources (php is loaded *all
>> the time*). You need to balance http performance against php performance
>> (sometimes), and you have to remember that you are essentially sharing
>> resources between the two environments (shared memory resources, that is).
>>
>>>> and that's a whole other set of configurations (php.ini and the
>>>> settings for the PHP apps themselves).
>>>
>>> php.ini (resource limit part)
>>>
>>> max_execution_time = 180
>>> max_input_time = 600
>>> max_input_time = 180
>>> memory_limit = 256M
>
> max_input_time = 600 this is a typo (since it's commented in php.ini)
>

Okay.

> others now reduced to theese:
>
> max_execution_time = 60
> max_input_time = 120
>
> i raised up them in the past years for Prestashop CMS, but now i've no
> prestashop websites, so....
>

The higher they are, the more the chance the PHP instance will get stuck for
no good reason.

Okay. It doesn't really matter what you use. The point is that you need to
watch the value over time and under different conditions to get a good feel
for how many threads you're using.

>> Also, 2 hours is a vry short time to get an impression. Check it at
>> random intervals or when the system seems to be loaded and over a few
>> days of uptime. That should give you a better picture.
>
> this server do 2 reboots (setboot/b) per days in the night
> and since 18 jen 2020 i've 2 hang/freeze per day
> i've moved here from another webserver a terrible WP website
> but now i've moved back to the external webserver
>

You should not have to bounce the box twice per day. I was down to every
four days, and now every 6 (I think; I need to check crontabs). Of course,
if your memory is becoming fragmented sooner, you'll need to reboot sooner,
and all of these values play into that fragmentation problem, as we have
seen over the years.

Agreed, and a point worth repeating. As with any tuning changes, making too
many at one time is just bad practice. You need to do this gradually and
methodically to watch for unexpected changes, and document behavior as you
go (and not rely on sheer brain memory alone).

>> MaxKeepAliveRequests should be a higher value (Apache docs recommend 500
>> or so; I use 500). The server will labor when this is set lower, as
>> connections will have to be reestablished. Reestablishing a connection is
>> an expensive proposition. There are conflicting user reports about this,
>> with some recommending very low values. I've *never* had good luck on
>> OS/2 with this set low (on NetWare, I've not found it to make much
>> difference; again, it all depends upon the memory model and how expensive
>> these operations are on a given platform).
>
> ok raised from 45 to 100
> i've to do all theese tests slowly to understand the behaviour
>

The mantis links I provided should give you some background as to the tests
we conducted and what we found at the time.

>> KeepAliveTimeout 5 is a reasonable number. I would *not* recommend
>> setting this higher.
>>
>>>
>>> about MinSpareTd and MaxSpareTd i've right now modified the apache
>>> configuration and now i have:
>>>
>>> <IfModule mpm_mpmt_os2_module>
>>> StartServers 4
>>> MinSpareThreads 29
>>> MaxSpareThreads 39
>>> MaxRequestsPerChild 95
>>> </IfModule>
>>>
>>>
>>> ..ThreadStackSize 262144
>>> what this value?
>>> i don't know much about this function is this value the one to use on
>>> mpmt_os2 or is a generic one?
>>>
>
> i will try also this option
> since the default i read i 65536 and it seems that's too low
>

It's only too low if you are seeing crashes. If Apache isn't crashing with a
blown stack, you don't need to bump this value; you're just wasting
resources. That said, these are rather cheap by today's standards, so feel
free to experiment. I should probably lower this figure, I guess. My notes
aren't exceptionally clear, though I did note in one of the Mantis tickets
that values *below* 65536 seemed to cause consistent problems (crashes).
Again, YMMV.

>> If you are *not* getting crashes with Apache blowing the stack, you
>> probably don't need to adjust this. Various MPMs use this directive to
>> set the stack size allocated for each worker thread. The default on OS/2
>> is (still, IIRC) 65536 (see: https://mantis.smedley.id.au/view.php?id=558
>> and https://mantis.smedley.id.au/view.php?id=637 for more).
>>
>> (That second link contains a lot of research and testing we did back in
>> 2014/2015.)
>>

<snip>

>>> slowly i'm moving websites to https
>>>
>>
>> Best recommendation:
>>
>> https://www.ssllabs.com/ssltest/
>>
>> Test your server at the above. While you're there, try testing
>> www.arcanoae.com or www.2rosenthals.com (A+ on both of those). Don't
>> discount the server load for poor SSL configuration.
>
> tried and i get B due to this:
>
> This server's certificate chain is incomplete. Grade capped to B.

You probably need to install the intermediate cert. I don't use LE certs,
and I *specifically* remove the intermediates as they are unnecessary for my
CA (this is for performance reasons, as every cert fetch is another
transaction). I have no idea for LE what may be required.

> since i don't have this entry in the dns zone:
>
> mydomain.org. CAA 128 issue "letsencrypt.org"
>
> i'm running bind 9.6.0pl1 and it don't support CAA function
> i've to upgrade at least to 9.9.6
>

Unfortunately (or fortunately, depending on one's POV), CAA is a whole new
record type and not something which can be faked with a TXT record.

> i've tried newer binds in the last months, but it seems to me that all were
> giving issues, so i sticket to 9.6.0
>

Not having a CAA record should only limit you to A, not B. B cap is due to
the cert chain being incomplete, not the lack of CAA.

> will see what to do thanks
>
> for now i've moved too much parameters so i will wait 1 or 2 days to see
> how the things goes on
>

Good idea.

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS, EA

[wkit...@windstream.net]

On 1/23/20 11:28 AM, Lewis G Rosenthal wrote:
> On 01/23/20 08:07 am, wkit...@windstream.net wrote:
>> thanks for that ssllabs link... it looks to be quite handy and provides a lot
>> of good information on a site's https implementation... i'd quote the link
>> here but for some reason, my ISP's SMTP server thought it was a spam message
>> just now when i tried to respond quoting only that link and writing a sentence
>> or two so i've replied like this without any quotes ;)
>
> Gotta commend that AI concerning potential spam you might be trying to relay. LOL

yeah, it rather took me aback at first... it arrived here just fine so i guess
it was to do with it being mostly by itself... no big deal LUL

> Yes, it's a great site to get a feel for how the server looks and behaves to the
> outside world. Sometimes when errors are reported, they will link to substantive
> suggestions for improvement. I learned an awful lot about how all of these
> pieces fit together and how different cipher strings work just from reading the
> comments there. Invaluable information, truly.

i posted about it to several of the devs i work with and at least one has
already released changes disabling all of the TLS_RSA_* ciphers as well as
disabling TLS 1.0 and 1.1 in the server... am updating and building that code as
we write... testing coming up soonest ;)

thanks again for that link... i'll be sharing it with others over time :)

[Massimo S.]

Il 23/01/2020 17:50, Lewis G Rosenthal ha scritto:
>>> Best recommendation:
>>>
>>> https://www.ssllabs.com/ssltest/
>>>
>>> Test your server at the above. While you're there, try testing www.arcanoae.com or
>>> www.2rosenthals.com (A+ on both of those). Don't discount the server load for poor SSL
>>> configuration.
>>
>> tried and i get B due to this:
>>
>> This server's certificate chain is incomplete. Grade capped to B.
>
> You probably need to install the intermediate cert. I don't use LE certs, and I *specifically*
> remove the intermediates as they are unnecessary for my CA (this is for performance reasons, as
> every cert fetch is another transaction). I have no idea for LE what may be required.
>
>> since i don't have this entry in the dns zone:
>>
>> mydomain.org. CAA 128 issue "letsencrypt.org"
>>
>> i'm running bind 9.6.0pl1 and it don't support CAA function
>> i've to upgrade at least to 9.9.6
>>
>
> Unfortunately (or fortunately, depending on one's POV), CAA is a whole new record type and not
> something which can be faked with a TXT record.
>
>> i've tried newer binds in the last months, but it seems to me that all were
>> giving issues, so i sticket to 9.6.0
>>
>
> Not having a CAA record should only limit you to A, not B. B cap is due to the cert chain being
> incomplete, not the lack of CAA.

thanks
i've seen and i confirm

i've a comodo certificate on a domain and it gives me "A" as result

the difference is that the Comodo's certificate has also a certificate chain file

SSLCACertificateFile x:/path/mydomin_it.ca-bundle

while let's encrypt has only private key and the domain's certificate

uacme client only download they key and the certificate
but not the chain/cabundle/intermediate or whatever it's being called :(

lucklily browsers don't complain about that...

anyway i've downloaded chain/cabundle or whatever it's being called :D
from here:

https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

added to the apache vhost and now i've get my "A" ;)


thanks a lot again for the suggestion

massimo

[Zdenek Wagner]

čt 23. 1. 2020 v 22:18 odesílatel Massimo S. <m...@ecomstation.it> napsal:
>
>
>
...

>
> thanks
> i've seen and i confirm
>
> i've a comodo certificate on a domain and it gives me "A" as result
>
> the difference is that the Comodo's certificate has also a certificate chain file
>
> SSLCACertificateFile x:/path/mydomin_it.ca-bundle
>
> while let's encrypt has only private key and the domain's certificate
>

Let's encrypt does have a certificate chain but for some reason it is
marked as invalid. I looked at the details and the server certificate
is also in the chain which is most probably the culprit. I have not
tried to solve it yet.

> uacme client only download they key and the certificate
> but not the chain/cabundle/intermediate or whatever it's being called :(
>
> lucklily browsers don't complain about that...

Zdeněk Wagner
http://ttsm.icpf.cas.cz/team/wagner.shtml
http://icebearsoft.euweb.cz

[Massimo S.]

i'm modifying parameters very slowly

theese are the actual parameters (the one modified):

httpd.conf:

Timeout 50
MaxKeepAliveRequests 100

<IfModule mpm_mpmt_os2_module>
StartServers 4
MinSpareThreads 39
MaxSpareThreads 50
MaxRequestsPerChild 99
</IfModule>

i've also removed tlsv1, tlsv1.1 and old cypher protocols

php.ini:

max_execution_time = 120 max_input_time
= 120


the situation seems to be improved a lot
i will do further improvements but slowly in the next days :)


massimo

[Massimo S.]

LOL what to say...
yesterday night a hang/freeze just after about 45 minutes from the scheduled night reboot

and last connections recorded in fw logs are on port 80 and 443...

massimo

[Massimo S.]

situation is not improving... 4 hangs in about 10 hours


massimo

[Massimo S.]

today 12 hours of uptime and at 21:44 and at about 22:00 2 hang/freeze
i'm start to think that someone find a sort of vulnerability flaw/security exploit and they use
to hang the server

massimo

[wkit...@windstream.net]

On 1/27/20 4:07 PM, Massimo S. wrote:
[... trim waaaay too many lines to count ...]

> today 12 hours of uptime and at 21:44 and at about 22:00 2 hang/freeze
> i'm start to think that someone find a sort of vulnerability flaw/security
> exploit and they use to hang the server

is the entire machine freezing or just the application?

if you think someone is doing this, perhaps you should run a capture of the
network packet traffic coming in to your port 80... if the machine is not
locking up, then it should be easy enough to analyze the pcap file and see the
offending traffic...

tcpdump -i ppp0 -s0 -w http-%Y%m%d%H%M%S.pcap -G 3600 -C 200 'tcp port 80 and
dst host you.host.ip.address'

the above looks on the ppp0 interface for all traffic on port 80 destined to the
specified host IP number... all these packets are written to the file... the
file name is rotated every hour or when file hits 20meg in size...

the filenames are made of a 4 digit year, 2 digit month, 2 digit day followed by
the hour, minute, and second of the file creation...

eg:
http-20200127204300.pcap

if it takes 10 minutes to capture 200Meg, the new filename will be

http-20200127205300.pcap

this is simply a way to obtain workable files and be able to easily kill off old
ones not needed any more...


aside: if you were running an IDS/IPS and someone were actually doing something
nefarious, the detection/protection system would likely already be raising
alerts so you would know what was happening and not need to be guessing if stuff
is happening ;)

[Massimo S.]

Il 28/01/2020 02:47, wkit...@windstream.net ha scritto:
> On 1/27/20 4:07 PM, Massimo S. wrote:
> [... trim waaaay too many lines to count ...]
>> today 12 hours of uptime and at 21:44 and at about 22:00 2 hang/freeze
>> i'm start to think that someone find a sort of vulnerability flaw/security exploit and they
>> use to hang the server
>
>
> is the entire machine freezing or just the application?

unfortunately yes

and there is a worst thing too...
that a number of logs write on a RAM disk
so i also loose a number of logs each time it freeze :(


massimo

[Massimo S.]

sorry, i mean that it's entire server that hang/freeze

massimo

[wkit...@windstream.net]

On 1/27/20 8:47 PM, wkit...@windstream.net wrote:
> tcpdump -i ppp0 -s0 -w http-%Y%m%d%H%M%S.pcap -G 3600 -C 200 'tcp port 80 and
> dst host you.host.ip.address'
>
> the above looks on the ppp0 interface for all traffic on port 80 destined to the
> specified host IP number... all these packets are written to the file... the
> file name is rotated every hour or when file hits 20meg in size...

there is a typo above... -C 200 is 200Meg (in base 10) not 20meg as i wrote...

[wkit...@windstream.net]

On 1/28/20 4:03 AM, Massimo S. wrote:
>
>
> Il 28/01/2020 02:47, wkit...@windstream.net ha scritto:
>> On 1/27/20 4:07 PM, Massimo S. wrote:
>> [... trim waaaay too many lines to count ...]
>>> today 12 hours of uptime and at 21:44 and at about 22:00 2 hang/freeze
>>> i'm start to think that someone find a sort of vulnerability flaw/security
>>> exploit and they use to hang the server
>>
>>
>> is the entire machine freezing or just the application?
>
> unfortunately yes

yes to which part of the question??

> and there is a worst thing too...
> that a number of logs write on a RAM disk
> so i also loose a number of logs each time it freeze :(

based on this response, i guess you mean the entire machine is freezing/hanging...

[wkit...@windstream.net]

On 1/28/20 4:05 AM, Massimo S. wrote:
> Il 28/01/2020 10:03, Massimo S. ha scritto:
>> Il 28/01/2020 02:47, wkit...@windstream.net ha scritto:
>>> is the entire machine freezing or just the application?
>>
>> unfortunately yes
>>
>> and there is a worst thing too...
>> that a number of logs write on a RAM disk
>> so i also loose a number of logs each time it freeze :(
>

> sorry, i mean that it's entire server that hang/freeze

i should have read ahead but yeah, i got it :lol:

thanks for the clarification...

[Massimo S.]

sorry for the big quota

i realized since some months that the real source of the problem was a faulty ram (Kingston
KTH-XW4300E/2G)

in the last month i've changed: UPS, power supply, motherboard, CPU and all the cables...

...but, i only kept ram... sigh

now i'm running with only 2GB of ram sometimes the machine (due for apache)
go at 512KB of ram (no ram free at all:D) so that rarely something trap
at ring0 and the machine reboot since i have REIPL=ON

i've ordered 1GB ram, i guess that with 3GB will surely not run out of memory


massimo

[Massimo S.]

after the change of the RAM anyway the server has had a number of freezes

so that i changed completely another mother board (see $$$) and freezes continues... about 1-3
per days

it' clear that since i've modified the apache mmpt configuration
the situation has worstned not improved

i suspect the high number of threads

what i did...

i bought a new server now i'm using Vbox 6.1.2 VMs
the server web has an AOS 5.0.3 machine and *it only run:
apache+php, ftpd, cron, injoy fw and nothing more

i'm moving all the websites into the new AOS503 web server only guest VMs

the guest VMs still has reboot scheduled, 2 days ago come into production
yesterday at 00 o'clock at a scheduled reboot (setboot /b) the server freezed/hang due to
apache running out of "who knows system resources"

this is no good

i've *isolated apache+php in a 4GB VM only* to have it running completely
away by itself and also with this dedicated configuration there are issues

now i'm asking to Paul and Steven a bit of help, we have to understand what goes wrong *in
apache and php* and i'm not sufficient to undrestand what cause this i've seen in the past
maybe an HTTPD child that become unkillable
so bad that even a setboot /b can't reboot the server

setboot /b is the last resource if this freeze the server we are fuc$ed

only Steven and Paul can help here

Server Version: Apache/2.2.34 (OS/2) PHP/5.4.45 mod_ssl/2.2.34 OpenSSL/1.0.2k
Server Built: Feb 3 2019 17:11:06

i'm at full disposal to run diagnostics and such

this is the apache setup:

Timeout 60
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
UseCanonicalName On
ServerTokens Full
HostnameLookups On

<IfModule mpm_mpmt_os2_module>
StartServers 4
MinSpareThreads 40
MaxSpareThreads 60
MaxRequestsPerChild 99
</IfModule>



please help me



massimo

[Steven Levine]

In <eca69d2c-1294-f4e8...@ecomstation.it>, on 02/26/20
at 09:57 AM, "Massimo S." <m...@ecomstation.it> said:

Hi Massimo,

>only Steven and Paul can help here

Since my name came up, the only question I have at this time is where is
the mantis ticket for this issue?

Steven

--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------

[Massimo S.]

Il 29/02/2020 06:48, Steven Levine ha scritto:
> In <eca69d2c-1294-f4e8...@ecomstation.it>, on 02/26/20
> at 09:57 AM, "Massimo S." <m...@ecomstation.it> said:
>
> Hi Massimo,
>
>> only Steven and Paul can help here
>
> Since my name came up, the only question I have at this time is where is
> the mantis ticket for this issue?
>
> Steven

Hi Steven,

i hope that Paul will build php 5.6.40, soon i'll upgrade the webserver
that now run isolated inside a dedicated virtual machine

if the issue that sometimes a child "zombify" and become unkillable
i will open a ticket and do further verifies/tests


massimo

[Paul Smedley]

Hey Max,

On 29/2/20 10:04 pm, Massimo S. wrote:
> i hope that Paul will build php 5.6.40, soon i'll upgrade the webserver
> that now run isolated inside a dedicated virtual machine

I can look at it... however...
Support for PHP 5 has been discontinued since 10 Jan 2019.Please
consider upgrading to 7.

So 5.6.40 potentially leaves you exposed to security risks - albeit less
exposed than the jurassic 5.4.x build you're currently using....

[Massimo S.]

for now i can't upgrade to 7 due to a number of websites that at the moment clients wont to
spend money to fix or upgrade them
so that 5.6.40 it's the best option for me at the moment
surely better than 5.4.45
(5.6.x is still used at a number of ISPs)

i'm also running apache 2.2.34 is php 7 compatible with it?

thanks

massimo