apache DDOS

[Massimo]

hi all,

i just received a bad DDOS on apache (amp)

Server Version: Apache/2.2.17 (OS/2) PHP/5.2.8
Server Built: Jan 27 2011 18:46:38

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 84 0/25/25 _ 333.44 4 25 0.0 0.34 0.34 net-2-33-145-36.cust.dsl.teletu www.mydomain.XYZ GET
/nuovo_index.php HTTP/1.1
0-0 84 0/24/24 _ 333.15 5 26 0.0 0.33 0.33 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/24/24 _ 334.28 4 27 0.0 0.33 0.33 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/24/24 _ 334.64 3 29 0.0 0.33 0.33 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/24/24 _ 334.01 4 25 0.0 0.33 0.33 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/22/22 _ 335.41 2 35 0.0 0.31 0.31 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/22/22 _ 334.50 3 33 0.0 0.31 0.31 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/22/22 _ 335.13 3 31 0.0 0.31 0.31 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/24/24 _ 334.92 3 33 0.0 0.33 0.33 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/29/29 _ 331.71 6 33 0.0 1.08 1.08 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/24/24 _ 335.88 2 49 0.0 0.33 0.33 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/23/23 _ 336.09 2 55 0.0 0.32 0.32 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/23/23 _ 329.24 9 39 0.0 0.30 0.30 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz GET
/nuovo_index.php HTTP/1.1
0-0 84 0/24/24 _ 336.43 1 117 0.0 0.33 0.33 net-2-33-145-36.cust.dsl.teletu www.mydomain.xyz

etc... etc...

is there any way to void this kind of DDOS from httpd.conf?

thanks a lot in advance for any help
bye

massimo s.

[Ian Manners]

Hi Massimo,

It's the Festive Season so :)

>i just received a bad DDOS on apache (amp)
>Server Version: Apache/2.2.17 (OS/2) PHP/5.2.8
>Server Built: Jan 27 2011 18:46:38

There is a connection limit plugin but only for Apache v1.3.xx, I dont
think anyones bothered porting one to Apache v2.xx.xx for OS/2.

Otherwise drop this into your InJoyFW

Connection-Limit
Comment = "BL on 25 concurrent connections",
Rule-Action = Blacklist,
Log-Control = Enabled,
Log-Mask = "date time severity message dump
resolved_source resolved_dest",
Log-Message = "BL (Reason: more than 25 connections)",
Log-Details = "BL - exceed 25 connections",
Log-Severity = Warning,
Blacklist-Period = "0:3:0",
Direction = Incoming,
Connections = ">=25"

In Blacklist-Period = "0:3:0", 3 is hours, change to taste, ditto
the Connections setting.
To be more restrictive and narrow it down to one internal IP etc try

Connection-Limit-1ip
Comment = "BL on 10 concurrent connections",
Destination = "192.168.1.20",
Rule-Action = Blacklist,
Log-Control = Enabled,
Log-Mask = "date time severity message dump
resolved_source resolved_dest",
Log-Message = "BL (Reason: more than 10 connections)",
Log-Severity = Warning,
Blacklist-Period = "1:0:0",
Direction = Incoming,
Connections = ">=10"

Blacklist period is set for 1 day in the above example, change Destination
IP to suit, ditto any logging stuff etc.

You can also narrow it down by checking your logs for common
trigger points, ie,

awstatstotals.php
Comment = "/awstatstotals.php BL",
Destination-Port = "http",
Rule-Action = Blacklist,
Log-Control = Enabled,
Log-Mask = "date time severity message dump
resolved_source resolved_dest",
Log-Message = "/awstatstotals.php IP Blacklisted",
Log-Details = "Blacklisted /awstatstotals.php",
Log-Severity = Warning,
Hex-String = "<nocase>/awstatstotals.php",
Blacklist-Period = "2:0:0",
Direction = Incoming

and

wp_content
Comment = "/wp-content/ BL",
Destination-Port = "http",
Rule-Action = Blacklist,
Log-Control = Enabled,
Log-Mask = "date time severity message dump
resolved_source resolved_dest",
Log-Message = "/wp-content/ IP Blacklisted",
Log-Details = "Blacklisted /wp-content/.ro",
Log-Severity = Warning,
Hex-String = "<nocase>/wp-content/",
Blacklist-Period = "2:0:0",
Direction = Incoming

Cheers
Ian Manners
http://www.os2site.com/

[Lewis G Rosenthal]

Hi, guys...

On 12/26/11 11:15 am, Ian Manners thus wrote :

> Hi Massimo,
>
> It's the Festive Season so :)
>
>> i just received a bad DDOS on apache (amp)
>> Server Version: Apache/2.2.17 (OS/2) PHP/5.2.8
>> Server Built: Jan 27 2011 18:46:38
> There is a connection limit plugin but only for Apache v1.3.xx, I dont
> think anyones bothered porting one to Apache v2.xx.xx for OS/2.
>
> Otherwise drop this into your InJoyFW
>

<snip>

I concur with you, Ian, that this is a firewall issue. Kudos to the
concise ruleset you provided, too!

Max, is your web server *really* your bastion server, or do you have a
separate firewall machine in front of it?

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS
Rosenthal& Rosenthal, LLC www.2rosenthals.com
Need a managed Wi-Fi hotspot? www.hautspot.com
visit my IT blog www.2rosenthals.net/wordpress
please do not add my address to any non-bcc mass mailings
-------------------------------------------------------------

[Massimo]

i've 2 server

one has dns, webmail and mta
one has ftp + apache + mysql (web server)

of course the fw is injoy pro on the same 2 machines
not separated hardware

massimo

[Lewis G Rosenthal]

On 12/26/11 03:05 pm, Massimo thus wrote :

> i've 2 server
>
> one has dns, webmail and mta
> one has ftp + apache + mysql (web server)
>
> of course the fw is injoy pro on the same 2 machines
> not separated hardware
>

So, essentially, you're duplicating your firewall setup (i.e., two
independent firewalls on separate machines), instead of a single
firewall acting as an internet router for the servers behind it. It
might be more efficient to invest in a single firewall machine, whether
it is running eCS with InJoy on it, some Linux-based appliance, or a
Linux-based server running some IPS and other utilities. Just a thought.
It doesn't have to be a powerful machine, either. I routinely deploy
these as single or dual Pentium III Xeon servers (such as Compaq
Proliant DL360 G1) with a pair of mirrored 18GB drives.

> Il 26/12/2011 20.10, Lewis G Rosenthal ha scritto:
>> Hi, guys...
>>
>> On 12/26/11 11:15 am, Ian Manners thus wrote :
>>> Hi Massimo,
>>>
>>> It's the Festive Season so :)
>>>
>>>> i just received a bad DDOS on apache (amp)
>>>> Server Version: Apache/2.2.17 (OS/2) PHP/5.2.8
>>>> Server Built: Jan 27 2011 18:46:38
>>> There is a connection limit plugin but only for Apache v1.3.xx, I dont
>>> think anyones bothered porting one to Apache v2.xx.xx for OS/2.
>>>
>>> Otherwise drop this into your InJoyFW
>>>
>> <snip>
>>
>> I concur with you, Ian, that this is a firewall issue. Kudos to the concise ruleset you
>> provided, too!
>>
>> Max, is your web server *really* your bastion server, or do you have a separate firewall
>> machine in front of it?
>>

--

[waldo kitty]

On 12/26/2011 16:00, Lewis G Rosenthal wrote:
> On 12/26/11 03:05 pm, Massimo thus wrote :
>> i've 2 server
>>
>> one has dns, webmail and mta
>> one has ftp + apache + mysql (web server)
>>
>> of course the fw is injoy pro on the same 2 machines
>> not separated hardware
>>
> So, essentially, you're duplicating your firewall setup (i.e., two independent
> firewalls on separate machines), instead of a single firewall acting as an
> internet router for the servers behind it. It might be more efficient to invest
> in a single firewall machine, whether it is running eCS with InJoy on it, some
> Linux-based appliance, or a Linux-based server running some IPS and other
> utilities. Just a thought.

agreed 100000000000000000000~ percent... these days a central perimeter firewall
is a necessity, IMHO... especially one that can also possibly act as an IDS and
at least log acts that may be beating on one's servers... why let the server
take the brunt of these beatings when you can easily have a "bouncer at the
door" to beat them back with...

> It doesn't have to be a powerful machine, either. I
> routinely deploy these as single or dual Pentium III Xeon servers (such as
> Compaq Proliant DL360 G1) with a pair of mirrored 18GB drives.

that's a lot more than i recommend as a basic perimeter firewall system... the
minimum i recommend is a 800mhz with at least 512Meg RAM and a 12G HD... i use
an old AMD box i got dumpster diving (aka trash bin)... it has 4 NICs for 3
internal zones and the 4th NIC is for the internet... the internal zones are
protected from each other and all are equally protected from the internet...
IDS, SMTP filtering, proxy, QOS and other services are standard...

it is a no brainer, really...

[Lewis G Rosenthal]

Hi, Waldo!

On 12/26/11 08:19 pm, waldo kitty thus wrote :

> On 12/26/2011 16:00, Lewis G Rosenthal wrote:
>> On 12/26/11 03:05 pm, Massimo thus wrote :
>>> i've 2 server
>>>
>>> one has dns, webmail and mta
>>> one has ftp + apache + mysql (web server)
>>>
>>> of course the fw is injoy pro on the same 2 machines
>>> not separated hardware
>>>
>> So, essentially, you're duplicating your firewall setup (i.e., two
>> independent
>> firewalls on separate machines), instead of a single firewall acting
>> as an
>> internet router for the servers behind it. It might be more efficient
>> to invest
>> in a single firewall machine, whether it is running eCS with InJoy on
>> it, some
>> Linux-based appliance, or a Linux-based server running some IPS and
>> other
>> utilities. Just a thought.
>
> agreed 100000000000000000000~ percent... these days a central
> perimeter firewall is a necessity, IMHO... especially one that can
> also possibly act as an IDS and at least log acts that may be beating
> on one's servers... why let the server take the brunt of these
> beatings when you can easily have a "bouncer at the door" to beat them
> back with...
>

Exactly.

>> It doesn't have to be a powerful machine, either. I
>> routinely deploy these as single or dual Pentium III Xeon servers
>> (such as
>> Compaq Proliant DL360 G1) with a pair of mirrored 18GB drives.
>
> that's a lot more than i recommend as a basic perimeter firewall
> system... the minimum i recommend is a 800mhz with at least 512Meg RAM
> and a 12G HD... i use an old AMD box i got dumpster diving (aka trash
> bin)... it has 4 NICs for 3 internal zones and the 4th NIC is for the
> internet... the internal zones are protected from each other and all
> are equally protected from the internet... IDS, SMTP filtering, proxy,
> QOS and other services are standard...
>
> it is a no brainer, really...
>

:-)

Indeed, I've been testing Astaro 8 on a PIII/800 with 512MB in it. I
thought of the DL360, because I can get those chassis quite cheaply
these days (one live; one for a hot spare - together for under $400).
That said, in most of my Astaro configurations, I have dozens of IPSec
VPNs terminating, and there really *is* a difference when running the
software on something more substantial (I have a dual P4 Xeon DL360 G3
in one place, and a DL360 G4 elsewhere...the G4 is nice because HP
finally added support for redundant power supplies). Still, without the
VPNs, you're absolutely correct: there isn't much need for a heavy box.

Cheers/2

[Massimo]

Il 26/12/2011 17.15, Ian Manners ha scritto:
> Connection-Limit-1ip
> Comment = "BL on 10 concurrent connections",
> Destination = "192.168.1.20",
> Rule-Action = Blacklist,
> Log-Control = Enabled,
> Log-Mask = "date time severity message dump
> resolved_source resolved_dest",
> Log-Message = "BL (Reason: more than 10 connections)",
> Log-Severity = Warning,
> Blacklist-Period = "1:0:0",
> Direction = Incoming,
> Connections = ">=10"

i've used this rule
thanks a lot

one question how must work these "concurrent connections"?

e.g. if a customer open 10 connections
and after 20 minutes oper other 15 connections
it get banned or not?

how much last this conc. connections?

thanks
bye

massimo s.

[Ian Manners]

Hi Massimo,

>> Connection-Limit-1ip
>> Comment = "BL on 10 concurrent connections",
>> Destination = "192.168.1.20",
>> Rule-Action = Blacklist,
>> Log-Control = Enabled,
>> Log-Mask = "date time severity message dump
resolved_source
resolved_dest",
>> Log-Message = "BL (Reason: more than 10 connections)",
>> Log-Severity = Warning,
>> Blacklist-Period = "1:0:0",
>> Direction = Incoming,
>> Connections = ">=10"
>
>i've used this rule
>thanks a lot
>
>one question how must work these "concurrent connections"?

>e.g. if a customer open 10 connections
>and after 20 minutes oper other 15 connections
>it get banned or not?

customer will not be banned, they need to be >= x connections
in parallel, ie, the same time. If you want a safe margin then set
the connections = ">=15".

You could also trim the Blacklist-Period down to a couple of hours,
ie Blacklist-Period = "0:2:0",

>how much last this conc. connections?

depends on your system setup, if keepalive is set to 60 second,
and your apache is setup to the same then your customer will be
reusing connection sockets anyway.

The connections have to be concurrent. If your customer uses POP3
for example, as soon as they have finished then the connection will
be reset anyway. So it also depends on the protocol you are using
as well as the software and system setup.

Connections = ">=10" on my system normally gets bots, dos's, and
smegheads using massively parallel downloaders, oh, and the odd
OS/2'er that has tweaked his/her prefs.js file to have > 10 download
threads :o)

Normal programs like wget and others all obey the unofficial rules
of connections to a server.

[Massimo]

i had to put >40 connections, expecially for search engine crawlers
i guess since i've a lot of websites hosted on the same ip/server

massimo

[Ian Manners]

Hi Massimo,

>i had to put >40 connections, expecially for search engine crawlers
>i guess since i've a lot of websites hosted on the same ip/server

You might want to analyse what bots are hitting your websites, 90%
or more are likely to simply be data slurpers or 'test' search engines
that dont help you at all. I only allow the main ones in, any search
engine (or bot) that reads (or doesnt read) robots.txt and ignores the
directives gets added to Apache's rewrite rules, or banned in the FW
depending on where its from and what it does.

Google, Bing, and even Yahoo bots are all smart enough to not use
more than 4 connections on the one IP even with a lot of virtual hosts
on that IP. The one Google bot that keeps getting banned here is NOT
from google but a search bot using a fake UA.

<http://www.os2site.com/rewrite.txt>

The above rules will need to be modified as it does includes some
legitimate UA strings, and it changes on whim and severity of ignoring
robots.txt, many are certainly permanent fixtures. Others like Baiduspider
are now in the FW due to ignoring robots.txt and internet edicate.

The REQUEST_URI will also need to be checked if you use public
facing PHP scripts etc.