Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

apache swapping the system to death

3 views
Skip to first unread message

unruh

unread,
May 21, 2012, 9:35:14 PM5/21/12
to
I am running a web server on a Mandriva 2010.2 system. Today the system
suddenly became increadibly non-responsive. When I finally logged on as
root, the swap file was up around 3GB, (no wonder response was horrible)
and there were 160 instances of httpd running. After I finally managed
to shut them down, (killall -9 httpd) response was restored. I looked in
the /var/log/httpd/access_log there did not seem to be much unusual
there. There were some google.com with weird addreses, and some internal
connections which were the only things that looked out of the oridinary.

Eg
66.249.68.198 - - [21/May/2012:18:19:23 -0700] "GET /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/bit.ly/www.guardian.co.uk/business/2012/may/04/pay-vince-cable HTTP/1.1" 200 72658 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.68.73 - - [21/May/2012:18:19:26 -0700] "GET /aggregator/www.nytimes.com/2012/04/05/opinion/node/node/www.bbc.co.uk/news/uk-17769717?page=226 HTTP/1.1" 200 38984 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

and

::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"
::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"
::1 - - [21/May/2012:17:32:05 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"
::1 - - [21/May/2012:17:32:07 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"

....


But could any of these be respoinsible for 160 connections?

The other suspicious thing is that there was 6 hour gap in
the logs

112.111.174.175 - - [21/May/2012:11:34:11 -0700] "GET /user/register
HTTP/1.0" 200 29860 "http://emergentgravity.org/user/register"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) Opera 6.01 [en]"
::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"

Bit Twister

unread,
May 21, 2012, 10:28:34 PM5/21/12
to
On Tue, 22 May 2012 01:35:14 GMT, unruh wrote:
> I am running a web server on a Mandriva 2010.2 system.

That is getting a bit long in the tooth.
Hope you kept up with all the security updates.

> The other suspicious thing is that there was 6 hour gap in
> the logs

That does not look good. Can we assume you checked all the logs for same gap.

A cursory look at https://isc.sans.edu/diary.html?date=2012-05-08
and http://www.mandriva.com/en/support/security/advisories/?dis=2010.1
seems to show Mandriva a bit behind in patches.

If I did not have something like aide, osiris, ossec-hids, samhain,
tripwire, snare, integrit,... I would be a bit nervous about the
integrity of the system.

unruh

unread,
May 21, 2012, 11:19:42 PM5/21/12
to
The problem definitely seems to be the things like

66.249.68.198 - - [21/May/2012:20:12:34 -0700] "GET
/aggregator/www.guardian.co.uk/media/2011/jun/29/www.nytimes.com/2012/04/20/world/middleeast/syria-united-nations-secretary-general-ban-ki-moon-cease-fire.html?page=34
HTTP/1.1" 200 76079 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"

which seem to stay up forever. Of course my system has no such file. The
address is definitely claimed to be google by whois.
But these requests seem to be hanging that httpd which is trying to
answer them. And they come back up almost immediately if I shut down
httpd.

unruh

unread,
May 21, 2012, 11:27:14 PM5/21/12
to
On 2012-05-22, unruh <un...@invalid.ca> wrote:
> On 2012-05-22, Bit Twister <BitTw...@mouse-potato.com> wrote:
>> On Tue, 22 May 2012 01:35:14 GMT, unruh wrote:
>>> I am running a web server on a Mandriva 2010.2 system.
>>
>> That is getting a bit long in the tooth.
>> Hope you kept up with all the security updates.
>>
>>> The other suspicious thing is that there was 6 hour gap in
>>> the logs
>>
>> That does not look good. Can we assume you checked all the logs for same gap.

The other logs (eg /var/log/messages) have no such gap.
>>
>> A cursory look at https://isc.sans.edu/diary.html?date=2012-05-08
>> and http://www.mandriva.com/en/support/security/advisories/?dis=2010.1
>> seems to show Mandriva a bit behind in patches.
>>
>> If I did not have something like aide, osiris, ossec-hids, samhain,
>> tripwire, snare, integrit,... I would be a bit nervous about the
>> integrity of the system.
>
> The problem definitely seems to be the things like
>
> 66.249.68.198 - - [21/May/2012:20:12:34 -0700] "GET /aggregator/www.guardian.co.uk/media/2011/jun/29/www.nytimes.com/2012/04/20/world/middleeast/syria-united-nations-secretary-general-ban-ki-moon-cease-fire.html?page=34 HTTP/1.1" 200 76079 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
>
> which seem to stay up forever. Of course my system has no such file. The
> address is definitely claimed to be google by whois.
> But these requests seem to be hanging that httpd which is trying to
> answer them. And they come back up almost immediately if I shut down
> httpd.
>

Ie, when I start apache, the httpd count is 10 instances. after a
minute, it is up to 17 and stays there. And the 7 GET requests are of
that above type. Those GET requests make no sense whatsoever to me. I
certainly have no such files on my system, and what in the world are
they doing running guardian, nytimes, together like that?


J.O. Aho

unread,
May 22, 2012, 12:57:40 AM5/22/12
to
On 22/05/12 03:35, unruh wrote:
> I am running a web server on a Mandriva 2010.2 system. Today the system
> suddenly became increadibly non-responsive. When I finally logged on as
> root, the swap file was up around 3GB, (no wonder response was horrible)
> and there were 160 instances of httpd running. After I finally managed
> to shut them down, (killall -9 httpd) response was restored. I looked in
> the /var/log/httpd/access_log there did not seem to be much unusual
> there. There were some google.com with weird addreses, and some internal
> connections which were the only things that looked out of the oridinary.

Tweak your apache settings, so that no 160 instances will be started,
with some google you should be able to find some setting recommendation
formulas.

I think I had some issue with an earlier wordpress version (pre 3.3),
which got really strange, specially when it tried to access a page which
didn't exist and it came into a horrible loop (yes, wordpress may
request pages itself).

> Eg
> 66.249.68.198 - - [21/May/2012:18:19:23 -0700] "GET /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/bit.ly/www.guardian.co.uk/business/2012/may/04/pay-vince-cable HTTP/1.1" 200 72658 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
> 66.249.68.73 - - [21/May/2012:18:19:26 -0700] "GET /aggregator/www.nytimes.com/2012/04/05/opinion/node/node/www.bbc.co.uk/news/uk-17769717?page=226 HTTP/1.1" 200 38984 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
>
> and
>
> ::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
> "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
> connection)"

This is all okey, nothing to worry about, if you don't want to see them,
there is an instruction how to filter those at apache.org.
Has nothing to do with the increased number of services loaded.


> But could any of these be respoinsible for 160 connections?
>
> The other suspicious thing is that there was 6 hour gap in
> the logs

That don't sound good at all, could have been caused by the swapped out
syslog, but I would think it's wise to take a look at your system with
rkhunter and/or chkrootkit. Also check your other logs and see if you
miss 6 hours.


--

//Aho

Aragorn

unread,
May 22, 2012, 1:04:56 AM5/22/12
to
On Tuesday 22 May 2012 03:35, unruh conveyed the following to
alt.os.linux...
Possibly, yes. That's a denial of service, but I'm not enough of an
expert to know whether that was a external DDoS attack, or whether
Apache DoS'ed itself due to some bug.

> The other suspicious thing is that there was 6 hour gap in
> the logs

That's not suspicious. That's normal. The system simply doesn't have
the resources anymore to commit data to the log. Back when I was still
running an IRC network with a number of people, we were DDoS'ed too
several times, and then we saw this happening too.

> 112.111.174.175 - - [21/May/2012:11:34:11 -0700] "GET /user/register
> HTTP/1.0" 200 29860 "http://emergentgravity.org/user/register"
> "Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) Opera 6.01 [en]"
> ::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
> "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
> connection)"

Windows 95??? Okay, that starts to sound like a DDoS. I'm guessing
your server made acquaintance with a botnet.

--
= Aragorn =
(registered GNU/Linux user #223157)

Wolfram Gloger

unread,
May 22, 2012, 5:24:38 AM5/22/12
to
unruh <un...@invalid.ca> writes:

> I am running a web server on a Mandriva 2010.2 system. Today the system
> suddenly became increadibly non-responsive. When I finally logged on as
> root, the swap file was up around 3GB, (no wonder response was horrible)
> and there were 160 instances of httpd running.

If you are running apache versions from 2010, this sounds suspiciously
like you have been a victim of

http://httpd.apache.org/security/CVE-2011-3192.txt

It would be best to upgrade to newer binaries then.

Regards,
Wolfram.

unruh

unread,
May 22, 2012, 11:21:30 AM5/22/12
to
Thanks. I had partially upgraded to 2.2.22, but not all of the installed
apache stuff had been upgraded. I have now done so and will watch to see
if I get further problems. So far so good.

It seems that the attack vector is via those weird google searches. Ie,
they also seem to be spoofing google bot requests as part of the attack.


>
> Regards,
> Wolfram.
0 new messages