Apache crashes every Sunday like a clockwork

[Toth, Csaba]

Hey Linux Gurus,

I wonder about your advices how I can solve this.
Our lab's web server at Vandy crashes every Sunday at 7:41am exactly.
I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS Precise.
Apache has php5 and even Drupal. We don't use Drupal and very little php.

What I suspect that it's some scheduled task doing the harm.
The Anachron weekly starts at 7:30am. After that it start the cron.weekly jobs with 10 minutes separation. Cron.weekly:
"0anacron
apt-xapian-index
debsums
man-db
sysklogd"

7:41 crash means that it can be the second thing it starts. I don't know if 0anacron counts or not. Debsums: this is a package with which you can identify damaged package. (The server had HDD hardware failure during the summer. That was degrading for a while, and we had logical file system issues too. I was down at the Medical Center DataCenter and I had to issue more fscks to recover the filesystem fully. I'm even happy that I could save the RAID volume, since it fell apart for sometime too.)
I went ahead and started all scripts in cron.weekly manually from a root command prompt, but nothing caused a crash. More info about the crash: I cannot see anything in the Apache access logs for 7:41am, the last log is for 7:37am.

Error log:

"[Sun Mar 10 01:29:09 2013] [error] [client 108.222.66.23] File does not exist: /var/www/www.hiplab.org/favicon.ico
[Sun Mar 10 01:50:02 2013] [error] [client 66.249.76.209] File does not exist: /var/www/www.hiplab.org/people/malin/BMIF380
[Sun Mar 10 03:36:04 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico
[Sun Mar 10 03:36:08 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico
[Sun Mar 10 03:36:40 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico
[Sun Mar 10 03:52:46 2013] [error] [client 199.30.20.73] File does not exist: /var/www/www.hiplab.org/people/weiyixia/icon.jpg
[Sun Mar 10 04:46:58 2013] [error] [client 78.162.114.93] File does not exist: /var/www/www.hiplab.org/favicon.ico
[Sun Mar 10 06:52:30 2013] [error] [client 69.163.180.32] File does not exist: /var/www/www.hiplab.org/wp-admin
[Sun Mar 10 07:05:50 2013] [error] [client 69.163.187.202] File does not exist: /var/www/www.hiplab.org/wordpress
[Sun Mar 10 07:26:48 2013] [error] [client 69.163.148.155] File does not exist: /var/www/www.hiplab.org/wordpress
[Sun Mar 10 07:41:47 2013] [notice] Graceful restart requested, doing restart
[Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: read() from pipe failed (0)
[Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
*** glibc detected *** /usr/sbin/apache2: corrupted double-linked list: 0x00007fda105bae40 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fda0e8b1b96]
/lib/x86_64-linux-gnu/libc.so.6(+0x7eec8)[0x7fda0e8b1ec8]
/lib/x86_64-linux-gnu/libc.so.6(+0x80436)[0x7fda0e8b3436]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x75)[0x7fda0e8b5fb5]
/lib/x86_64-linux-gnu/libc.so.6(+0xbb50b)[0x7fda0e8ee50b]
/usr/lib/libapr-1.so.0(apr_dir_open+0x29)[0x7fda0ee24e76]
/usr/sbin/apache2(ap_process_resource_config+0xed)[0x7fda0f71030d]
/usr/sbin/apache2(+0x35aaa)[0x7fda0f702aaa]
/usr/sbin/apache2(+0x40741)[0x7fda0f70d741]
/usr/sbin/apache2(+0x42304)[0x7fda0f70f304]
/usr/sbin/apache2(ap_build_config+0x9c)[0x7fda0f70f8cc]
/usr/sbin/apache2(+0x42d01)[0x7fda0f70fd01]
/usr/sbin/apache2(ap_process_resource_config+0x209)[0x7fda0f710429]
/usr/sbin/apache2(ap_read_config+0x2df)[0x7fda0f71104f]
/usr/sbin/apache2(main+0xbb6)[0x7fda0f6f93d6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fda0e85476d]
/usr/sbin/apache2(+0x2c559)[0x7fda0f6f9559]
======= Memory map: ========"

In the beginning you can see nice tries from bots trying to grab vulnerabilities: mofo bots tirelessly try to seek for admin consoles. Then at 07:41:47 restart requested, and apache segfaults. Crash dump is generated also.

One question: what else can start regularly weekly which is not anacron?
Any other advice is welcome, I am clueless at this moment.

Thanks,
Csaba

[andrew mcelroy]

On Sun, Mar 10, 2013 at 8:18 PM, Toth, Csaba <csaba...@vanderbilt.edu> wrote:
> Hey Linux Gurus,
>
> I wonder about your advices how I can solve this.
> Our lab's web server at Vandy crashes every Sunday at 7:41am exactly.
> I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS
> Precise.
> Apache has php5 and even Drupal. We don't use Drupal and very little php.
>
> What I suspect that it's some scheduled task doing the harm.
> The Anachron weekly starts at 7:30am. After that it start the cron.weekly
> jobs with 10 minutes separation. Cron.weekly:
> "0anacron
> apt-xapian-index

Is it possible that apt-xapian-index is causing a cpu spike and
somehow causing apache to have that behavior?
Do you have any logs on xapian?

http://forums.linuxmint.com/viewtopic.php?f=90&t=82985

Andrew McElroy

> --
> --
> You received this message because you are subscribed to the Google Groups
> "NLUG" group.
> To post to this group, send email to nlug...@googlegroups.com
> To unsubscribe from this group, send email to
> nlug-talk+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nlug-talk?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "NLUG" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nlug-talk+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[andrew mcelroy]

On Sun, Mar 10, 2013 at 8:26 PM, andrew mcelroy <soph...@gmail.com> wrote:
> On Sun, Mar 10, 2013 at 8:18 PM, Toth, Csaba <csaba...@vanderbilt.edu> wrote:
>> Hey Linux Gurus,
>>
>> I wonder about your advices how I can solve this.
>> Our lab's web server at Vandy crashes every Sunday at 7:41am exactly.
>> I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS
>> Precise.
>> Apache has php5 and even Drupal. We don't use Drupal and very little php.
>>
>> What I suspect that it's some scheduled task doing the harm.
>> The Anachron weekly starts at 7:30am. After that it start the cron.weekly
>> jobs with 10 minutes separation. Cron.weekly:
>> "0anacron
>> apt-xapian-index
>
> Is it possible that apt-xapian-index is causing a cpu spike and
> somehow causing apache to have that behavior?
> Do you have any logs on xapian?
>
> http://forums.linuxmint.com/viewtopic.php?f=90&t=82985
>
> Andrew McElroy

related:
There was an old bug where the mod_python would deadlock due to xapian.

[Kent Perrier]

First thing I would try is removing PHP. If you are not using it that much why suffer the security issues? Depending on how it is being use can you move that functionality to a standard cgi process? If you can't remove it, implement modsecurity to see if that does anything to help.

Are you sure you have not been compromised through a PHP security issue?

One other thing to try would be to stand up a new webserver, copy the content and functionality over and put it up in its place. See if the issue happens on the new server. If it does, then I would think that points to an external issue exploiting a configuration issue. Its an interesting issue, that's for sure!

Kent

[Andrew Farnsworth]

Try restarting apache yourself and check the logs.  It is possible something is cycling apache (maybe for log rotation) and that FastCGI is preventing it from closing down cleanly. If you restart apache yourself and get the same error, it should lead you down the right path.

I will second stripping out everything you don't actively use.  Not just for security purposes but for simplicity in management.

Andy

[David R. Wilson]

What about trying to renice the task before the other routines get the
chance to get a lot of CPU time?

It does sound strange that it would crash on a predictable basis.
If Drupal is not being used, I would consider trying to remove it and
see what happens.

Dave

> Then at07:41:47 restart requested, and apache segfaults. Crash dump is

> generated also.
>
> One question: what else can start regularly weekly which is not
> anacron?
> Any other advice is welcome, I am clueless at this moment.
>
> Thanks,
> Csaba
>
>

> --
> --
> You received this message because you are subscribed to the Google
> Groups "NLUG" group.
> To post to this group, send email to nlug...@googlegroups.com
> To unsubscribe from this group, send email to nlug-talk

> +unsub...@googlegroups.com

[Perkins, Jerry]

On 03/10/2013 08:18 PM, Toth, Csaba wrote:

Our lab's web server at Vandy crashes every Sunday at 7:41am exactly.

   Interesting, my Debian did the same thing about 5 years ago for a few weeks.   Update took care of it.

--
    Jerry Perkins

          Home page   http://jperkins.us

[Drew from Zhrodague]

On 3/10/13 9:18 PM, Toth, Csaba wrote:
> Hey Linux Gurus,
>
> I wonder about your advices how I can solve this.
> Our lab's web server at Vandy crashes every Sunday at 7:41am exactly.
> I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1
> LTS Precise.
> Apache has php5 and even Drupal. We don't use Drupal and very little php.

/etc/logrotate.d/apache or /etc/logrotate.d/httpd


--

Drew from Zhrodague
lolcat divinator
dr...@zhrodague.net

[Howard Coles Jr]

Along with all the others, if you are not using it remove it.

Also, I would look at every user's crontab file, as well as root's. Then do what you did to root's crontab file. It may not be any one by itself, but it may be a combo of user a's and root's. If any of the commands are scripts, check to see if they're setting up at scheduled jobs.

Also look for any recurring "at" scheduled jobs. running "atq" as root (sudo atq" may do it) will give you a list of all pending jobs. "at" is generally a "once and done" kind of scheduler, however, a script may be resetting it if you can't find the job in cron. I used it to do some dynamic scheduling for TSM server processes.

--

See Ya'

Howard Coles Jr.

John 3:16

[Michael Chaney]

Here's a weird data point for you.  I have an ubuntu server, probably version 8 or 10, that has a few rails sites running under lighttpd.  It used to crash every two or three weeks at the same time.  Guess what time it crashed.

I never did track it down, I'd just restart it and go to church.

Something with ubuntu.

Michael

--
--
You received this message because you are subscribed to the Google Groups "NLUG" group.
To post to this group, send email to nlug...@googlegroups.com

To unsubscribe from this group, send email to nlug-talk+...@googlegroups.com

For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en
 
---
You received this message because you are subscribed to the Google Groups "NLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nlug-talk+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Michael Darrin Chaney, Sr.
mdch...@michaelchaney.com
http://www.michaelchaney.com/

[Toth, Csaba]

Here's a cure for the problem: look at the /etc/logrotate.d/apache2 file. It has a postrotate section, which does an /etc/init.d/apache2 reload.
This reload crashes, I tried it manually too, and it causes the same crash I see on Sundays. In the logs we can see also, that something starts to restart apache:
"Graceful restart requested, doing restart" before the crash. My cure is to simply remove that postrotate section and the apache2 reload. The whole issue was associated with the logrotate,
this runs weekly.
Apache2 restart works well, without a crash though. I tried to remove unnecessary stuff from apache config, because the real cause is why the apache crashes for a reload.

I can confirm my findings next Sunday morning. I have to say thanks for the best sysadmin I know, his everyday job is an MRI radiologist medical doctor: my twin brother Ati. I explained the problem and he knew the possible cause right away.

Csaba

---

From: nlug...@googlegroups.com [nlug...@googlegroups.com] on behalf of Michael Chaney [mdch...@michaelchaney.com]
Sent: Tuesday, March 12, 2013 9:27 PM
To: nlug...@googlegroups.com
Subject: Re: [nlug] Apache crashes every Sunday like a clockwork