Apache crash problem after
Massimo
yesterday night i added two new virtual host to my httpd.conf (for a new website).
Stopped apache, restarted apache do not start anymore.
I raised shellhandelsince in config.sys from 200 to 220.
Apache do not restart neither after reboot :(
This is a production server, so this is a really big problem (customers
do not see their websites, they are calling and complaining!!!...)
I pray someone to help me.
Thanks a lot in advance.
ERROR_LOG:
[Fri Dec 11 10:39:16 2009] [notice] Apache/2.2.13 (OS/2) mod_ssl/2.2.13
OpenSSL/1.0.0-beta3 PHP/5.2.8 configured -- resuming normal operations
[Fri Dec 11 10:39:21 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:25 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:27 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:32 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:33 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:33 2009] [notice] Digest: generating secret for digest authentication ...
[Fri Dec 11 10:39:34 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:36 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:36 2009] [notice] Digest: generating secret for digest authentication ...
[Fri Dec 11 10:39:36 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:36 2009] [notice] Digest: generating secret for digest authentication ...
[Fri Dec 11 10:39:37 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:37 2009] [notice] Digest: generating secret for digest authentication ...
[Fri Dec 11 10:39:38 2009] [warn] RSA server certificate is a CA certificate
(BasicConstraints: CA == TRUE !?)
[Fri Dec 11 10:39:38 2009] [notice] Digest: generating secret for digest authentication ...
[Fri Dec 11 10:39:38 2009] [notice] Digest: done
[Fri Dec 11 10:39:39 2009] [notice] Digest: done
[Fri Dec 11 10:39:39 2009] [notice] Digest: done
[Fri Dec 11 10:39:39 2009] [error] (OS 105)The previous ownership of this semaphore has
ended. : apr_socket_accept
[Fri Dec 11 10:39:39 2009] [error] (OS 105)The previous ownership of this semaphore has
ended. : apr_socket_accept
[Fri Dec 11 10:39:39 2009] [notice] Digest: done
[Fri Dec 11 10:39:39 2009] [error] (OS 105)The previous ownership of this semaphore has
ended. : apr_socket_accept
[Fri Dec 11 10:39:39 2009] [notice] Digest: done
[Fri Dec 11 10:39:39 2009] [error] (OS 105)The previous ownership of this semaphore has
ended. : apr_socket_accept
[Fri Dec 11 10:39:39 2009] [notice] caught SIGTERM, shutting down
--
Massimo S.
- eCS Irc Network
- irc.ecomstation.it 6667
- channels: #eCS #eCSita
- eComStation FREE demo CD download http://www.ecomstation.com/democd/
Lewis G Rosenthal
On Fri, 11 Dec 2009 10:45:41 +0100 "Massimo" <m...@ecomstation.it> wrote:
>Hi all,
>
>yesterday night i added two new virtual host to my httpd.conf (for a new
website).
>
>Stopped apache, restarted apache do not start anymore.
>
>I raised shellhandelsince in config.sys from 200 to 220.
>Apache do not restart neither after reboot :(
>
>This is a production server, so this is a really big problem (customers
>do not see their websites, they are calling and complaining!!!...)
>
>I pray someone to help me.
>Thanks a lot in advance.
>
>
>ERROR_LOG:
>
>[Fri Dec 11 10:39:16 2009] [notice] Apache/2.2.13 (OS/2) mod_ssl/2.2.13
>OpenSSL/1.0.0-beta3 PHP/5.2.8 configured -- resuming normal operations
>[Fri Dec 11 10:39:21 2009] [warn] RSA server certificate is a CA
certificate
>(BasicConstraints: CA == TRUE !?)
>
recently, but on NetWare). Are you sure the CA bundle matches the cert? Did
anything change related to SSL prior to reconfiguring for the new vhosts?
RSA server certificate is a CA certificate implies that the crt is not the
proper type in the expected place. I would review httpd-ssl.conf.
In fact,
http://www.question-defense.com/2008/10/26/rsa-server-certificate-is-a-ca-certificate-basicconstraints-ca-true
might help...
GL
___
Lewis G Rosenthal
Rosenthal a Rosenthal, LLC
Sent with SnapperMail
Massimo
The real error is this, and it's plain http (not https related):
[Fri Dec 11 10:39:39 2009] [error] (OS 105)The previous ownership of this semaphore has
ended. : apr_socket_accept
nothing to do with SSL
bye
massimo s.
>
> You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.
> To post to this group, send email to apa...@googlegroups.com.
> To unsubscribe from this group, send email to apache2+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/apache2?hl=en.
>
>
>
Lewis G Rosenthal
On Fri, 11 Dec 2009 15:21:40 +0100 "Massimo" <m...@ecomstation.it> wrote:
>No SSL/Certificate problems, apache crashes even fi i turn off the HTTPS
"service".
>
>The real error is this, and it's plain http (not https related):
>
>[Fri Dec 11 10:39:39 2009] [error] (OS 105)The previous ownership of this
semaphore has
>ended. : apr_socket_accept
>
>this error make apache2 not to start
>nothing to do with SSL
>
abnormally. I was seeing this on mine when I was experiencing my difficulty
with the MySQL queries under PHP.
Essentially, it's just telling you that it died. Now, we need to determine
the cause.
If you turn off SSL & turn up logging, what do you get?
Or...
If you simply remove the two new vhosts & try to start, what do you get?
Massimo
semaphore has ended. : apr_socket_accept
what i can do it's to raise the log verbosity, let me know if it can help
thanks a lot
bye
massimo
Lewis G Rosenthal
On Fri, 11 Dec 2009 16:05:04 +0100 "Massimo" <m...@ecomstation.it> wrote:
>already tried i still get this: [error] (OS 105)The previous ownership of
this
>semaphore has ended. : apr_socket_accept
>
>what i can do it's to raise the log verbosity, let me know if it can help
>
issue). Check your php.err log, & perhaps turn up logging there (see the
appropriate section of php.ini for settings).
As for Apache log level, review the docs at apache.org. You may already be
turned up all the way; I can't quite tell.
Assuming your Apache confs are readable & syntactically correct (you can
test this with httpd -t), the issue is probably with a module you're
loading. To me, SSL is the likely suspect, because that's the error in the
log (and because I just went through it when I replaced an expired cert on
one of my servers and forgot to update the CA bundle), but it *could* be
coming from somewhere else.
Massimo
Il 11/12/2009 20.13, Lewis G Rosenthal ha scritto:
> Sorry for the delay...traveling today...
>
> On Fri, 11 Dec 2009 16:05:04 +0100 "Massimo" <m...@ecomstation.it> wrote:
>> already tried i still get this: [error] (OS 105)The previous ownership of
> this
>> semaphore has ended. : apr_socket_accept
>>
>> what i can do it's to raise the log verbosity, let me know if it can help
>>
> It could be something in PHP which is taking it down, too (that was my
> issue). Check your php.err log, & perhaps turn up logging there (see the
> appropriate section of php.ini for settings).
>
> As for Apache log level, review the docs at apache.org. You may already be
> turned up all the way; I can't quite tell.
>
> Assuming your Apache confs are readable & syntactically correct (you can
> test this with httpd -t), the issue is probably with a module you're
> loading. To me, SSL is the likely suspect, because that's the error in the
> log (and because I just went through it when I replaced an expired cert on
> one of my servers and forgot to update the CA bundle), but it *could* be
> coming from somewhere else.
it's the same even with all the HTTPS stuff tunerd off
i thank you a lot for the help, i'm going to raise php and apache log levels
but please let's stop talking about https that's not the source of these problems
i had these problems since years ago and i'm using HTTPS since about six months...
of course i can't turn off HTTPS since i host services and webmail for clynics
that process sensible data
so anyway ho https, no ecomstation
thanks
bye
massimo
Massimo
LIBC PANIC!!
fmutex deadlock: Owner died!
0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
Desc="LIBC Heap"
pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
LIBC PANIC!!
fmutex deadlock: Owner died!
0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
Desc="LIBC Heap"
pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
LIBC PANIC!!
fmutex deadlock: Owner died!
0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
Desc="LIBC Heap"
pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
LIBC PANIC!!
fmutex deadlock: Owner died!
0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
Desc="LIBC Heap"
pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
the fault daemon tried to restart apache for several times, at 5 o'clock
also bkup of the server started, antispam (clamav) scan and other "heavy"
tasks, so the server locked (giving me a downtime of about 3 hours...)
i pray someone to help me
my back account is very low (-6900 euros)
i can't loose customers since apache2 or stunell continuosly crash or die!
hope someone can comprehend the serious and critical situation
i WANT to keep on using os2-ecomstation, but i can't suffer downtimes
expecially with this hard economical situation
Lewis G Rosenthal
>
> LIBC PANIC!!
> fmutex deadlock: Owner died!
> 0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
> Desc="LIBC Heap"
> pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
> D:\APACHE\BIN\HTTPD.EXE
> Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
>
>
Assuming there is no data corruption or other issue reading data from
the volume, *something* must have changed. If you can't figure out what,
then restore from a backup.
The *right* way to fix the problem is to use a tool such as Theseus to
find out what is happening in memory at the time of the crash. If you
don;t want to do that level of forensic work, then you'll need to take
the elephant gun approach and roll back either your Apache binary or
perhaps a very basic conf & slowly roll forward until you hit on what
triggers the condition.
That said, this looks suspiciously similar to what you reported back on
October 5: http://groups.google.com/group/apache2/msg/d2199720cde7bdd9? .
> also bkup of the server started, antispam (clamav) scan and other "heavy"
> tasks, so the server locked (giving me a downtime of about 3 hours...)
>
>
before giving up. Often, already taxed resources will become quickly
depleted by repeated attempts to re-run the same failing operation.
> my back account is very low (-6900 euros)
> i can't loose customers since apache2 or stunell continuosly crash or die!
>
>
>
>
provided are two entirely different things, sorry.
If you ask nicely, our good friend Steven Levine has drafted an
excellent (brief) guide to reporting Apache errors, including what
details are necessary and how much is reasonably expected of the
reporter in order to solicit assistance. I don't see it published on his
OS/2 Diagnostic Tools and Tips page yet
(http://home.earthlink.net/~steve53/os2diags/index.html ), but I'm sure
he'll provide you with a pre-release copy.
While the exe is HTTPD.EXE, that does not at all imply that the LIBC
report relates to it directly; merely that HTTPD is reporting the
condition. *Many* modules call on LIBC, and any *one* (or more) of them
could be causing this situation. This brief snippet doesn't give
*anyone* enough information to be able to suggest a real cause or how to
effect repairs. At the minimum, what's needed is a look at the relevant
conf's, the access log, other error logs (e.g., POPUPLOG.OS2, php.err,
etc.) and CONFIG.SYS. For accurate diagnosis with Theseus, you'll need
symbol files and process dumps. I know these things are difficult with
production systems, which is why I suggested restoring from backup or
rolling back conf and/or binary versions.
> expecially with this hard economical situation
>
>
then you might want to consider using something else. That said, Apache
is Apache; I've dealt with my share of Apache issues on NetWare, Linux,
and Windows, as well, so likely you'll just be trading one set of
variables for another. In the case of my last persistent crash, the
problem turned out to be (apparently) an errant MySQL db table and the
manner in which a particular PHP app (SITR) was trying to access it.
Thus, while my real problem *might* have had something to do with OS/2
(assuming the code was significantly different in the PHP port to that
platform than in the other platforms), the solution was on a completely
different server, running a completely different OS...
GL
--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE
Rosenthal & Rosenthal, LLC www.2rosenthals.com
Need a managed Wi-Fi hotspot? www.hautspot.com
Secure, stable, operating system www.ecomstation.com
-------------------------------------------------------------
Massimo
Il 16/12/2009 14.57, Lewis G Rosenthal ha scritto:
> On 12/16/09 05:27 am, Massimo thus wrote :
>> another crash
>>
>> LIBC PANIC!!
>> fmutex deadlock: Owner died!
>> 0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
>> Desc="LIBC Heap"
>> pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
>> D:\APACHE\BIN\HTTPD.EXE
>> Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
>>
>>
> <snip>
>
> Assuming there is no data corruption or other issue reading data from
> the volume, *something* must have changed. If you can't figure out what,
> then restore from a backup.
nothing changed, just two virtual hosts addd
no HD crc or problems
since it's a logical unit RAID1 mirroring on a LSI Logic 150-6 sata raid
i activated debug-log and i've seen some erros on reverse-proxy feature
since, lucklily, at the moment i don't need it
i've disabled reverse-proxy feature
now it works better, but see just a little bit better
since after server reboot it's allways quite difficult to get it running again
could the debug-log be of any interest or helpful?
thanks really a lot
massimo
Steven Levine
at 02:13 PM, Massimo <m...@ecomstation.it> said:
Hi Massimo,
Just a few comments that might help you track this down.
>>> another crash
Not really. The LIBC PANIC messages are a side effect of something else
failing.
>>> LIBC PANIC!!
>>> fmutex deadlock: Owner died!
>>> 0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
>>> Desc="LIBC Heap"
>>> pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
What this says is that a thread (TID 1) wanted access to the semaphore
that controls one of the heaps and could not get access because the
semaphore owner (TID 6) died while it owned the semaphore.
I suspect there is a message in the logs indicating why TID 6 died. Can
you find it?
>since after server reboot it's allways quite difficult to get it running
>again
What do you mean by this?
>could the debug-log be of any interest or helpful?
What would be interesting is to see the "missing" mesasge about what
happened to TID 6. While it's possible, it is unusual for code in LIBC to
die silently.
Since the error is in LIBC, perhaps you can convince Paul to do a build of
LIBC603 with logging enabled in the heap management code.
Steven
--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> eCS/Warp/DIY etc.
www.scoug.com www.ecomstation.com
----------------------------------------------------------------------
Massimo
Il 24/12/2009 21.14, Steven Levine ha scritto:
> In <4B3368F...@ecomstation.it>, on 12/24/09
> at 02:13 PM, Massimo <m...@ecomstation.it> said:
>
> Hi Massimo,
>
> Just a few comments that might help you track this down.
>
>>>> another crash
>
> Not really. The LIBC PANIC messages are a side effect of something else
> failing.
>
>>>> LIBC PANIC!!
>>>> fmutex deadlock: Owner died!
>>>> 0x0027013c: Owner=0x99850006 Self=0x99850001 fs=0x3 flags=0x0 hev=0x00010012
>>>> Desc="LIBC Heap"
>>>> pid=0x9985 ppid=0x98f5 tid=0x0001 slot=0x00ee pri=0x0200 mc=0x0000
>
> What this says is that a thread (TID 1) wanted access to the semaphore
> that controls one of the heaps and could not get access because the
> semaphore owner (TID 6) died while it owned the semaphore.
>
> I suspect there is a message in the logs indicating why TID 6 died. Can
> you find it?
nothing about tid 6 in the logs :(
>> since after server reboot it's allways quite difficult to get it running
>> again
>
> What do you mean by this?
when apache "die" in this way i find easy to reboot the server
after the reboot apache "die" again in the same way
a rexx fault-daemon detect that the process HTTPD isn't running
so it try to restart it automatically (it check every 5 minutes)
after a minimum of 3 o 4 restarts apache start to "live" again
don't ask me why this happens
it's a total mistery for me
>> could the debug-log be of any interest or helpful?
>
> What would be interesting is to see the "missing" mesasge about what
> happened to TID 6. While it's possible, it is unusual for code in LIBC to
> die silently.
>
> Since the error is in LIBC, perhaps you can convince Paul to do a build of
> LIBC603 with logging enabled in the heap management code.
>
> Steven
i switched down the reverse provy feature since i don't need it anymore
apache seems to work better, but it was just a "feeling"
the problems are still the same than before...
thanks
bye
massimo s.