[AOLSERVER] AOLserver Crash!

[Rami Jadaa]

Hello Everyone,

We are running multiple instances of AOLserver on different machines, and I am enjoying the reload functionality to reload the proc libraries using ns_eval source {fileName} in each one of them...

However, one of the AOLservers crashes after few minutes from the reload.

The strange thing is that this is the only AOLserver that crashes, while others don't!!! and I noticed that just before the crash, the following error happens (which means something in the C breaks, and I am assuming that it could be in the TCL interpter, Curently tcl 8.4.16  ( not AOLserver...But this is only an assumption):

"called Tcl_CreateHashEntry on deleted table"

We use this  server to serve multiple domains and have a  pound load balancer in the front  , For example if the request comes for www.xyz.com we serve xyz service related site and contents and if the request comes for www.abc.com we serve abc related contents and site. In total we are serving around 25 different sites like this . We are not using any virtual hosting module or feature of Aolserver . The total traffic of the server is not high .

Any idea anybody!!! Have anyone using the reload functionality noticed that it could crash the AOLserver?

Environment :
Aolserver 4.0.10 , fetched from CVS almost 6 months back .
nsoracle Oracle Driver version 2.8a1
nsmysql CVS
Oracle 10gR2  Libraries
AMD x86_64 RHEL 4
Curently tcl 8.4.16 also tried tcl 8.4.11


Please help as this is driving me crazy :(

Thanks in advance

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Scott Goodwin]

Rami,

Tcl is attempting to create a new hash table entry on a hash table that was either never created or was created but has ceased to exist -- most likely the pointer to that hash table is null or corrupted. This could be something in AOLserver that uses the Tcl_Hash* API. First steps:

1. Send a copy of the nslog output for a clean startup through to the point where it crashes; that might indicate where it's getting fouled up. If that portion of the nslog is not very long (say no more than 100-150 lines) you can cut and paste into the message; otherwise attach it as a separate file (but limit it to the smallest necessary size -- don't want multimegabyte files).

2. Do a checksum of all your own Tcl code files used by AOLserver on a known good machine and those same Tcl files on the bad one; compare the two outputs to see what Tcl files on the bad machine differ from the good one. Investigate those differences.

/s.

[Rami Jadaa]

Hi Scott,
Thanks for your reply.

I don't think that I can send the log as it will be so big , as AOlserver initiates and load a lot of ACS code...

And for the checksum, we did the following:
Using pound, we shifted the load going to this webserver to another server on another machine where it uses a different local copy of the same application, and then after the reload, the server were we shifted the load to crashed, and the old one didn't!!
So i can take out he doubt on file corruption, right?

[Scott Goodwin]

It appears that you have the same problem in all of your servers; the goal is to find out what part of the code is failing and under what conditions. Three things stand out: failed servers are under a heavier load than those that don't exhibit the failure; the failure happens shortly after shifting a load via pound onto an already running aolserver instance; the failure happens after a reload of your procs on that server. Since you've said that none of the loads is heavy I don't think this problem is triggered by aolserver being overwhelmed with traffic. This leaves two things: the shifting of the load itself to an already running aolserver instance and the reloading of the procs on that aolserver instance. I suspect the problem is related to reloading the procs with ns_eval, and not due to load shifting or load volume, but we need to confirm that. 

Is there a way you can run an aolserver instance directly answering queries without using pound? Maybe you could set up a test server that you then use http_load or apache bench on. Once running, hit it with a load and see if it stays up for at least 10-20 minutes. If it does, do a reload of your procs on that server without doing anything else -- what I expect is that the aolserver instance will crash shortly after doing the proc reload. You can then restart the server and try it again, this time reloading the procs immediately. Then repeat, but reload the procs after 5 minutes or so. In each case, determine how long it takes the server to crash after the proc reload (make sure the aolserver instance has started and continues to server connections before, during and after the reload).

If anyone else is experiencing the same problems, please post your information along with your configuration.

/s.

[Juan José del Río]

Hello,

>From my experience, I think the problem may be related with the 64 bits.

I've servers with AOLServer 32 bits, and AOLServer 64 bits, and I have
seen 64 bits growing faster in memory (and even not decreasing through
time), until it takes a considerable amount of memory (then I have to
restart it). Also, around each 2-3 hours, AOLServer will go wild and eat
100% of one CPU core for around 1 minute... but will continue serving
requests slower than usual.

My 32 bits server is a FreeBSD 7, and my 64 bits server is an up-to-date
Debian Linux. I don't know if it has something to do with the OS or with
the 32/64 bits, but the fact is that my Debian Linux 64 bits gives
problems that the FreeBSD 32 bits doesn't give.

Regards,

Juan José

-
Juan José del Río |
(+34) 616 512 340 | juan...@simpleoption.com

Simple Option S.L.
Tel: (+34) 951 930 122
Fax: (+34) 951 930 122
http://www.simpleoption.com

[Scott Goodwin]

Hi Juan,

That's a good point. I noticed Rami was hosting on 64 bit AMD systems
and it is possible that if he were running on a 32 bit architecture
the problems he's experiencing might not surface. This could mean a
problem with Tcl on 64 bit or something specific to AOLserver. I
suspect there are several 64 bit issues with AOLserver that need to be
resolved. I'll look into purchasing some 64 bit hardware soon to test
with.

/s.

[Gustaf Neumann]

Rami,

it looks to me as if the problem is due to a c-extension you are using
and happens after a thread exit. When a thread exists, it frees among
other things the associated tcl interpreter. At this time, all c
extensions have to unload cleanly as well. Note that ns_eval creates
and destroys as well a thread. The problem might be due to some leftover,
due to uninitialized memory, etc. This is inherently cumbersome to
debug in C code, since sometimes, you might be "lucky" and
the server survives the real bug, some other times not. Furthermore,
the bug and the crash are often in two different parts of the code.
Maybe, you have simply "luck" on the other machines.

i would recommend the following:
a) try to make the crash happen reproducible, in as simple as
possible setup. I would recommend to stress thread destroys
(e.g. setting maxconnections to e.g. 2, test with calls
doing an ns_eval)
b) reduce all c-extensions (do you have to use nsmysql and nsoracle?)
In the best of all possible worlds, you might not need all
c-modules in the crash case, so dropping might help
to detect the culprit.

From some distant, my first suspicion falls to nsmysql, i am not
sure how frequent this is used.

hope this help a little
-gustaf neumann

Rami Jadaa schrieb:

> Hi Scott,
> Thanks for your reply.
>
> I don't think that I can send the log as it will be so big , as
> AOlserver initiates and load a lot of ACS code...
>
> And for the checksum, we did the following:
> Using pound, we shifted the load going to this webserver to another
> server on another machine where it uses a different local copy of the
> same application, and then after the reload, the server were we
> shifted the load to crashed, and the old one didn't!!
> So i can take out he doubt on file corruption, right?
>
>
>
>>

>> Environment :
>> Aolserver 4.0.10 , fetched from CVS almost 6 months back .
>> nsoracle Oracle Driver version 2.8a1
>> nsmysql CVS
>> Oracle 10gR2 Libraries
>> AMD x86_64 RHEL 4
>> Curently tcl 8.4.16 also tried tcl 8.4.11
>>
>>
>> Please help as this is driving me crazy :(
>>
>> Thanks in advance
>>
>> -- AOLserver - http://www.aolserver.com/
>>
>> To Remove yourself from this list, simply send an email to

>> <list...@listserv.aol.com <mailto:list...@listserv.aol.com>>

>> with the body of "SIGNOFF AOLSERVER" in the email message. You
>> can leave the Subject: field of your email blank.
>>
>
> -- AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to

> <list...@listserv.aol.com <mailto:list...@listserv.aol.com>>

[SUBSCRIBE AOLSERVER Bjoern Kiesbye]

Hello,

I disoverd problems with Aolserver when tcl (8.5) is configured with the option --enable-64-bit , if I configure tcl with out this option, Aolserver runs fine on a  OpenSuse 10.3  64-bit System. If you are using a precompiled tcl package for a 64-bit System, it is likely that tcl was configured with this option.

You wrote you are using Aolserver 4.0.10, as far as i know its an older release of Aolserver 4 , I'm using Aolserver 4.5.0 upgrading may solve your problem.
I think v40_r10 (the version you are using) is the tag which is pointed out in the openacs installation documentation, the current release is v45_r0 , I think. .   

good luck,

Bjoern

-----Ursprüngliche Mitteilung-----
Von: Juan José del Río <juan...@SIMPLEOPTION.COM>
An: AOLS...@LISTSERV.AOL.COM
Verschickt: Mi., 29. Okt. 2008, 12:26
Thema: Re: [AOLSERVER] AOLserver Crash!

---

AOL eMail auf Ihrem Handy!Ab sofort können Sie auch unterwegs Ihre AOL email abrufen. Registrieren Sie sich jetzt kostenlos.

[Rami Jadaa]

Hello all,
Thanks for your support..
I am writing here the last updates, as -I think- we know what was crashing the AOLserver:
1- In all  AOLserver machines, is procedure is run on the fly to track user actions through the site (Like registrations, log in...moving around between links).
2- This procedure writes these operations to InnoDB tables in MySQL, and it runs in both, crashing AOLserver and non-crashing ones.
3- As I said before, the crashing AOLserver is the only one serving multiple sites.
4- Bearing in mind Nuemann advice about nsmysql, we disabled the action tracking procedure to prevent further writes to MySQL.
5- We applied a reload..and..BANG.. "no" Crash happened!
6- Hence the conclusion that we had is that it _something_ to do with nsmysql -especially that no further updates is done on that module.

So any idea what could be there in nsmysql that would crash AOLserver?

Thank you all!

[Dossy Shiobara]

Rami Jadaa wrote:
> So any idea what could be there in nsmysql that would crash AOLserver?

Are you using the latest nsmysql (CVS HEAD)?

Can you get a core dump from the crash and a gdb backtrace?

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)