[AOLSERVER] Site hanging

[Janine Ohmer]

Another day, another problem. ;)

This time it's a somewhat more recent (but still old) OpenACS site, that has always run under AOLserver 4.0.10. Overnight it got hit pretty hard by the Yahoo crawler, and it has stopped responding. Netstat shows 117 connections, 78 in CLOSE_WAIT state and the rest in SYN_RECV.

This site would hang up very occasionally on my old system; I would just restart it, since it didn't happen often enough to worry about. But it has only been running for about 16 hours, and that is way too soon to be doing this.

The new system is more powerful than the old one, and runs a much newer 64 bit Linux, vs a very old 32 bit one. It's entirely possible that my configuration (connections) is not ideal, but it hasn't changed so I would expect it to run at least as well on the new system as it did on the old.

For now I will just restart it but I'm sure this will happen again. Any suggestions on how to troubleshoot? I have never had much luck over the years figuring this out; I usually just end up installing keepalive to restart, but that is really not ideal.

thanks,

janine

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Alexey Pechnikov]

Haproxy and other load-balancers and reverse-proxy can limit connections to backends. As example (only 10 connections to backend):

global

        maxconn 20000

...

frontend f_test

        bind 127.0.0.1:80

        acl test  hdr_sub(host) test

        use_backend b_test   if test

...

backend b_test

        # backend servers

        server main 127.0.0.1:8080 maxconn 10

2010/11/6 Janine Ohmer <jan...@furfly.net>

--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/

[Gustaf Neumann]

Dear Janine

The symptoms sound similar to bug #1615787
https://sourceforge.net/tracker/?func=detail&aid=1615787&group_id=3152&atid=103152
this bug is fixed in the 4.0 branch (not sure, if you are using
a version before this fix) and in aolserver 4.5.1

best regards
-gustaf neumann

2007-10-19 Gustaf Neumann neu...@wu-wien.ac.at

* nsd/driver.c:
* nsd/queue.c:

Better fix for BUG #1615787 in aolserver 4.5.
Although the
original fix apparently helped for www.openacs.org, it
appears to simply change the timing, but did not fix the
problem itself. The problem was that under limited
resource
configurations (e.g. maxthreads 5, maxconnections 3) and
heavy traffic, incoming requests were queued but not
processed in new connection threads. The situation was
especially bad, when the number of queued requests was
larger than maxconnections, since after processing n
requests the server was idling with a high number of
queued
requests.

2006-12-15 Dossy Shiobara <do...@panoptic.com>

* nsd/driver.c: [BUG #1615787] Fix deadlock bug when driver
thread stalls under specific conditions. Thanks to Jeff
Rogers and Gustaf Neumann.

[Janine Ohmer]

Thanks Gustaf! That very well could be the problem. I fixed it by increasing the number of database connections, which might just have the effect of preventing me from getting to this point.

janine

---
Janine Ohmer (formerly Sisk)
President/CEO of furfly, LLC
503-693-6407