Web server unavailable with lots of HTTP connections in CLOSE_WAIT state

[Leon Miller-Out]

Hi! Every couple of days, my Apache web server stops responding. When I look at the connections with `netstat`, I see that there are a large number* of TCP connections in the CLOSE_WAIT state. From my limited understanding of TCP, I believe this means that the HTTP client has indicated that it is done with the connection and has asked to terminate it, but my server has not agreed to terminate the connection. I'm guessing that the problem is more likely to be with my Tomcat or Lucee setup than a bug in Apache, hence my asking for help here. My current workaround is to restart Apache when this occurs.

Are there any error conditions that could result in Tomcat/Lucee leaking TCP connections?

Are there any settings I can change in Lucee or Tomcat to prevent connection leakage?

Thanks!

* "large number" means that there are about 10x the normal number in nominal operation

[Robert Munn]

Server Fault to the rescue:

http://serverfault.com/questions/160558/how-to-not-get-so-many-apache-close-wait-connections

Maybe you have a bug in your app that prevents connections from closing, e.g. an unhandled exception that leaves these connections in a zombie state. Might be worth reviewing app/error logs to see if something obvious appears.

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/5d6dbeed-8e84-430a-acec-179e31d8b974%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leon Miller-Out]

Thanks, Robert. I had seen that SO post, and I believe I upvoted that great answer!

Since CFML apps aren’t generally responsible for handling their own HTTP connections, I wonder what kinds of things one could do in CFML that could cause connection leakage.

I could try to correlate the last requests from each hung connection (by looking up their IP in the apache access log), but I’m not sure if their would be lines in the access log for these requests gone bad. It’s worth a shot, I suppose.

Leon

You received this message because you are subscribed to a topic in the Google Groups "Lucee" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lucee/OCHdtCx9Huc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/B83FBDD0-78BB-4E4E-99C5-0292D3408C2C%40gmail.com.

[Leon Miller-Out]

Update:

Our Apache proxies requests to a couple of other services (one being Lucee). We suspected that one of these services (probably the non-Lucee service) might be misbehaving and failing to disconnect in some cases, leaking connections. We added `ProxyTimeout 60` to our Apache config, and it seems like the server has been more stable since that change.

[Ken Redler]

Hey Leon,

If you're not already doing this, you can probably find the culprit (slightly laboriously) by adding timeouts line by line, then checking netstat. Something like this should be supported in (I think) Apache 2.4+:

    ProxyPassMatch ^/(.+\.cf[cm])$ ajp://localhost:8009/$1 timeout=60

 - Ken