close_wait connections locking up application server

[Lawrence Johns]

Hi,

I took over a scala/Lift project not knowing either before starting, so let me apologize in advance :)

I have two questions on which I would appreciate some guidance from those more knowledge in the arts of scala/lift/dispatch.

So, the Lift app I have built hits another service over HTTP to get all pertinent data for display/authentication/etc. ie. User clicks login -> http request with u/pw -> http handler receives true/false -> app repsonds to response accordingly. There is one object (which I inherited when I took over the project) that handles much of this and I am now questioning if its design/implementation is causing my troubles.  To trim it down to one basic call, the code is like this:

object Service {
  val http = new Http() with thread.Safety
  
  // actually sending u/pw over message body (log message above was just to make it clear)
  def login(user:String, password:String) = 
    http.when(code => match {
      case 200|201|202|203|204 => true
      case _  => Login.loginFailure(code)}}
    ) (loginUrl << compact(render(Map("username"->user, "password"->password))) ># {json =>
      //do stuff here
     })
}

 

1) So, I am noticing that the app is locking up due to the fact that CLOSE_WAITs are building up the server and eventually the app just spins unable to process the request.  I can see the following log message logged, but the app doesn't return: 

INF: [console logger] dispatch: [server] GET [path]/login?u=username&pw=password HTTP/1.1

It appears like the close waits are filling up the default pool size on the thread.Safety executor, then boom.

The questions for #1 are: Is this the right executor to be using? Is it being used correctly?  What is the "right" way to do it?  1000s of users, maybe 100 logged in at once clicking all over on a busy day.

My current plan of action:  It doesn't seem right to me; I was going to switch to the basic Http executor and construct a new one for every single call and shutdown() upon completion.

2) If my current plan of action is okay, what is a common pattern to implement that shutdown call (I feel like adding a line to save the reference in every method is probably not "right").  I was going to build another object that takes the when() and handler() parameters.  And that new object can create new Http, call it with the when() and handler(), then shutdown().  (maybe using case classes for specific response code handling via pattern matching eventually, but I think I'm to beginner to get my head around that right now :)).  Otherwise, I thought all the methods could stay like they are and there is a scala-y way to get the http executor to shutdown after that one call is finished.

Thanks in advance! 

[Lawrence Johns]

When I say "My current plan of action: It doesn't seem right to me".  I mean the way the current Service object is implemented doesn't seem right to me...  Switching to the basic Http executor seems more okay.

And, maybe it isn't obvious, but there are like 20 methods in that Service object that all use that same http val right now.  I should have put another comment after the login function. 

[Nathan Hamblen]

Hi Lawrence,

Sorry to take so long to reply, I'm just now catching up on Dispatch mail.

When you mix in thread.Safety you are getting this:
http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/tsccm/ThreadSafeClientConnManager.html

Apache Http components recommends the use of this connection manager with a single client so that connections can be reused, but in Dispatch and older versions particularly it is more common to create a new client for each thread. That worked pretty well in most cases, but then at some point the underlying Apache client changed such that it needed to be closed after use, making this mode of interaction cumbersome. Since then I've updated the documentation to encourage using one client with thread.Safety.

Another wrinkle is that I don't know what version of Dispatch you are using via Lift, but older versions of the underlying Apache client did not have the ability to set a higher maximum for default connections per route, making it really terrible for simultaneous requests to one server.

You are probably doing the safest thing by switching to one non-thread-safe client per connection, and you should also close it when finished.

And if you want to do a performance comparison that would be really interesting for us, there's this https://github.com/dispatch/reboot but it could take some work to adapt your code.

Nathan

[Lawrence Johns]

Awesome, Thanks for the reply! I will try to get you some details early this week.

[Lawrence Johns]

Dispatch Version Info (I can test updating this if 0.8.7 is available in the maven repository)

Maven dependency being used:

        <dependency>

            <groupId>net.databinder</groupId>

            <artifactId>dispatch-lift-json_2.9.1</artifactId>

            <version>0.8.5</version>

        </dependency>

Appears to be pulling in the following jars:

• dispatch-lift-json_2.9.1-0.8.5.jar 
• dispatch-core_2.9.1-0.8.5.jar 
• dispatch-http_2.9.1-0.8.5.jar 
• dispatch-futures_2.9.1-0.8.5.jar 

Application code:

Since you think the one client/connection would be the safest, I will work on finalizing my code base to explicitly close those connections.  Related to that (my second question), does anyone in this group use any cool scala-y patterns that are a good/common way to encapsulate the "when", "execute handler", and  "shutdown" logic?  If not, no biggie... but, being a newb, I worry that not coding in proper scala fashion can itself create problems.

Would starting down the path of the following pseudo code be "scala approved"?  =)  I may run into a bunch of difficulties getting it implemented due to variability of when functions, handlers, and responses (but, usually those are what lead me to finding new scala-y ways of doing things):

object HttpHelper {

  def http_close(when, handler) { 

    http = new Http

    var res = http.when(when)(handler)

    http.shutdown()

    res

  }

}

Performance comparison:

I can look into the dispatch/reboot to see if I can get you some stats.  We did use "web securify" as a quick way to throw a ton of login attempts at the application.  In non-reboot testing, the thread-safe client would obviously fail around 50 or so.  And, the "new client/connection" would run fine (we let the test run to about 30,000 hits) even letting the gc() clean up connections.  I can put together some test based on the web securify tool to compare with dispatch/reboot if that would be helpful until I have time to get a full performance testing  tool in place.

Thanks for your help, Nathan!  I appreciate all your hard work.

Larry

[Nathan Hamblen]

On 01/30/2012 12:44 PM, Lawrence Johns wrote:

Dispatch Version Info (I can test updating this if 0.8.7 is available in the maven repository)

That's unlikely to make a difference, 0.8.5 is new enough to be using the higher max connections per route.

Application code:

Since you think the one client/connection would be the safest, I will work on finalizing my code base to explicitly close those connections.  Related to that (my second question), does anyone in this group use any cool scala-y patterns that are a good/common way to encapsulate the "when", "execute handler", and  "shutdown" logic?  If not, no biggie... but, being a newb, I worry that not coding in proper scala fashion can itself create problems.

def withHttp[T](block: Http => T): T

would be a typical interface. The HttpHelper that you sketched out requires you do to more wrapping for whatever parts of Http you want to expose. I would just enable use of the actual Http type.

[...]

Performance comparison:

I can look into the dispatch/reboot to see if I can get you some stats.  We did use "web securify" as a quick way to throw a ton of login attempts at the application.  In non-reboot testing, the thread-safe client would obviously fail around 50 or so.  And, the "new client/connection" would run fine (we let the test run to about 30,000 hits) even letting the gc() clean up connections.  I can put together some test based on the web securify tool to compare with dispatch/reboot if that would be helpful until I have time to get a full performance testing  tool in place.

Another option would be to override the maxConnections defined in thread.Safety [1] to some number higher than 50. In the hundreds perhaps? Apache's defaults of 2 per route and 20 in total always seemed absurdly low to me, but since I don't know why they set them so low I wasn't comfortable setting them very much higher.

[1]: https://github.com/dispatch/dispatch/blob/master/http/src/main/scala/thread/thread.scala#L10

Nathan