Lots of CLOSE_WAIT connections

[Evan Chan]

Hi guys,

I have been using Iago to replay HTTP logs and test against production-level systems.  The distro I use has this commit at HEAD:

commit 41cf6a3ee5872c517ac14b948cfb85985a0cb558

Merge: 86c94d7 7f41856

Author: Tom Howland <thow...@twitter.com>

Date:   Tue Jan 13 13:30:39 2015 -0800

    Merge pull request #42 from mzalimeni/master

    Default maxRequests to Integer.MAX_VALUE rather than 1000

When I ramp up the request rate, I see many CLOSE_WAIT connections on my server, and the server becomes unresponsive.  This happens even after I kill the Iago processes (via -k in the options, as recommended by the docs).   The CLOSE_WAIT connections do not go away until after I reboot my service. 

Our service uses Jetty under the covers, and I haven't found barely any issues with Jetty folks and CLOSE_WAIT.

Could it be that Iago is somehow not closing or cleaning up connections properly?  Any ideas?

I'm planning to test the services using Apache bench or something else, but none of them quite test the system (using real logs) like Iago does.

thanks,

Evan

[James Waldrop]

These are your application (Jetty in this case) not closing the sockets. CLOSE_WAIT has nothing to do with the client side unfortunately. As to why this is happening, I'm unsure -- there are many reasons why Jetty might not be closing the socket on its end, up to and including spending all of its time in GC. Someone who is a Jetty expert may be able to help. I did find this report of what sounds like a very similar problem and it may help you:

https://jira.codehaus.org/browse/JETTY-1505

--

---
You received this message because you are subscribed to the Google Groups "Iago Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iago-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Evan Chan]

Thanks.  It turns out it was our app after all - rather the CLOSE_WAITs will eventually close, when giving the app enough time to finish things.

--

---
You received this message because you are subscribed to a topic in the Google Groups "Iago Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/iago-users/bE8ACxFLYIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to iago-users+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Evan Chan   |   Principal Engineer, Socrata     |    @evanfchan

Socrata | The Data Experience Company™

Socrata isn't about worldly success, but about changing the world successfully. This is why I'm excited to be here.

What does Socrata do? (watch our 2 min video)