SRVE0120E: IO Error java.net.SocketException: Connection reset

[park...@rogers.com]

Hey All,

I've seen alot of posts pertaing to connection resets with a port number....but the message we get below does not associate with a port number .. Does anyone have an idea what might be casuing the message below?

I would say we get this message at least 20 times a day, and every few days we experience a massive slow down effecting all our servers but it auto clears after 30 to 45 minutes and the server resumes normal operations.

[10/12/07 2:33:23:067 GMT+08:00] 5caaa915 SRTServletReq E SRVE0120E: IO Error java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
at com.ibm.ws.io.Stream.read(Stream.java(Compiled Code))
at com.ibm.ws.io.ReadStream.read(ReadStream.java(Compiled Code))
at com.ibm.ws.http.ContentLengthInputStream.read(ContentLengthInputStream.java(Compiled Code))
at com.ibm.ws.io.ReadStream.read(ReadStream.java(Compiled Code))
at com.ibm.ws.webcontainer.http.HttpConnection.read(HttpConnection.java(Inlined Compiled Code))
at com.ibm.ws.webcontainer.srp.SRPConnection.read(SRPConnection.java(Compiled Code))
at com.ibm.ws.webcontainer.srt.SRTInputStream.read(SRTInputStream.java(Compiled Code))
at com.ibm.ws.webcontainer.srt.http.HttpInputStream.read(HttpInputStream.java(Compiled Code))
at java.io.InputStream.read(InputStream.java(Inlined Compiled Code))
at com.ibm.ws.webcontainer.srt.SRTServletRequest.finish(SRTServletRequest.java(Compiled Code))
at com.ibm.ws.webcontainer.srt.SRTConnectionContext.finishConnection(SRTConnectionContext.java(Compiled Code))
at com.ibm.ws.webcontainer.srt.WebAppInvoker.doForward(WebAppInvoker.java(Compiled Code))
at com.ibm.ws.webcontainer.srt.WebAppInvoker.handleInvocationHook(WebAppInvoker.java(Compiled Code))
at com.ibm.ws.webcontainer.cache.invocation.CachedInvocation.handleInvocation(CachedInvocation.java(Compiled Code))
at com.ibm.ws.webcontainer.cache.invocation.CacheableInvocationContext.invoke(CacheableInvocationContext.java(Compiled Code))
at com.ibm.ws.webcontainer.srp.ServletRequestProcessor.dispatchByURI(ServletRequestProcessor.java(Compiled Code))
at com.ibm.ws.webcontainer.oselistener.OSEListenerDispatcher.service(OSEListener.java(Compiled Code))
at com.ibm.ws.webcontainer.http.HttpConnection.handleRequest(HttpConnection.java(Compiled Code))
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java(Compiled Code))
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java(Compiled Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))

When we experience a massive slowdown the following message appears...

[10/1/07 21:16:50:150 GMT+08:00] 65245a53 WSJdbcConnect W DSRA9400E: Fatal error
occurred during Connection reassociation: com.ibm.websphere.ce.j2c.ConnectionWa
itTimeoutException: Connection not available, Timed out waiting for 208309
at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePo
ol.java(Compiled Code))
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Comp
iled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager

Any ideas or direction to something woul be much appreciated...Thanks

[Ben_]

> [10/12/07 2:33:23:067 GMT+08:00] 5caaa915 SRTServletReq E SRVE0120E: IO
> Error java.net.SocketException: Connection reset

Typically, an impatient / upset end-user cancelled the navigation (browser
aborts the connection and WebSphere observes a connection reset).

> When we experience a massive slowdown the following message appears...
>
> [10/1/07 21:16:50:150 GMT+08:00] 65245a53 WSJdbcConnect W DSRA9400E: Fatal
> error
> occurred during Connection reassociation:
> com.ibm.websphere.ce.j2c.ConnectionWa
> itTimeoutException: Connection not available, Timed out waiting for 208309

Database connection pool is exhausted (all connections are in use) -->
Application blocks waiting for a connection --> Application slow or
returning an error --> End-user hits the stop button --> Message #1 is
logged.

This can help spot who is holding the connections:
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21203225.

But if it may also very we a "normal" peak due to a more intensive use (many
people arriving at 8:00 AM or many people surfing your site in the evening
after the movie). In this case, it's nothing but a sizing problem.

[Ken Hygh]

In addition to what Ben said, you might also look at your network
between the HTTP server and the WAS server - you could be overrunning
that network, or there could be something misconfigured somewhere.

Ken

[park...@rogers.com]

Thanks Ben and Ken......Our HTTP servers reside in Hong Kong and were connecting from the united states. Dont ask why? I'm still scratching my head as to why they would set this up in this way. Darn project managers!!!!

Anyways, our average ping times from the U.S. to Hong Kong is about 266 to 290 MS. Can you guys tell me what the best threshhold for this kind of setup would be. I have been suspecting for the longest time that network latency was our issue and I needed some proof ( You guys ROCK!)... Would any of you happen to know where I can get a hold of some technical white papers on the what normal latency requirements are needed for best performance.

Best Regards

[park...@rogers.com]

[Ken Hygh]

I don't know of any such papers off the top of my head, you might google
around.

And frankly, I don't know that you need any such. Your application is
failing because of latency in at least two ways from the logs in your
original post.

Some are failing between the user and WAS. Some of these will definitely
be what Ben described: impatient users are pressing 'stop'. This could
be because they're tired of waiting for the latency in the overall
application.

But the second trace appears to be between WAS and database servers.
Hopefully you're not sending these packets halfway around the world too!

Ken

[Ben_]

>> Our HTTP servers reside in Hong Kong and were connecting from the united
>> states. Dont ask why? I'm still scratching my head as to why they would
>> set this up in this way. Darn project managers!!!!

Honestly, I don't think a "project manager" will decide on this, unless he
has to and is under-informed by engineers in charge.

And anyway, it can make sense... It all depends on the environment and
associated constraints. And if it was a regular proxy server instead of an
HTTP server with a (proxying) plug-in, would you still find it a strange
topology ?

>> Anyways, our average ping times from the U.S. to Hong Kong is about 266
>> to 290 MS.

Such ping times are common over the Internet and I don't think it can alone
be the cause of all the problem.

OK, users are impatient, but it's more in terms of seconds (and even 10
seconds and more). Can you see the difference between 250 ms or 500 ms ping
time ? I can't... Maybe you can say "well, it's not too fast", but no more.

>> Can you guys tell me what the best threshhold for this kind of setup
>> would be. I have been suspecting for the longest time that network
>> latency was our issue and I needed some proof ( You guys ROCK!)... Would
>> any of you happen to know where I can get a hold of some technical white
>> papers on the what normal latency requirements are needed for best
>> performance. Best Regards
>
> I don't know of any such papers off the top of my head, you might google
> around.
>
> And frankly, I don't know that you need any such. Your application is
> failing because of latency in at least two ways from the logs in your
> original post.
>
> Some are failing between the user and WAS. Some of these will definitely
> be what Ben described: impatient users are pressing 'stop'. This could be
> because they're tired of waiting for the latency in the overall
> application.

Yes, the best advise is to first watch in the application for *seconds* of
delays before looking at *ms* improvements on the network.

IHS access logs can tell you the time it receives the requests and the time
it delivered a response. This can be a start.