Still getting the dreaded "java.nio.channels.ClosedChannelException" from Netty

[Luciano Fiandesio]

Hello,

some weeks ago I have reported a problem with Akka remote module
failing on heavy load (http://bit.ly/dJDOx1).
The problem was resolved by recompiling the Akka source and increasing
a Netty parameter.
Later, Viktor added a setting in the Akka server configuration section
to tune this value.
So, now I have deployed my application on a Oracle Weblogic 10.3.3
instance and I'm experiencing the same issue, this time with a minimal
load.
Here is a link to the stack trace: http://pastebin.com/ywidBtg8

The setup is pretty much identical to the one in the first post:
Akka 1.0
a client invoke a remote typed actor (Java) to execute some business
logic.
At the moment the client war and the server war are deployed on the
same VM.

I'm running the server with the backlog parameter set to 4096. I have
to say that reducing or increasing the parameter doesn't really make
any difference, the Netty socket dies after few connections (not even
concurrent ones). Weblogic is famous for having all sort of problems
with Classloaders and multi-hreaded applications but that is our
target platform.

Anything I can try?

Thanks
Luciano

[√iktor Klang]

If I recall correctly (on train to airport), it is a case where a consumer can't keep up with the producer (hence 'backlog').
You don't mention if there's anything else that has changed, OS, OS version, jvm version or anything else so I don't have access to all variables.

One guess is that if you changed OS or machine, the TCP stack has a too small buffer, I also think you can increase the Socket buffer in Java via a JVM parameter, but I can't recall what it's called.

Another guess is that the IO-workers netty uses gets downplayed scheduling-wise.

Might also, as you suggested, be that the container is less capable than desired.

Thoughts?

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Luciano Fiandesio]

Hi Viktor

Thanks for getting back to me.
The platform differences are mainly OS (win7 and now Win 2008 server)
and memory (less Ram installed on the server then on my laptop...).
Same 64 bit JVM (sun) and different container (Jetty vs. Web logic).

I have increased the backlog value to a whopping 8192 and things look
better. The server can cope with a bit more concurrent load (will do
some proper testing tomorrow).

Do you guys see any potential issue in running Netty with such a high
backlog?

Thanks
L

On Mar 16, 5:47 pm, √iktor Klang <viktor.kl...@gmail.com> wrote:
> If I recall correctly (on train to airport), it is a case where a consumer
> can't keep up with the producer (hence 'backlog').
> You don't mention if there's anything else that has changed, OS, OS version,
> jvm version or anything else so I don't have access to all variables.
>
> One guess is that if you changed OS or machine, the TCP stack has a too
> small buffer, I also think you can increase the Socket buffer in Java via a
> JVM parameter, but I can't recall what it's called.
>
> Another guess is that the IO-workers netty uses gets downplayed
> scheduling-wise.
>
> Might also, as you suggested, be that the container is less capable than
> desired.
>
> Thoughts?
>

> On Mar 16, 2011 5:26 PM, "Luciano Fiandesio" <i...@lucianofiandesio.com>

On Mar 16, 5:47 pm, √iktor Klang <viktor.kl...@gmail.com> wrote:
> If I recall correctly (on train to airport), it is a case where a consumer
> can't keep up with the producer (hence 'backlog').
> You don't mention if there's anything else that has changed, OS, OS version,
> jvm version or anything else so I don't have access to all variables.
>
> One guess is that if you changed OS or machine, the TCP stack has a too
> small buffer, I also think you can increase the Socket buffer in Java via a
> JVM parameter, but I can't recall what it's called.
>
> Another guess is that the IO-workers netty uses gets downplayed
> scheduling-wise.
>
> Might also, as you suggested, be that the container is less capable than
> desired.
>
> Thoughts?
>

> On Mar 16, 2011 5:26 PM, "Luciano Fiandesio" <i...@lucianofiandesio.com>

[Jonas Bonér]

Except higher latency and more memory consumption I can't think of any potential issues.

Please report back your results. How many txs per sec are you sustaining?

--
Jonas Bonér

http://scalablesolutions.se
http://jonasboner.com
http://akka.io
http://letitcrash.com

[Luciano Fiandesio]

Hi Jonas

I will run some proper test during the next couple of day.
Unfortunately I'm not able yet to run full end to end test son our
platform and I have to resort to system stubs returning "static" data
(mostly Webservices).
Nevertheless, I was worried to see Netty failing with little load -
not even parallel load - after few transactions and a backlog value
set to 4096.

Also, do you have any cases of Akka deployed on Weblogic?

Thanks
Luciano

On Mar 16, 6:34 pm, Jonas Bonér <jo...@jonasboner.com> wrote:
> Except higher latency and more memory consumption I can't think of any
> potential issues.
>
> Please report back your results. How many txs per sec are you sustaining?
>
> --
> Jonas Bonér
>

> http://scalablesolutions.sehttp://jonasboner.comhttp://akka.iohttp://letitcrash.com