CPU usage spikes while making multiple connections using single client bootstrap

[Jestan Nirojan]

Hi all!

I have a Netty SCTP application which is developed using my topic
branch https://github.com/jestan/netty. When application starts, it
creates a single client bootstrap and assemble a client pipeline with
2 level OMTPE execution handlers. Following things happen afterwards.

1) Application call bootstrap.connect to make new connections (about 4
connections) and register future listeners to monitor the connection
success/failure. In case of failure, the registered future listener
schedule a timer task to reconnect (again reconnect happens by
submitting a Runnable using an executor of a OMTPE).

If a connection refused, it takes 30% CPU to retry/reconnect, if all
connections are refused at application start up, application takes
most of the CPU power.

2)When connection a connection is lost, a handler will call same non
blocking method which does the connect in 1 step, but this case, no
CPU spike at all (even if 4 connections are lost).

So I profiled the application and found out that, most of the CPU
cycles are spent in following call tree

SctpClientPipelineSink.Boss.run() ->
org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(),

Any thoughts why this is happening?

since LinkedTransferQueue was access through Queue interface, I
replacing the Netty LinkedTransferQueue by
java.util.concurrent.LinkedTransferQueue (I use JDK 7) in
SctpClientPipelineSink. After that, the problem was solved (java and
netty LinkedTransferQueues are quite similar, but java
LinkedTransferQueue uses unsafe mechanics and it will be the best
choice in JDK 7)

I also replaced all Netty LinkedTransferQueue usages in SCTP classes
and found that, other hot-spots shown (related to LinkedTransferQueue)
are gone. Now my applications run smoothly.

thanks and regards.
- Jestan Nirojan

[Kevin Burton]

It looks like the one in Netty was written by Doug Lea so I assume it's a backport of the JDK 1.7 LinkedTransferQueue ?  so I imagine the one Netty uses may have a bug?

On Tue, Dec 6, 2011 at 10:53 AM, Jestan Nirojan <jestan...@gmail.com> wrote:

Hi all!

[Norman Maurer]

Very interesting. Let me check this today..

Stay tuned,
Norman

2011/12/7 Kevin Burton <burto...@gmail.com>

[Norman Maurer]

Ok I think I have some solution for this. I will commit it shortly.

Bye,
Norman

2011/12/7 Norman Maurer <norman...@googlemail.com>

[Norman Maurer]

Forgot to link the issue:
https://github.com/netty/netty/issues/102

[Nirojan]

Thanks Norman, 

after your fix, I will be able to merge my pull request without much changes.

regards.

-Jestan

["이희승 (Trustin Lee)"]

Now that you signed the new CLA, what don't you just merge it to the upstream? :-)

--
http://gleamynode.net/
http://twitter.com/trustin
http://twitter.com/trustin_ko
http://twitter.com/netty_project

[Jestan Nirojan]

Trustin, It is done :-)

Please review the merge.

thanks
-Jestan

On Dec 7, 3:58 pm, "이희승 (Trustin Lee)" <trus...@gmail.com> wrote:
> Now that you signed the new CLA, what don't you just merge it to the
> upstream? :-)
>
>
>
>
>
>
>
>
>
> Nirojan wrote:
> > Thanks Norman,
> > after your fix, I will be able to merge my pull request without much
> > changes.
>
> > regards.
> > -Jestan
>
> > On Wed, Dec 7, 2011 at 3:25 PM, Norman Maurer

> > <norman.mau...@googlemail.com <mailto:norman.mau...@googlemail.com>>

> > wrote:
>
> >     Forgot to link the issue:
> >    https://github.com/netty/netty/issues/102
>
> >     Bye,
> >     Norman
>

> >     2011/12/7 Norman Maurer <norman.mau...@googlemail.com
> >     <mailto:norman.mau...@googlemail.com>>

>
> >         Ok I think I have some solution for this. I will commit it
> >         shortly.
>
> >         Bye,
> >         Norman
>

> >         2011/12/7 Norman Maurer <norman.mau...@googlemail.com
> >         <mailto:norman.mau...@googlemail.com>>

>
> >             Very interesting. Let me check this today..
>
> >             Stay tuned,
> >             Norman
>

> >             2011/12/7 Kevin Burton <burtona...@gmail.com
> >             <mailto:burtona...@gmail.com>>

>
> >                 It looks like the one in Netty was written by Doug Lea
> >                 so I assume it's a backport of the JDK 1.7
> >                 LinkedTransferQueue ?  so I imagine the one Netty uses
> >                 may have a bug?
>
> >                 On Tue, Dec 6, 2011 at 10:53 AM, Jestan Nirojan

> >                 <jestanniro...@gmail.com