Cometd bayeux load testing

[Yankee]

Hi,

I am running some comet load testing with cometd jars from jetty
version 6.1.14.
I have java clients harness which subscribes to the server and
receives notifications.
Then there is a server side harness which receives subscriptions and
pumps in notifications to all the clients.

Earlier, i was facing a memory leak issue, so i replaced the cometd-
api jar with patch 9 which was shipped with jetty 6.1.18. Now i dont
see any memory leak issues, but after an hour of load testing, clients
start dieing.

I see the following exception on my client side harness :

2009-10-06 23:16:15.425::WARN: EXCEPTION
java.io.IOException: Connect failed:{successful=false,
error=402::Unknown client, channel=/meta/connect, advice=
{interval=500, reconnect=handshake}}
at org.mortbay.cometd.client.BayeuxClient
$Connect.onResponseComplete(BayeuxClient.java:662)
at org.mortbay.jetty.client.HttpExchange
$Listener.onResponseComplete(HttpExchange.java:584)
at org.mortbay.jetty.client.HttpExchange.setStatus
(HttpExchange.java:179)
at org.mortbay.jetty.client.HttpConnection
$Handler.messageComplete(HttpConnection.java:509)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:746)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:
213)
at org.mortbay.jetty.client.HttpConnection.handle
(HttpConnection.java:263)
at org.mortbay.io.nio.SelectChannelEndPoint.run
(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run
(QueuedThreadPool.java:522)

The test case had 5000 subscriptions and 2000 notifications per second
for 2 hours.
After 1 hour, i started loosing clients. At the end of 2 hours, i lost
almost 2000 subscriptions.

Does anyone know about this exception?
Has someone faced this issue earlier?

Thanks,
Y

[Simone Bordet]

Hi,

That is all you get on the client ? Do you see a "Connection reset by
peer" exceptions ? The server side shows no exception ?

As for load testing we went through it for the latest cometd release
which is based on Jetty7, fixing many Jetty bugs and Cometd bugs
related to high load testing.
We consider Cometd based on Jetty7 more stable and reliable, and the
base for future development.
The Cometd implementation included in Jetty6 will be deprecated in
favor of the official one hosted at http://cometd.org.

Simon
--
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless. Victoria Livschitz

[Yankee]

Hi Simon,

Thanks for your reply.
That is all i get on the client. I dont see any other exception.
The server side shows exception of not being able to send notification
to that client.

Will try to use jars from jetty 7 and see what client has to say.
But is there anything else due to which this is failing or is there
something i could do on client side to rectify this issue?

Thanks,
Y

> --http://bordet.blogspot.com

[Yankee]

Jetty 7 doesnt have cometd in it. I tried using cometd-api beta 10 jar
from the cometd site and executed the same test case.
It yet showed the same error.

- Y

[Dustin Whitney]

Yankee, please write back with your results.  I am interested in reading them because I am building an app using Jetty 6.1.19.

Thanks,
Dustin

[Simone Bordet]

Hi,

On Thu, Oct 8, 2009 at 22:44, Yankee <anki...@gmail.com> wrote:
> Hi Simon,
>
> Thanks for your reply.
> That is all i get on the client. I dont see any other exception.
> The server side shows exception of not being able to send notification
> to that client.

The exception you reported happens when the server detects that a
request is coming from a client it does not know.
The server expires clients if it does not see any activity from them
within a certain period of time. For example, the server expects a
connect message from client 1; if it does not come within 10 seconds,
it expires the client.
This is quite common in load testing, as the server and the client are
so busy that the server may not be able to process a connect from a
client and think that the client expired.

This mechanism of client expiration has been improved in Jetty7 and in
the latest Cometd code with better configurability (the relevant
parameter is "maxInterval") and with a couple of bug fixes that were
causing a race between the client and the server.
Unfortunately it has not been backported to Jetty6.

I am deploying now a snapshot of cometd-1.0.rc1-snapshot which is
quite stable and likely to be very similar (if not identical apart
some demo improvements) to 1.0 final, and that's based on Jetty7.
I would suggest to base on that, if possible.

Simon
--