Delay observed in messages delivered to client

[Satyen]

Hi Simone,

I have a CometD application which is using the following options:

• CometD 3.0.4
• JSR WebSockets
• Ack Extension
• Tomcat 8.0.14

The server pushes a lot of messages to the clients in this application. These messages are delivered properly when the application is deployed in a local network. I am seeing issues, however, when deploying the server in the cloud (Google Compute Engine), and running the clients remotely (over the Internet).

The specific issue is that the server queues a large number of messages (in the batch) between successive responses to meta-connect messages. The server appears to attempt sending all messages in a batch at once. On occasion, all messages in the batch take a long time (5 - 10 seconds) to be delivered to the client.

The messages being pushed are very time-sensitive, and I am looking for ways to get the messages to the client quickly. I have tried changing the 'maxQueue' setting in web.xml to 2, but that hasn't helped.

I will appreciate any guidance on how to coerce the server to send out the messages bound for the clients quicker.

Thanks,

Satyen

[Simone Bordet]

Hi,

You have the classical case where the producer is faster than the
consumer (for whatever reason - in your case the network).
Typical solution is to slow down the producer, or to coalesce messages
(if you can), or improve the transport.

The first two solution are application solutions, for the third you
need to switch to use the WebSocket transport (and not HTTP).

As for maxQueue=2, you have to implement a MaxQueueListener that will
be notified when the queue is full.
Along with a DeQueueListener you will know when the queue is maxed out
and when it's drained, so for example you can turn on/off a flag that
tells the producer to pause/resume.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.

[Satyen]

Thanks Simone.

I should have mentioned that the messages are bursty in nature. The producer generates a large (400/sec) number of messages at times, but averages around 20/sec. It's during the bursts that the queue builds up quickly.

I am already using the WebSockets transport. Not sure what gave you the impression that I am using Long-polling.

I understand your suggestion to have the application coalesce messages. Can ws.messagesPerFrame help toward this goal? My messages are under 1K on average - is there a setting you recommend?

Thanks,

Satyen

[Simone Bordet]

Hi,

On Mon, Jul 6, 2015 at 2:32 PM, Satyen <sat...@aagyaa.com> wrote:
> Thanks Simone.
>
> I should have mentioned that the messages are bursty in nature. The producer
> generates a large (400/sec) number of messages at times, but averages around
> 20/sec. It's during the bursts that the queue builds up quickly.
>
> I am already using the WebSockets transport. Not sure what gave you the
> impression that I am using Long-polling.

My bad.

> I understand your suggestion to have the application coalesce messages. Can
> ws.messagesPerFrame help toward this goal? My messages are under 1K on
> average - is there a setting you recommend?

You have to try yourself.
The default setting for messagesPerFrame is 1, and with the ack
extension will make the communication very chatty.
Try to increase that value, and you should get better batching of messages.
Be aware that if the WebSocket message is too large, it may be
rejected by the client.

When you say 20 msgs/s you mean per server or per each client ?
If it's per client, then it's one message every 50 ms, so you must be
able to have a roundtrip to the client within that time.
If it's longer, you are going to queue up.

Note that using the TimeSync extension, you can actually get the
roundtrip time at the application level, and you may apply different
policies based on whether one session has a fast connection or a slow
one.
For example, you can coalesce for the slow ones, but not for the fast ones.

Use:

long roundtrip =
((Number)session.getAttribute(TimesyncExtension.LAG_ATTRIBUTE)).longValue();

to get the roundtrip time for that session.

[Satyen]

When you say 20 msgs/s you mean per server or per each client ?

If it's per client, then it's one message every 50 ms, so you must be

able to have a roundtrip to the client within that time.

If it's longer, you are going to queue up.

I mean 20 msgs/s per client. Some of these are broadcast messages (all clients receive same message); others are directed at just one client.

Does every message sent by the server involve a roundtrip? What comes back from the client?

In a scenario with 9 messages in a batch (I'm using the Ack Extension), I see the nine messages dequeued together, and they are followed by a meta-connect response. This elicits a single meta-connect request from the client. Is this observation correct?

[Simone Bordet]

Hi,

On Wed, Jul 8, 2015 at 7:09 PM, Satyen <sat...@aagyaa.com> wrote:
> In a scenario with 9 messages in a batch (I'm using the Ack Extension), I
> see the nine messages dequeued together, and they are followed by a
> meta-connect response. This elicits a single meta-connect request from the
> client. Is this observation correct?

You queue on the server at 20 msgs/s.
On the first message queued, the delivery is woken up.
Meanwhile other messages may be queued.
When the delivery runs, it drains the queue; let's say it find 2 messages in it.
The 2 messages are written to the client, along with a /meta/connect
response (because you're using the ack extension).
Meanwhile, other messages are queued, but they are not sent until the
client has issued a /meta/connect request and this has been seen by
the server.
Let's say that the rountrip to the client is 160 ms, so now you have 8
messages in the queue.
When the /meta/connect request from the client is processed, it sees
there are messages in the queue, so it triggers the delivery
immediately.
The 8 messages are drained, written to the client, along with the
/meta/connect response.

And so on.

What causes queueing of messages is the message rate, the roundtrip
and the connection write window (the client is "slow", even though it
may be attached to a large pipe or have a short roundtrip).

If you have bursts at 400 msgs/s, and the client is slow or far away
(or both), then you're going to queue.

Why do you have to send 20 msgs/s to the client ?
Is the client a human ?
If so, it cannot possibly process 20 messages/s (let alone 400),
unless it's a movie to watch.

[Satyen]

Thanks Simone. This information is very helpful !!

[Simone Bordet]

Hi,

On Thu, Jul 9, 2015 at 1:46 PM, Satyen <sat...@aagyaa.com> wrote:
> Thanks Simone. This information is very helpful !!

No problem.
I would be interested, however, in your use case to send 20 msgs/s to
the client.

There may be some improvement that we can make to CometD.

[Satyen]

The application I am working on is used by Call Center agents and supervisors. Agents receive calls, and manipulate them (answer, hold, transfer, etc.) using a request/event interface.

In addition, the client application receives a lot of realtime information about status of other agents, and the state of calls in queue.

The server side of our application gets information from a "Monitoring Server", which doesn't do much to aggregate information. The number of events related to agents and calls is very large. Passing them along to the client has posed plenty of challenges as we have tried deploying the server in the Cloud, and as we have driven up the number of concurrent agents and high call-rates. We have made several passes at coalescing and throttling, and are continuing to make improvements in this direction.

We have pure communication issues, however. We started noticing that messages were simply not being delivered to the client. I added in the Ack Extension at that point, but that doesn't help with timely delivery, which is an important requirement for this application. Also, the Ack extension has made the conversation very chatty, and may be aggravating our issues at this time.

What kind of improvement(s) are you considering? Let me know if I can put any of your ideas to test.

Thanks,

Satyen

[Simone Bordet]

Hi,

On Thu, Jul 9, 2015 at 5:45 PM, Satyen <sat...@aagyaa.com> wrote:
> The application I am working on is used by Call Center agents and
> supervisors. Agents receive calls, and manipulate them (answer, hold,
> transfer, etc.) using a request/event interface.
>
> In addition, the client application receives a lot of realtime information
> about status of other agents, and the state of calls in queue.
>
> The server side of our application gets information from a "Monitoring
> Server", which doesn't do much to aggregate information. The number of
> events related to agents and calls is very large. Passing them along to the
> client has posed plenty of challenges as we have tried deploying the server
> in the Cloud, and as we have driven up the number of concurrent agents and
> high call-rates. We have made several passes at coalescing and throttling,
> and are continuing to make improvements in this direction.

Yes.

Have you considered using lazy channels ?
That is another way of batching messages for a more efficient delivery
(at the cost of slightly increased latency).

> We have pure communication issues, however. We started noticing that
> messages were simply not being delivered to the client.

That may be due to a lot of reasons, and depending on the network you
are dealing with.
Sometimes there is not much you can do, but depends from case to case.

Remember that Webtide offers commercial support for CometD, we can
help reviewing your application, analyzing the current bottlenecks,
etc.