The decoded text message was too big for the output buffer and the endpoint does not support partial messages

[Boris Petrov]

I'm using CometD 3.1.8 on the server and the client. The server is running on Tomcat 9.0.19. The only option I've set on the server is `bayeux.setOption("ws.messagesPerFrame", 5);` and I've added the ACK extension: `bayeux.addExtension(new AcknowledgedMessagesExtension());`

When a message is sent from the server that is around 300KB (a bit bigger than that), I get an error in the client - `The decoded text message was too big for the output buffer and the endpoint does not support partial messages`. The client is running on Chrome 74.0.3729.131.

Reading through this thread - https://groups.google.com/forum/#!topic/cometd-users/CbR1IY9Wx4U - makes me think that I should do nothing on my side and that CometD should split messages and send them accordingly and I shouldn't worry however big messages I send. This is obviously not the case. What am I doing wrong?

Thanks!

[Simone Bordet]

Hi,

I think you read the thread you linked above the other way around.

So in your case Chrome has a limit on the size of WebSocket messages
it can receive.
You have chosen to put 5 Bayeux messages into 1 WebSocket message.
If your Bayeux messages sum up to 300+ KiB, then Chrome chokes.

Solution1: ws.maxMessagesPerFrame=1 (the default)
This will work unless a single Bayeux message is larger than 300+ KiB,
otherwise you will have the same problem.

Solution2: you have to split your Bayeux messages in smaller chunks.
CometD cannot do that for you because it won't know where to split the messages.
You have to do this at the application level.
For example, you are sending an array of items, you want to send N
messages with the array split in N parts.

Solution3: you disable the WebSocket transport and use only HTTP.
HTTP transports have no limit on the message size.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.

[Boris Petrov]

Hi,

Thanks for the reply.

On Wednesday, May 8, 2019 at 6:32:27 PM UTC+3, Simone Bordet wrote:

Solution2: you have to split your Bayeux messages in smaller chunks.
CometD cannot do that for you because it won't know where to split the messages.
You have to do this at the application level.
For example, you are sending an array of items, you want to send N
messages with the array split in N parts.

I haven't thought this through very much but shouldn't this be handled by the library itself (CometD)? Isn't it possible to have a configuration option that specifies at what size to split messages and then to assemble them again on the client side? Otherwise every application will have to implement their own solutions which will pretty much be all the same. What am I missing?

[Simone Bordet]

Hi,

On Wed, May 8, 2019 at 10:04 PM Boris Petrov <bo...@profuzdigital.com> wrote:
> I haven't thought this through very much but shouldn't this be handled by the library itself (CometD)? Isn't it possible to have a configuration option that specifies at what size to split messages and then to assemble them again on the client side? Otherwise every application will have to implement their own solutions which will pretty much be all the same. What am I missing?
>

CometD cannot split a user-generated JSON, because it does not know
the JSON structure.

Imagine a simple structure of you application data that you want to
send on a Bayeux message, like:

{ "image": "<long base64 of the image>" }

CometD cannot obviously split it on the byte because it will be:

{ "image": "<first half base64>

and

<second half base64>" }

and neither is a valid JSON structure.

In general, CometD cannot split user-provided JSON structures, only
your application can do it.
For example, your application can split the previous example in this way:

{
"chunk": 1,
"chunks": 2,
"image": "<first chunk of base64>"
}

{
"chunk": 2,
"chunks": 2,
"image": "<second chunk of base64>"
}

Then your client can put the 2 chunks of base64 together and get the image.

As you can see the application has to put some additional metadata
(how many total chunks and what chunk is the current one), and format
the message into proper JSON.
Then it can make Bayeux messages out of those splitted JSONs, batch
them together so that the client receives them in sequence.

[Boris Petrov]

Well, in the end the user JSON is serialized into a string and sent that way, right? So theoretically CometD could split that string in parts and do the magic with "chunk" and "chunks" and then combine all the parts and expose the result to the user. It will be a bit slower (as JSON serialization will be done twice - once for the user data and then for the part with "chunk" and "chunks") but at least it will be transparent for the user.

I'm not saying it should be done this way, just shouting out ideas. Otherwise this problem will probably bite a lot of people who, like me, don't realize there is such a limit.

[Simone Bordet]

Hi,

On Wed, May 8, 2019 at 11:36 PM Boris Petrov <bo...@profuzdigital.com> wrote:
>
> Well, in the end the user JSON is serialized into a string and sent that way, right? So theoretically CometD could split that string in parts and do the magic with "chunk" and "chunks" and then combine all the parts and expose the result to the user. It will be a bit slower (as JSON serialization will be done twice - once for the user data and then for the part with "chunk" and "chunks") but at least it will be transparent for the user.
>
> I'm not saying it should be done this way, just shouting out ideas. Otherwise this problem will probably bite a lot of people who, like me, don't realize there is such a limit.

The Bayeux protocol won't support that out of the box so other
non-CometD clients using the Bayeux protocol won't support it.
It can be done with an extension, but max portability will be when the
application is doing it.

File an issue about this; if there is enough request it will
implemented - you can give it a try if you want.