is it possible to compress json data in websocket frames from cometd bayeux server to client ?

[Karthik Balachandran]

Hello,

Is it possible to compress json data in websocket frames from bayeux server to client ?

Read up on permessage-deflate but looks like cometd's bayeux server does not support it and the only client that supports it is chrome.

Is there a recommended approach ?

Thanks much,

kb

[Simone Bordet]

Hi,

On Wed, Oct 8, 2014 at 5:41 AM, Karthik Balachandran
<bkar...@signalfuse.com> wrote:
> Hello,
>
> Is it possible to compress json data in websocket frames from bayeux server
> to client ?

In theory, yes.
In practice, much less so.

> Read up on permessage-deflate but looks like cometd's bayeux server does not
> support it and the only client that supports it is chrome.

CometD does support configuring WebSocket extensions, although it
could be made simpler, at least for the permessage-deflate.
However, like you note, only few browsers support it.

> Is there a recommended approach ?

If you can restrict your browser usage to Chrome (or not care much for
those that don't have it), I think permessage-deflate is the way to
go.
I'd be happy to update CometD to make this case simpler, which has not
yet been done due to lack of support from browsers, and even for those
that supports it, the implementation was often broken.

Alternatively, you can perform compression of data at the application
level, typically gzip+base64, and have the client app perform the
opposite conversion.

Lastly, disabling WebSocket and enabling HTTP will allow you to
leverage gzip filters (such as Jetty's), which are widely supported.
Sometimes, the gains of this configuration are much bigger than the
loss of WebSocket.
You have to measure for your particular case to know.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.

[Antonio]

Hello Simone,

Am I right in saying that permessage-deflate extension is enabled by default in CometD 3.0 +  ?

Regards,

[Simone Bordet]

Hi,

On Wed, Jan 20, 2016 at 6:10 PM, Antonio <po...@visokio.com> wrote:
> Hello Simone,
>
> Am I right in saying that permessage-deflate extension is enabled by default
> in CometD 3.0 + ?

No, it is not, for the reasons explained in this thread.

--
Simone Bordet
----
http://cometd.org
http://webtide.com

[Antonio]

Thanks Simone, I see the comments in Jetty bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=431459 about instability too

If I want to experiment with it programmatically with Jetty in embedded mode, and for this reason I don't have the WebSocketServletFactory, 

what is the way to register the permessage-deflate  extension on the server side ? Or is there a "init parameter" to set ?

The closest thing I found (but wrong) is :

WebSocketServerContainerInitializer.configureContext(servletContextHandler).getClient().getExtensionFactory().register("permessage-deflate", PerMessageDeflateExtension.class);

where I can set the WebSocketClient extensions, not the server ones.

Regards,

Antonio

[Simone Bordet]

Hi,

On Wed, Jan 20, 2016 at 9:01 PM, Antonio <po...@visokio.com> wrote:
> Thanks Simone, I see the comments in Jetty bug
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=431459 about instability too
>
> If I want to experiment with it programmatically with Jetty in embedded
> mode,

I have not tried myself, so you're opening the way here, be ready with
the machete :)

> and for this reason I don't have the WebSocketServletFactory,
> what is the way to register the permessage-deflate extension on the server
> side ? Or is there a "init parameter" to set ?

There is not yet; can you please file an issue about this ?

> The closest thing I found (but wrong) is :
> WebSocketServerContainerInitializer.configureContext(servletContextHandler).getClient().getExtensionFactory().register("permessage-deflate",
> PerMessageDeflateExtension.class);
> where I can set the WebSocketClient extensions, not the server ones.

I had a quick look and there is no simple API; it requires some Jetty
API coding.

The idea would be to access Jetty's WebSocketServerFactory:

WebSocketServerContainerInitializer.configureContext(servletContextHandler);
WebSocketUpgradeFilter wsFilter =
servletContextHandler.getAttribute(WebSocketUpgradeFilter.class.getName());
WebSocketServerFactory wsServerFactory = wsFilter.getFactory()
wsServerFactory.getExtensionFactory().register("permessage-deflate",
PerMessageDeflateExtension.class);

Then the browser would negotiate the permessage-deflate extension, and
everything should be automatic.

There may be some additional tuning to be performed in
WebSocketTransport.Configurator.getNegotiatedExtensions(), but I am
not sure.

[Antonio]

Hello Simone,

FYI, the code snippet works fine, I verified the compressed content exchanged through Wireshark, 

Also checked in Chrome Dev tools the presence of the Sec-WebSocket-Extensions:permessage-deflate; header.

I did not try any additional tuning, just registered the extension, and it seems sufficient.

I filed the issue about the init parameter here https://github.com/cometd/cometd/issues/630

Thanks for your help

Regards,

Antonio

[Antonio]

Hello all,

Unfortunately adding the deflate compression causes a ParseException on the server side when the CometD client sends a JSON Object as message.

It seems to be caused by the size of the message > 1KB .

I am using latest Chrome browser, that sends client_max_window_bits header, and server opts out ignoring this request.

I am still looking into it, but if anyone has got a clue that would be great :)

Here is a good link about it https://www.igvita.com/2013/11/27/configuring-and-optimizing-websocket-compression/ 

This is the exception stack trace, with different causes every time:

 WARN [Jetty [Omniscope]-153] (AbstractWebSocketTransport.java158) - Error parsing JSON: 

at org.cometd.common.JettyJSONContext.parse(JettyJSONContext.java:75)

at org.cometd.server.AbstractServerTransport.parseMessages(AbstractServerTransport.java:150)

at org.cometd.websocket.server.AbstractWebSocketTransport.access$200(AbstractWebSocketTransport.java:45)

at org.cometd.websocket.server.AbstractWebSocketTransport$AbstractWebSocketScheduler.onMessage(AbstractWebSocketTransport.java:271)

at org.cometd.websocket.server.JettyWebSocketTransport$WebSocketScheduler.onWebSocketText(JettyWebSocketTransport.java:203)

at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:189)

at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)

at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:64)

at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:158)

at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:160)

at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:331)

at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:170)

at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:105)

at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:131)

at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:85)

at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)

at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)

at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:256)

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:647)

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:485)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:238)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:57)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:191)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:126)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)

at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.IllegalStateException: unknown char ','(44) in |||,

at org.eclipse.jetty.util.ajax.JSON.parse(JSON.java:879)

at org.cometd.common.JettyJSONContext.parse(JettyJSONContext.java:70)

or 

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.cometd.bayeux.Message$Mutable

at org.cometd.common.JettyJSONContext.adapt(JettyJSONContext.java:87)

at org.cometd.common.JettyJSONContext.parse(JettyJSONContext.java:71)

... 27 more

Regards,

Antonio

[Simone Bordet]

Hi,

On Fri, Jan 22, 2016 at 1:14 PM, Antonio <po...@visokio.com> wrote:
> Hello all,
>
> Unfortunately adding the deflate compression causes a ParseException on the
> server side when the CometD client sends a JSON Object as message.
> It seems to be caused by the size of the message > 1KB .
> I am using latest Chrome browser, that sends client_max_window_bits header,
> and server opts out ignoring this request.

There is no API in Java to handle the max window bits, so there is not
much that one can do about this.

If you have a reproducible test case I may have a look, but I would
not use it for now.

It may well be that enabling compression is actually counter
productive, and as I said, it is looking for troubles.

You don't *need* it, and you can enable it later, if and when it works.

[Antonio]

Hello Simone,

Problem solved by having a look at https://bugs.eclipse.org/bugs/show_bug.cgi?id=471055 and then updated Jetty to version 9.2.14.

Basically I was using a version just before that fix.

Thank you again

Antonio