Where to do GZIP encoding/decoding

[N8]

My team is working on a proxy server that will have to read GZIP encoded responses and respond to some requests with GZIP encoded responses. Since there is nothing built in, we will roll our own. The idea is to implement a decorator for ReadStream and WriteStream that install a handler that does the de/encoding. The question came up whether doing the encoding/decoding on the event thread is appropriate since it will busy the thread. I maintain that the event thread is the place to the work being done for compression and is akin to the work being done for SSL encryption. Another alternative is to pass the content to a BusMod do the gzip there, then send the result back to the main verticle - this seems overly complicated and inefficient.

I imagine someone has done this before in Vertx (I've seen some posts on it) and I'd like to hear opinions on the topic and any lessons learned.  

BTW: This seems like a common enough thing to do, it behooves Vertx to have an official way to do this, be it some utility classes that wrap a Read/Write Stream, or at least an official recommendation on how to do it.

Thanks in advance for your thoughts. 

 

  

[Norman Maurer]

I think adding support for it would be quite easy as Netty supports GZIP / ZLIB out of the box. So the only thing that would be needed is to add the encoder/decoder to the ChannelPipeline of the Channel. 

The question would be more how you would enable it. Maybe just have a module that add it to the pipeline and so enable it ? So this would basically mean once the module is installed GZIP would be active.

@tim: wdyt ?

Am 02.04.2013 um 19:25 schrieb bytor99999 <bytor...@gmail.com>:

Just adding some two cents here.

"this seems overly complicated and inefficient."

It seems, but it is kind of  way of what you do in an async message design. You are choosing vert.x because of this type of design, where you pass things back and forth through the eventBus all the time. It is written to handle loads and loads of messages fast. So what would be considered too chatty in say a Java EE, Servlet, world. It is different here. You have to look at it differently.

Hope that helps, while still not actually answering your question. 

Personally, I would send any incoming message out of the single main event thread onto the eventBus to be picked up by a Worker verticle, unzipped, then sent back on the EventBus. It doesn't have to be sent back to the Main verticle, it can be sent to some other address with some other Handler.

Mark

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Pïd stèr]

On 2 Apr 2013, at 18:45, Norman Maurer <norman...@googlemail.com> wrote:

I think adding support for it would be quite easy as Netty supports GZIP / ZLIB out of the box. So the only thing that would be needed is to add the encoder/decoder to the ChannelPipeline of the Channel. 

The question would be more how you would enable it. Maybe just have a module that add it to the pipeline and so enable it ? So this would basically mean once the module is installed GZIP would be active.

I was thinking about this last night in the context of the file system, I concluded that a xip read/write stream impl would be useful - as then the Pump could be used.

If there's a way to turn it on selectively in Netty that would be good.

p

[Norman Maurer]

Am 02.04.2013 um 20:01 schrieb Pïd stèr <p...@pidster.com>:

On 2 Apr 2013, at 18:45, Norman Maurer <norman...@googlemail.com> wrote:

I think adding support for it would be quite easy as Netty supports GZIP / ZLIB out of the box. So the only thing that would be needed is to add the encoder/decoder to the ChannelPipeline of the Channel. 

The question would be more how you would enable it. Maybe just have a module that add it to the pipeline and so enable it ? So this would basically mean once the module is installed GZIP would be active.

I was thinking about this last night in the context of the file system, I concluded that a xip read/write stream impl would be useful - as then the Pump could be used.

If there's a way to turn it on selectively in Netty that would be good.

What you mean with selectively ? In Netty you either have it enable or disabled per Channel (which represent a socket)

[N8]

If anyone is interested, I made a pull request (https://github.com/vert-x/vert.x/pull/555) to address this. By far, the simplest and most efficient solution is to take advantage of Nettty's ability to do this. We just needed to expose access to enable it via Vertx. Thanks to Pid for the suggestion.  

[Daryl Teo]

This might make more sense as a module. Like https://github.com/vert-x/mod-formupload

Daryl

[N8]

Someone would need to explain the benefit of doing that to me. From the looks of it, it would take a very simple and straight forward implementation and just complicate it. Perhaps there are some benefits I'm not aware of. 

As a precedent, enabling SSL is not written as a module, but part of the core Client and Server classes. Setting up SSL simply configure's Netty's SSL capabilities. The GZIP implementation is exactly the same.  

[jeremy....@hulu.com]

Is there currently an easy way to decompress POST requests with gzip'ed body contents? The approaches I've tried so far don't seem to be working.

On Tuesday, April 2, 2013 9:08:18 AM UTC-7, N8 wrote:

[Nathan Pahucki]

See https://github.com/vert-x/vert.x/pull/555/ which was closed without merging it in because it was left to languish so long that it's apparently hopelessly out of sync with master. If you're working on 1.3.x you should be able to patch it easy enough though. 

- N8 / 雷森 

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/USuv357tW6Q/unsubscribe.
To unsubscribe from this group and all of its topics, send an email to vertx+un...@googlegroups.com.

[jeremy....@hulu.com]

Will this change support the server decompressing requests from the client? The change looked like it was mainly allowing servers to compress data to clients and allowing vertx clients to compress requests. I may be missing something obvious here though. 

[Nathan Pahucki]

Yes, it turns on the built in support in Netty for compressing and decompressing requests.