Returning gzipped content, generic.json.gz ?

[Vincent]

Hi,

I want to return gzipped json content (although the json part is not really important). 

I did some googling and found this post which is somewhat related, 

https://groups.google.com/d/msg/web2py/cgSrsC73vzg/ZNYN3u7ChVwJ

as far as I could tell the minify part of the question was answered but not the gzipped part.

I know some servers can compress, but I want to do the compression inside web2py since my app may run on various server  configurations. My objective is to serve experimental data through a RESTful service.

Basically, I am wondering if something like "generic.json.gz" already exists built-in to web2py. Something that returns the gzipped form of the json of the response.

Does this already exist somewhere or do I need to write it myself?

[Ricardo Pedroso]

You can not "blindly" gzip your data unless you control the clients.

It's the client that tells if it can accept gzip content or not,
and the web server will choose to gzip it or not.

I will advice you to let this to the web server.

But if you really want to gzip yourself, than you should take care of the
request headers, specifically the Accept-Encoding one, ex:

Accept-Encoding: gzip, deflate

than if you gzip you should tell the client about it in the response
headers with:

Content-Encoding: gzip


http://en.wikipedia.org/wiki/HTTP_compression

Ricardo

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Vincent]

Hi Ricardo,

in this case I do control the clients. The scientific data would be on an intranet and would only really make sense to the a specialized client. 

So apart from having appropriate headers I just need to write the json.dumps and gzip myself?

There is no functionality already built-in?

Thanks

Vincent

[Ricardo Pedroso]

On Mon, Apr 22, 2013 at 1:28 AM, Vincent <vincent...@gmail.com> wrote:
> Hi Ricardo,
>
> in this case I do control the clients. The scientific data would be on an
> intranet and would only really make sense to the a specialized client.
>
> So apart from having appropriate headers I just need to write the json.dumps
> and gzip myself?
> There is no functionality already built-in?

I think web2py does not have gzip built-in, you have to do yourself.

Ricardo

[Derek]

Unless you are using IE lower than 5.5, all clients support gzip. Looks like 'Rocket' doesn't though. You could modify the 'write' on line 1758 to gzip files though. Just have to set the content-length and content-encoding yourself. You could add in a check to see if it's supported... 

if 'gzip' in environ.get('HTTP_ACCEPT_ENCODING', ''):

#(if the http accept-encoding specifies gzip...) gzip it.

here's how i do it with wsgi middleware, but it should work here with rocket... of course, it wouldn't work if you were using gevent to serve (as an example). You could just plug in a middleware though. I took this from the pylons tutorial...

buffer = StringIO.StringIO()

        output = gzip.GzipFile(

            mode='wb',

            compresslevel=self.compresslevel,

            fileobj=buffer

        )

I set compresslevel to 5, you could set it lower if you prefer speed to filesize, that's up to you. I did find that I got the most size benefit from compressing large json returns (like tables). However, I just set it to compress everything except images, it seems to work great.

[Michele Comitini]

a. Use a full blown webserver like apache or nginx and configure it to gzip content on client request

b. Use a simple wsgi gzipping filter.

For b. there is a simple and fast enough filter in wsgitools package.  You can use it in anyserver.py or any place when calling the wsgi application as:

from wsgitools.filters import WSGIFilterMiddleware, GzipWSGIFilter

# filter web2py wsgi application

wsgiapp = WSGIFilterMiddleware(gluon.main.wsgibase, GzipWSGIFilter)

2013/4/22 Derek <sp1...@gmail.com>

[Derek]

Option A is the best option, B is second best. I recommended a change in Rocket since it's included in w2py (it would be a nice enhancement), and since this is an internal site and Vincent may not be a webserver guy, that may fit the bill more.

[Vincent]

Thanks everyone for your answers. I now have several options to choose from!