[PATCH] allow gzip compression while posting json

[pla...@acipia.com]

Hi list,

This is a very small patch against 2.0RC to allow HTTP client to use gzip compression while posting to api/put endpoint. As it is quite simple (thanks to netty) i just post it here rather that forking on github.

A 32 kbytes clear JSON file is easy to compress with gzip algorithm (32kb -> 1666 bytes), allowing putting more values in a single POST (20 times more). It is really helpful if you prefer some cpu cycles rather than many http exchanges, for example on high latency networks (umts, satelite etc).

A curl example :

$ gzip -9c clear-32k.json > gzip-32k.json

$ file gzip-32k.json
gzip-32k.json: gzip compressed data, was "clear-32k.json", from Unix, last modified: Thu Jan 16 15:31:55 2014

$ ls -l gzip-32k.json
-rw-r--r-- 1 root root 1666 févr.  4 09:57 gzip-32k.json

$ curl -X POST --data-binary "@gzip-32k.json" --header "Content-Type: application/json" --header "Content-Encoding: gzip" http://mytsdb1:4242/api/put?details
{"errors":[],"failed":0,"success":280}

Note that while aiming to allow GZIPed POST, all netty http pipeline is affected, so using gzip with other API endpoints is possible. It only allows compression in requets and doesn't modify TSDB response behaviour. The patch doesn't break clients without gzip and should be safe to use. We're using it for a couple of weeks without problem.

P

[ManOLamancha]

On Tuesday, February 4, 2014 4:12:59 AM UTC-5, pla...@acipia.com wrote:

This is a very small patch against 2.0RC to allow HTTP client to use gzip compression while posting to api/put endpoint. As it is quite simple (thanks to netty) i just post it here rather that forking on github.

Sweet, thanks! Would you mind signing http://opentsdb.net/contributing.html? 

[Pierre Laden]

Done !

[Kyle Brandt]

It doesn't look like this has made into next? gzip support would be nice...

[ManOLamancha]

On Tuesday, April 15, 2014 9:39:49 AM UTC-4, Kyle Brandt wrote:

It doesn't look like this has made into next? gzip support would be nice...

Right, didn't make it into next unfortunately. I'll put it into 2.1 though. 

[Kyle Brandt]

Any chance of getting this in "next" now?

[ManOLamancha]

On Tuesday, May 27, 2014 4:46:25 PM UTC-4, Kyle Brandt wrote:

Any chance of getting this in "next" now?

Absolutely, I'll be working on the backlog this next week. 

[Kyle Brandt]

Hi Chris,

Just wanted to ping you on this as a reminder, hoping to see it in 2.1.

Best,

Kyle

[ManOLamancha]

On Friday, July 11, 2014 9:26:03 AM UTC-4, Kyle Brandt wrote:

Hi Chris,

Just wanted to ping you on this as a reminder, hoping to see it in 2.1.

Definitely want to get it in, keep reminding me :) Super swamped these next few weeks 

[Kyle Brandt]

Reminder as requested :-) 

Also, someone put this in a PR: 

https://github.com/OpenTSDB/opentsdb/pull/375

[ManOLamancha]

On Tuesday, July 29, 2014 8:01:23 AM UTC-7, Kyle Brandt wrote:

Reminder as requested :-) 

Also, someone put this in a PR: 

https://github.com/OpenTSDB/opentsdb/pull/375

Finally working again so I can take a look at this, thanks for pinging me :) 

[ManOLamancha]

On Monday, August 18, 2014 6:28:19 PM UTC-7, ManOLamancha wrote:

On Tuesday, July 29, 2014 8:01:23 AM UTC-7, Kyle Brandt wrote:

Reminder as requested :-)

Finally posted it to 2.1 (next) today so you're good to go. Thanks!