compress files?

[spicyj]

Anyone know how to write a Rack middleware to serve static files as
gzipped?

I can see how to set the Content-Encoding header, but the file still
needs to be zipped up.

[Adam Wiggins]

On Wed, Jun 11, 2008 at 8:30 AM, spicyj <spicyj...@gmail.com> wrote:
> Anyone know how to write a Rack middleware to serve static files as
> gzipped?

How about:

class GzipOutputFilter
def initialize(app)
@app = app
end

def call(env)
out = @app.call(env)

gzip = IO.popen('gzip', 'r+')
gzip.write out[2] # handles String only, would be nice to check
for IO object too
gzip.close_write

out[1]['Content-encoding'] = 'gzip'
out[2] = gzip

out
end
end

class HelloHandler
def call(env)
[ 200, { 'Content-type' => 'text/plain' }, 'hello, world' ]
end
end

use Rack::Lint
use GzipOutputFilter
run HelloHandler.new

Adam

[Ezra Zygmuntowicz]

Here is a middleware adapted from Mongrel's Deflate Handler:

require 'zlib'

class Deflater
include Zlib

def initialize(app)
@app = app
end

def call(env)
accept = env['HTTP_ACCEPT_ENCODING']
status, headers, body = @app.call(env)
if accept and accept.include? 'deflate'
headers["Content-Encoding"] = "deflate"
body = deflate(body)
end
[status, headers, body]
end

def deflate(body)
deflater = Deflate.new(
DEFAULT_COMPRESSION,
# drop the zlib header which causes both Safari and IE to choke
-MAX_WBITS,
DEF_MEM_LEVEL,
DEFAULT_STRATEGY)

case body
when IO
body.rewind if body.respond_to? :rewind
gzout = StringIO.new(deflater.deflate(body.read, FINISH))
body.close
when String
gzout = StringIO.new(deflater.deflate(body, FINISH))
end
gzout.rewind
gzout
end

end

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ez...@engineyard.com
-- EngineYard.com

[Christian Neukirchen]

Ezra Zygmuntowicz <e...@engineyard.com> writes:

> On Jun 11, 2008, at 8:30 AM, spicyj wrote:
>
>>
>> Anyone know how to write a Rack middleware to serve static files as
>> gzipped?
>>
>> I can see how to set the Content-Encoding header, but the file still
>> needs to be zipped up.
>
> Here is a middleware adapted from Mongrel's Deflate Handler:

I'd like to import this to the Rack tree.

--
Christian Neukirchen <chneuk...@gmail.com> http://chneukirchen.org

[Ezra Zygmuntowicz]

On Jun 12, 2008, at 7:33 AM, Christian Neukirchen wrote:

>
> Ezra Zygmuntowicz <e...@engineyard.com> writes:
>
>> On Jun 11, 2008, at 8:30 AM, spicyj wrote:
>>
>>>
>>> Anyone know how to write a Rack middleware to serve static files as
>>> gzipped?
>>>
>>> I can see how to set the Content-Encoding header, but the file still
>>> needs to be zipped up.
>>
>> Here is a middleware adapted from Mongrel's Deflate Handler:
>
> I'd like to import this to the Rack tree.

Please feel free, maybe just add a note that it was adapted from the
mongrel deflate handler.

[Stoyan Zhekov]

According to that document (and some others), http://developer.yahoo.com/performance/rules.html
, the returned header is:

Content-Encoding: gzip

( headers["Content-Encoding"] = "gzip" )

Which is better/right - deflate of gzip?

[Ezra Zygmuntowicz]

On Jun 12, 2008, at 6:40 PM, Stoyan Zhekov wrote:

>
> According to that document (and some others), http://developer.yahoo.com/performance/rules.html
> , the returned header is:
>
> Content-Encoding: gzip
>
> ( headers["Content-Encoding"] = "gzip" )
>
> Which is better/right - deflate of gzip?

I guess we should support both:

def call(env)
accept = env['HTTP_ACCEPT_ENCODING']
status, headers, body = @app.call(env)

if accept and [ 'deflate', 'gzip'].include? accept

headers["Content-Encoding"] = "deflate"
body = deflate(body)
end
[status, headers, body]
end

Cheers-
- Ezra

[Ezra Zygmuntowicz]

On Jun 12, 2008, at 8:56 PM, Ezra Zygmuntowicz wrote:

> def call(env)
> accept = env['HTTP_ACCEPT_ENCODING']
> status, headers, body = @app.call(env)
> if accept and [ 'deflate', 'gzip'].include? accept

> - headers["Content-Encoding"] = "deflate"
> + headers["Content-Encoding"] = accept

> body = deflate(body)
> end
> [status, headers, body]
> end

- Ezra

[Christian Neukirchen]

Ezra Zygmuntowicz <ezmo...@gmail.com> writes:

> if accept and [ 'deflate', 'gzip'].include? accept

Is this the best way we can parse HTTP headers? Seems a bit to eager to me.

[Ezra Zygmuntowicz]

On Jun 13, 2008, at 5:36 AM, Christian Neukirchen wrote:

>
> Ezra Zygmuntowicz <ezmo...@gmail.com> writes:
>
>> if accept and [ 'deflate', 'gzip'].include? accept
>
> Is this the best way we can parse HTTP headers? Seems a bit to
> eager to me.

I don't get what you are saying? There is no parsing going on here.

- Ezra

[Christian Neukirchen]

Ezra Zygmuntowicz <ezmo...@gmail.com> writes:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Accessing the value of Accept-Encoding seems to be more complex to me?

[spicyj]

WEBRick has the parsing implemented as
WEBrick::HTTPUtils::parse_qvalues(env['HTTP_ACCEPT_ENCODING']).

http://ruby-doc.org/stdlib/libdoc/webrick/rdoc/classes/WEBrick/HTTPUtils.html#M009863

It returns an array sorted in order of browser preference. (Though it
appears not to remove q=0.0 encodings.)

Also, I don't think that gzip and deflate are interchangable formats.
Maybe we need to use Zlib::GzipWriter?

[Christoffer Sawicki]

On Sat, Jun 14, 2008 at 2:02 PM, Christian Neukirchen
<chneuk...@gmail.com> wrote:
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
>
> Accessing the value of Accept-Encoding seems to be more complex to me?

I took a quick stab at implementing it properly:
http://pastie.org/private/gkznbcflnc3uhwxoebxa

The accepted_encodings methods definitely needs a second look and a new name.
Provide a little feedback and I might finish it for inclusion. :-)

Cheers,
Christoffer Sawicki

[Christian Neukirchen]

"Christoffer Sawicki" <christoff...@gmail.com> writes:

This looks good. How does it compare to
http://chneukirchen.org/repos/coset/lib/coset/mimeparse.rb ?

> Cheers,
> Christoffer Sawicki

[spicyj]

> This looks good. How does it compare to
> http://chneukirchen.org/repos/coset/lib/coset/mimeparse.rb?

Looks like the same functionality, and it should be -- the RFC2616
(HTTP/1.1) reads:
> The Accept-Encoding request-header field is similar to Accept, but restricts
> the content-codings (section 3.5) that are acceptable in the response.
>
> (from http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html)

So there should be new name applicable to all of the Accept headers,
as you suggested, and now we can finish the compression middleware.

[Christoffer Sawicki]

On Fri, Jun 27, 2008 at 1:40 PM, Christian Neukirchen
<chneuk...@gmail.com> wrote:

> "Christoffer Sawicki" <christoff...@gmail.com> writes:
>> I took a quick stab at implementing it properly:
>> http://pastie.org/private/gkznbcflnc3uhwxoebxa
>>
>> The accepted_encodings methods definitely needs a second look and a new name.
>> Provide a little feedback and I might finish it for inclusion. :-)
>
> This looks good. How does it compare to
> http://chneukirchen.org/repos/coset/lib/coset/mimeparse.rb ?

Encoding names are not on the type/subtype form so we can't use
MIMEParse as is for Accept-Encoding.

I don't think it's worth generalizing the current mimeparse.rb code to
cover both Accept and Accept-Encoding; handling Accept-Encoding is
much simpler than Accept (mostly because there's no need to deal with
type specificity).

Once I extend my Accept-Encoding code to handle * and drop the complex
regexp I'll ask the list for a review.

Cheers,
Christoffer Sawicki

[Christian Neukirchen]

"Christoffer Sawicki" <christoff...@gmail.com> writes:

> I don't think it's worth generalizing the current mimeparse.rb code to
> cover both Accept and Accept-Encoding; handling Accept-Encoding is
> much simpler than Accept (mostly because there's no need to deal with
> type specificity).
>
> Once I extend my Accept-Encoding code to handle * and drop the complex
> regexp I'll ask the list for a review.

Okay, so let's these things stay seperate.

[Christoffer Sawicki]

On Sat, Jun 28, 2008 at 6:07 PM, Christoffer Sawicki
<christoff...@gmail.com> wrote:
> Once I extend my Accept-Encoding code to handle * and drop the complex
> regexp I'll ask the list for a review.

I finished it yesterday: http://pastie.org/private/mfn66wqhxkjpky1rnvdha

Comments are very welcome - especially on the select_best_encoding tests.

Cheers,
Christoffer Sawicki

[Christian Neukirchen]

"Christoffer Sawicki" <christoff...@gmail.com> writes:

Looks good to me, thanks.

[Christoffer Sawicki]

On Tue, Jul 1, 2008 at 6:53 PM, Christian Neukirchen

<chneuk...@gmail.com> wrote:
> "Christoffer Sawicki" <christoff...@gmail.com> writes:

>> I finished it yesterday: http://pastie.org/private/mfn66wqhxkjpky1rnvdha
>>
>> Comments are very welcome - especially on the select_best_encoding tests.
>
> Looks good to me, thanks.

Okay, thanks. I pushed the patch to my public repo if anyone wants to fetch it:
http://github.com/Qerub/rack/commit/bd1f55e6729243b08f689a21d7716c208d6a2b62

Cheers,
Christoffer Sawicki

[Christian Neukirchen]

"Christoffer Sawicki" <christoff...@gmail.com> writes:

Is the rest of the gzip stuff good enough for merging now? (And does
it have tests?)

[Christoffer Sawicki]

On Tue, Jul 1, 2008 at 11:01 PM, Christian Neukirchen

<chneuk...@gmail.com> wrote:
>
> "Christoffer Sawicki" <christoff...@gmail.com> writes:
>
>> On Tue, Jul 1, 2008 at 6:53 PM, Christian Neukirchen
>> <chneuk...@gmail.com> wrote:
>>> "Christoffer Sawicki" <christoff...@gmail.com> writes:
>>>> I finished it yesterday: http://pastie.org/private/mfn66wqhxkjpky1rnvdha
>>>>
>>>> Comments are very welcome - especially on the select_best_encoding tests.
>>>
>>> Looks good to me, thanks.
>>
>> Okay, thanks. I pushed the patch to my public repo if anyone wants to fetch it:
>> http://github.com/Qerub/rack/commit/bd1f55e6729243b08f689a21d7716c208d6a2b62
>
> Is the rest of the gzip stuff good enough for merging now? (And does
> it have tests?)

I'm writing tests at the moment.

Cheers,
Christoffer Sawicki

[Christoffer Sawicki]

Okay, here comes Rack::Deflater:
http://github.com/Qerub/rack/commit/56a1e316b69303e642dc5146def52dc93db18d0c

Comments are very welcome, as usual.

Cheers,
Christoffer Sawicki

[Christian Neukirchen]

"Christoffer Sawicki" <christoff...@gmail.com> writes:

Add the missing newline and a useful error message, and I'll merge it.

[spicyj]

> Okay, here comes Rack::Deflater:http://github.com/Qerub/rack/commit/56a1e316b69303e642dc5146def52dc93...

I've added gzip support: http://github.com/soprano/rack/commit/6d835ba292c5dc2af68784ccf6a05352700d26b5

I can't get a full test case, because gzip needs a header for the
mtime of the data, changing the output. Is there a good way to test
the output (maybe gunzipping the output and comparing it to the
original)? I should probably also change it to use the Last-Modified
header as the mtime.

Also, should this be in a separate file? Apache's mod_deflate handles
both deflate and gzip encodings.

~spicyj

[Christian Neukirchen]

spicyj <spicyj...@gmail.com> writes:

>> Okay, here comes Rack::Deflater:http://github.com/Qerub/rack/commit/56a1e316b69303e642dc5146def52dc93...
>
> I've added gzip support: http://github.com/soprano/rack/commit/6d835ba292c5dc2af68784ccf6a05352700d26b5
>
> I can't get a full test case, because gzip needs a header for the
> mtime of the data, changing the output. Is there a good way to test
> the output (maybe gunzipping the output and comparing it to the
> original)? I should probably also change it to use the Last-Modified
> header as the mtime.

Sounds good.

> Also, should this be in a separate file? Apache's mod_deflate handles
> both deflate and gzip encodings.

One file is okay.

> ~spicyj

[Christoffer Sawicki]

On Fri, Jul 4, 2008 at 10:39 AM, spicyj <spicyj...@gmail.com> wrote:
> I can't get a full test case, because gzip needs a header for the
> mtime of the data, changing the output. Is there a good way to test
> the output (maybe gunzipping the output and comparing it to the
> original)?

That sounds like the best way to me.

> I should probably also change it to use the Last-Modified
> header as the mtime.

Sounds good.

> Also, should this be in a separate file? Apache's mod_deflate handles
> both deflate and gzip encodings.

I think it should be in the same file but that the file needs a better
name. Any ideas?

Cheers,
Christoffer Sawicki

[Christoffer Sawicki]

On Thu, Jul 3, 2008 at 11:27 AM, Christian Neukirchen
<chneuk...@gmail.com> wrote:
> Add the missing newline and a useful error message, and I'll merge it.

Okay, done!

I'd like you to merge these commits:

+ bd1f55e6729243b08f689a21d7716c208d6a2b62 Added support for
Accept-Encoding (via Request#accept_encoding and
Utils.select_best_encoding)
+ 56a1e316b69303e642dc5146def52dc93db18d0c Implemented Rack::Deflater
+ bb427052c1ecf1e8e673349c43c06cd683b28d4a deflater.rb - Added a TODO
+ e0bb0e4a22eeaeb09eea6977ef4eff481ac8ab5a deflater.rb - Removed an
unused require
+ 13a661e5af07fc1cdab88aa05c21d29667856c28 added gzip support to Rack::Deflater
+ 3fd03e6067164f63af4e3fe1ac60ca1e2dea07e9 deflater.rb - Added a
newline to the end
+ 070561230bc4de06b260a4f1a60aa72f1a6fad27 deflater.rb - Added an
error message for the 406 Not Acceptable case
+ 05fd9d0900f28ffdec5d62a9edba4fb7060a46e3 Removed some stray whitespace

Please note that spicyj's gzip support is included.

And while we're at it, these might also be a good candidate for inclusion:

+ e3f11a5d615fca9e7f6cb7ff0b97f1f4b05c659d spec_rack_utils.rb -
Reformulated two test case descriptions
+ dfe616dad05bf9fe38c2fcff07c15d7dec9fb8e9 spec_rack_utils.rb - Don't
print things on STDOUT in the Context tests
+ d2b8af6076e9159a0faeef21500fb501f83b71e9 spec_rack_handler.rb - Fixed typos

(Just to exercise my git-fu: Would it be a good way for you to cherry
pick most of these commits to a new local branch and then merge that
branch into master? This commit history isn't that interesting. Also,
I realize I should have used a feature branch for Rack::Deflater.)

Cheers,
Christoffer Sawicki

[Scytrin dai Kinthra]

dfe616dad05b was fixed upstream a few weeks past.
-- blink
stadik.net

[spicyj]

> I'd like you to merge these commits:
>

> [...]

>
> Please note that spicyj's gzip support is included.

It is? I don't see it (it could be there, though). Anyway, I fixed the
mtime and spec. You can merge these if you want:

6d835ba292c5dc2af68784ccf6a05352700d26b5 added gzip support to
Rack::Deflater
30210d6591c798f3f298297949c7ba7a92f149e1 added mtime for Deflater.gzip
and fixed gzip spec

I'm new to this whole Git(Hub) thing, so I might be doing all this
wrong. Just in case, here's the patch: http://pastie.org/private/ju0ngvbzvmuwzxecwqzeq.

~spicyj

[Christoffer Sawicki]

On Fri, Jul 4, 2008 at 5:04 PM, Scytrin dai Kinthra <scy...@gmail.com> wrote:
> dfe616dad05b was fixed upstream a few weeks past.

Oops, you're right... I wonder what went wrong. I either forgot to
pull from upstream before doing that commit (most likely) or you
waited more than 2 days before pushing your commit to GitHub. I find
it interesting that Git didn't inform me about the duplicate change
(in the shape of a merge conflict) but just ignored it.

Cheers,
Christoffer Sawicki

[Christoffer Sawicki]

>> I'd like you to merge these commits:
>>
>> [...]
>>
>> Please note that spicyj's gzip support is included.
>
> It is? I don't see it (it could be there, though).

It is there in the middle of the list -- trust me. Our commit SHA1s
are not the same though.

Please excuse the horrible formatting -- I promise to not paste the
output of git-cherry -v directly next time.

Cheers,
Christoffer Sawicki

[spicyj]

> It is there in the middle of the list -- trust me. Our commit SHA1s
> are not the same though.

Hey, it is. It seems I'm blind. You can merge 30210d6 if you want.

[spicyj]

The only thing I'm confused about is why it's not here:

http://github.com/Qerub/rack/tree/master/lib/rack/deflater.rb

Can you explain? Thanks!

[Christoffer Sawicki]

There is a simple explanation: I'm a moron and forgot to push my
changes to GitHub. My mind has obviously not gotten completely used to
the decentralized nature of Git yet. That said, everything should be
right now; take a look at http://github.com/Qerub/rack/commits/master.

Cheers,
Christoffer Sawicki

[spicyj]

> There is a simple explanation: I'm a moron and forgot to push my
> changes to GitHub. My mind has obviously not gotten completely used to
> the decentralized nature of Git yet.

Hey, this was my first time using Git too. Looks good now, though.

[spicyj]

Should we only compress text-based responses? Yahoo! Best Practices[1]
says:

> It's worthwhile to compress any text response including XML and JSON.
> Image and PDF files should not be gzipped because they are already
> compressed. Trying to gzip them not only wastes CPU but can potentially
> increase file sizes.

If so, here's a patch.

http://github.com/soprano/rack/commit/2621b014d5b16f91277e126832cce12369e75575

Of course, the other thing we could try is zipping the response
regardless,
but then only return the compressed body if it's shorter.

[Christian Neukirchen]

"Christoffer Sawicki" <christoff...@gmail.com> writes:

> (Just to exercise my git-fu: Would it be a good way for you to cherry
> pick most of these commits to a new local branch and then merge that
> branch into master? This commit history isn't that interesting. Also,
> I realize I should have used a feature branch for Rack::Deflater.)

Use rebase -i and remove all patch-lines you don't want in it.

[Christoffer Sawicki]

On Sat, Jul 5, 2008 at 4:00 PM, Christian Neukirchen
<chneuk...@gmail.com> wrote:
>
> "Christoffer Sawicki" <christoff...@gmail.com> writes:
>
>> (Just to exercise my git-fu: Would it be a good way for you to cherry
>> pick most of these commits to a new local branch and then merge that
>> branch into master? This commit history isn't that interesting. Also,
>> I realize I should have used a feature branch for Rack::Deflater.)
>
> Use rebase -i and remove all patch-lines you don't want in it.

Oh, right! I've rewritten the history now and pushed the new commits to GitHub:
http://github.com/Qerub/rack/commits/master

I'm not 100% certain it was a good idea and I therefore consider it an
experiment.

spicyj: Your next pull will include a notice about "forced update" and
you will probably have to --force the push afterwards because of my
rebasing.

Cheers,
Christoffer Sawicki

[Christoffer Sawicki]

On Sat, Jul 5, 2008 at 7:34 AM, spicyj <spicyj...@gmail.com> wrote:
>
> Should we only compress text-based responses? Yahoo! Best Practices[1]
> says:
>
>> It's worthwhile to compress any text response including XML and JSON.
>> Image and PDF files should not be gzipped because they are already
>> compressed. Trying to gzip them not only wastes CPU but can potentially
>> increase file sizes.

I think it's up to the user of Rack::Deflater to only use when it makes sense.

Cheers,
Christoffer Sawicki

[Christian Neukirchen]

"Christoffer Sawicki" <christoff...@gmail.com> writes:

> Oh, right! I've rewritten the history now and pushed the new commits to GitHub:
> http://github.com/Qerub/rack/commits/master
>
> I'm not 100% certain it was a good idea and I therefore consider it an
> experiment.

I rebased it on master and removed the unrelated commits, cherry
picking the ones I want too.

[Christoffer Sawicki]

On Sun, Jul 6, 2008 at 2:21 PM, Christian Neukirchen
<chneuk...@gmail.com> wrote:
>
> "Christoffer Sawicki" <christoff...@gmail.com> writes:
>
>> Oh, right! I've rewritten the history now and pushed the new commits to GitHub:
>> http://github.com/Qerub/rack/commits/master
>>
>> I'm not 100% certain it was a good idea and I therefore consider it an
>> experiment.
>
> I rebased it on master and removed the unrelated commits, cherry
> picking the ones I want too.

Thanks!

Cheers,
Christoffer Sawicki