GZip content using embedded Jetty and unfiltered

[Gerd Riesselmann]

I'd like to gzip content before passing it to the client (if client supports it) using the embedded Jetty server with unfiltered. However, I didn't manage to figure out how to do this.

Any suggestions?

[Doug Tangren]

On Tue, Jul 26, 2011 at 6:09 AM, Gerd Riesselmann <gerd.rie...@gmail.com> wrote:

I'd like to gzip content before passing it to the client (if client supports it) using the embedded Jetty server with unfiltered. However, I didn't manage to figure out how to do this.

Any suggestions?

You could dot this with a gzip output stream [1] providing a custom response combinator [2]
that wraps the response's [3] output stream. Make sure you verify your client's Accept-Encoding. header.

You get this for free if you proxy with an http server like nginx [4] though 



[1]: http://download.oracle.com/javase/6/docs/api/java/util/zip/GZIPOutputStream.html

[2]: https://github.com/n8han/Unfiltered/blob/master/library/src/main/scala/response/functions.scala#L10-21]
[3]: https://github.com/n8han/Unfiltered/blob/master/library/src/main/scala/response/HttpResponse.scala
[4]: http://wiki.nginx.org/HttpGzipModule


 
-Doug Tangren
http://lessis.me

[Nathan Hamblen]

Jetty also supports this as a handler [1] so it may be worth adding to our server interface, but I like the idea of doing it in unfiltered-library so that any backend can use it. It's not difficult since java gives us gzip streams. Dustin added the reverse to Dispatch way back when [2].

Nathan

[1]: http://download.eclipse.org/jetty/stable-7/apidocs/org/eclipse/jetty/server/handler/GzipHandler.html
[2]: https://github.com/n8han/Databinder-Dispatch/blob/master/core/src/main/scala/handlers.scala#L62

[Gerd Riesselmann]

Hello,

thanks for the replies. Unfortunately, I'm still stuck. I tried several approaches, but none of them worked.

I wrote a GZiptHttpResponse as a simple delegate to HttpRespone: 

/**

 * HttpResponse that gzips its content

 */

class GzipHttpResponse[T](val source: HttpResponse[T]) extends HttpResponse[T](source.underlying) {

source.addHeader("Content-Encoding", "gzip")

val gzipstream = new GZIPOutputStream(source.getOutputStream())

def setContentType(contentType : scala.Predef.String) : scala.Unit = source.setContentType(contentType)

def setStatus(statusCode : scala.Int) : scala.Unit = source.setStatus(statusCode)

def getWriter() : java.io.PrintWriter = source.getWriter

def getOutputStream() : java.io.OutputStream = gzipstream

def sendRedirect(url : scala.Predef.String) : scala.Unit = source.sendRedirect(url)

def addHeader(name : scala.Predef.String, value : scala.Predef.String) : scala.Unit = source.addHeader(name, value)

def cookies(cookie : scala.Seq[unfiltered.Cookie]) : scala.Unit = source.cookies(cookie)

}

I also added a ResponseFunction:

/**

 * GZip content, if client allows it

 */

object Gzip extends ResponseFunction[Any] {

def apply[T](res: HttpResponse[T]) = {

new GzipHttpResponse[T](res)

}

}

My first try was to write a Plan that checks Accept-Encoding and either returns GZip or Pass. This plan is then executed before all others. However, while it will successfully detects the encoding, it sends an empty response, since Gzip != Pass. I didn't figure out how to send both Gzip and Pass.

If I understand the code correctly, though, each Plan creates its own HttpRequest, so the approach would be generally wrong.

So my next try was to move it down the chain. There I figured the problem I need both the request and the response, since the request tells me, if the client supports compression. I however did not manage to write a Responder that takes both a request and a response in the apply() method.

My last approach was to overload Plan.doFilter. However I couldn't create a HttpRequest here, since the implementations are private[filter]. 

Now I'm stuck, and would be glad for some hints to get me on track again.

[Chris Lewis]

Hi,

I can't thoroughly test this at the moment, but this should do:

case class GZip(content: Array[Byte]) extends ResponseStreamer {
  def stream(os: OutputStream) {
    new java.util.zip.GZIPOutputStream(os).write(content)
  }
}

And you can use it like so:

GZip("Foo".getBytes)

or more correctly:

ContentEncoding("gzip") ~> GZip("Foo".getBytes) // chain to add the encoding header

ResponseStreamer is a Responder that writes to the output stream - this is what you want so you can wrap it with a gzip encoder. HttpResponse and HttpRequest are for you to use, but you needn't implement them unless you're creating a module for a new backend. Hopefully I'll be able to test this this evening; let me know how you fare.

-chris

[Gerd Riesselmann]

Hi Chris, thanks for your reply.

I tried it and it works, if I add an immediate close:

case class GZipString(content: String) extends ResponseStreamer {

def stream(os: OutputStream) {

val gos = new java.util.zip.GZIPOutputStream(os)

gos.write(content.getBytes)

gos.close()

}

}

Since I need the request to dermine if the client supports compression, I added it and ended up with this case class:

case class GZipString[T](request: HttpRequest[T], content: String) extends Responder[Any] {

override def respond(res: HttpResponse[Any]) {

request match {

case AcceptEncoding(values) => if (values.exists { s => s.equalsIgnoreCase("gzip") }) {

streamGZip(res)

} else {

stream(res)

}

case _ => stream(res)

}

}

def contentToBytes = content.getBytes("UTF-8")

def stream(res: HttpResponse[Any]) = res.getOutputStream.write(contentToBytes)

def streamGZip(res: HttpResponse[Any]) = {

res.addHeader("Content-Encoding", "gzip")

val gos = new java.util.zip.GZIPOutputStream(res.getOutputStream)

gos.write(contentToBytes)

gos.close()

}

}

(Error handling left out for clarity)

This works, as far as I can see now, as a simple drop in replacement for ResponseString - as long as you have access to the request.

Thank you all very much for your help!

[Gerd Riesselmann]

I uploaded it to GitHub: https://github.com/gerdriesselmann/unfiltered-gzip

[Gerd Riesselmann]

Hi there,

while the GZipResponseString generally works, I'm not really satisfied, since gzipping content is something that should be generally applied to all routes. 

I think the place for that is a Plan. 

But to be able to implement this, Plan needs some refactoring first. Here's how the doFilter function of the Plan trait is currently implement:

def doFilter(request: ServletRequest, response: ServletResponse, chain: FilterChain) {

    (request, response) match {

      case (hreq: HttpServletRequest, hres: HttpServletResponse) =>

        val request = new RequestBinding(hreq)

        val response = new ResponseBinding(hres)

        complete(intent)(request) match {

          ... Cut ...

        }

    }

}

The problem is the hard coded creation of RequestBinding and ResponseBinding, with make it very hard to switch them (e.g. to a response using a Gzip output stream). It should be refactored into a function:

def doFilter(request: ServletRequest, response: ServletResponse, chain: FilterChain) {

    (request, response) match {

      case (hreq: HttpServletRequest, hres: HttpServletResponse) =>

        val (request, response) = createBindings(hreq, hres)

          complete(intent)(request) match {

          ... Cut ...

        }

    }

}

 

def createBindungs(hreq: HttpServletRequest, hres: HttpServletResponse) = 

  (new RequestBindung(hreq), new ResponseBinding(hres))

I now should be able to create a GZipPlan like this:

trait GZipPlan extends Plan {

  override def createBindungs(hreq: HttpServletRequest, hres: HttpServletResponse) = {

    val (req, res) = super.createBindungs

    req match {

      case AcceptEncoding(values) => if (values.exists { s => s.equalsIgnoreCase("gzip") }) {

        (req, new GZipHttpResponse(res))

      } else {

        (req, res)

      }

      case _ => (res, req)

    }

  }

}

Having such a mechanism should also solve a couple of other problems like extracting a user and storing it on the request (or response).

[Chris Lewis]

Hi Gerd,

On 7/27/11 4:58 AM, Gerd Riesselmann wrote:

Hi there,

while the GZipResponseString generally works, I'm not really satisfied, since gzipping content is something that should be generally applied to all routes. 

I think the place for that is a Plan.

Well, the root plans aren't meant to need tweaking, but instead to wrap a backend and provide more or less uniform behavior to user plans implemented on those backends. In this specific scenario, you could use PassAndThen to apply post processing behavior (see PassAndThenSpec in the filter module). That said, I'm not convinced using PassAndThen for gzip would be a better choice than configuration at the server level. We've mentioned in other threads some work going on in the server interfaces to unify programming for different backends; I think this is worth rolling into those.

[Gerd Riesselmann]

Hi Chris, 

I was experimenting with PassAndThen, too, but it didn't worked. It caused state errors in the writer. I assume because content has already be sent to the original stream.

Cheers,

Gerd

[Nathan Hamblen]

On 07/27/2011 04:58 AM, Gerd Riesselmann wrote:
> The problem is the hard coded creation of RequestBinding and
> ResponseBinding, with make it very hard to switch them (e.g. to a
> response using a Gzip output stream). It should be refactored into a
> function:

These bindings are meant to abstract out the server backends, not to
contain application logic. You can write application logic above them in
filters, or higher than that in the server configuration, but most
application logic will go into the intent partial function.

Intents are composable, like any partial function. If you want to do
something for every request then you do it in the outermost function.
You see a bit of this in the SillyStore example:
http://unfiltered.databinder.net/Silly+Store.html

The first Path match has been factored out and made available to a
nested partial function. It isn't transforming the return value of that
function, but it easily could be.

We should start to exploit this in the core library more so that it's
clearer to people what they can do in applications, and to provide more
functionality that is reusable across all backends. I need to think a
little more about what it should be called and what exactly the types
are, but this is certainly feasible:

def intent = unfiltered.intents.Gzip {
case Path(...) => ResponseString(...)
case Path(...) => ResponseString(...)
}

Gzip#apply being a function that takes an intent partial function and
returns an intent partial function.

Nathan

[Dustin Whitney]

It would be nice to know that a response function should be terminal (I think you're suggesting that). For example, the Scalate response needs to be last because it will close the response when it's done with its business (Scalate the library does this).  It can be a little confusing when you first run into it - I don't really know how to address it though.

Dustin

[Doug Tangren]

   def intent = unfiltered.intents.Gzip {
      case Path(...) => ResponseString(...)
      case Path(...) => ResponseString(...)
  }

This style may also mix well with other "around" type behavior like authentication

   def privateResources = myapp.intents.Auth(...) {
      case Path(...) => // secure stuff
    }
 

[Dustin Whitney]

+1

[Chris Lewis]

+1

[Gerd Riesselmann]

I wouldn't say it was particularily easy, but I managed to write such an Intent:

object GZipIntent {

/**

* Wrapper around ResponseFunction that switches HttpResponse to GZip, if allowed

*/

case class GZipResponseFunctionWrapper[A](req: HttpRequest[A], f: ResponseFunction[Any]) extends ResponseFunction[Any] {

def apply[T](res: HttpResponse[T]) = req match {

case AcceptsEncoding.GZip(_) => {

val gos = new GZipHttpResponse(res)

f(gos)

}

case _ => f(res)

}

}

/**

* Factory to wrap original intent

*/

def apply[A](inner: Cycle.Intent[A,Any]): Cycle.Intent[A,Any] = {

case req @ _ => {

inner.orElse({ case _ => Pass }: Cycle.Intent[A,Any])(req) match {

case Pass => Pass

case responseFunction => GZipResponseFunctionWrapper(req, responseFunction)

}

}

}

}

There is some more, this is all @GitHub: https://github.com/gerdriesselmann/unfiltered-gzip

Thank you all for ideas and support!

Gerd

[Gerd Riesselmann]

I left out handling PassAndThen within the apply() function above, since I was not sure, what to do exactly. I now think wrapping it into a GZipResponseFunction would have too many side effects, so one should simply pass it along:

/**

 * Factory to wrap original intent

 */

def apply[A](inner: Cycle.Intent[A,Any]): Cycle.Intent[A,Any] = {

  case req @ _ => {

  inner.orElse({ case _ => Pass }: Cycle.Intent[A,Any])(req) match {

case after: PassAndThen => after

  case Pass => Pass

  case responseFunction => GZipResponseFunctionWrapper(req, responseFunction)

  }

  }

}

It may be necessary to wrap it for other use case, though.

[Nathan Hamblen]

On 07/28/2011 09:05 AM, Gerd Riesselmann wrote:

I left out handling PassAndThen within the apply() function above, since I was not sure, what to do exactly. I now think wrapping it into a GZipResponseFunction would have too many side effects, so one should simply pass it along:

...

It may be necessary to wrap it for other use case, though.

It's not... I'm thinking of deprecating PassAndThen entirely, unless someone objects.

Nathan

[Nathan Hamblen]

On 07/27/2011 09:08 AM, Dustin Whitney wrote:
> It would be nice to know that a response function should be terminal
> (I think you're suggesting that). For example, the Scalate response
> needs to be last because it will close the response when it's done
> with its business (Scalate the library does this). It can be a little
> confusing when you first run into it - I don't really know how to
> address it though.

Yeah... unfiltered.response.GZip won't work with the Scalate responder
at all, the way it's set up right now.

This reveals a bigger gap in the current response function hierarchy,
and now I see why Gerd was motivated to try to replace the streams in
the response binding. We don't really have much support for doing
streaming output the right way for arbitrarily large responses.

The idea was that you would extend ResponseStreamer and provide your own
implementation of `stream` to write to the outputstream. In writers you
have ResponseWriter. Those are both fine interfaces, but in practice we
have higher level responders (now including the gzip responder) that
serialize the output into one string or byte array.

So I see two problems: first that the provided response functions don't
seem to be meshing very well with common use cases, and second that we
don't give people an easy way to insert a FilterOutputStream into the
pipeline.

I think I have a good solution for the second problem, which is to
provide a copy function in the response binding trait similar to what
you have with case classes, so that you can effectively wrap and replace
the raw outputstream with another which will be used by any response
functions that compose around it. (I *knew* there was a reason I made
the type of these HttpResponse => HttpResponse.)

But that is still tricky to abstract out into an intent helper of the
type that we've been talking about.

def intent = unfiltered.intents.Gzip {
case Path(...) => ResponseString(...)
case Path(...) => ResponseString(...)
}

We need to cut to the front of the line of response functions... and we
can do that. I just need to come up with a way to structure and name all
of this that doesn't look like a total hack, because it's really not. :)

Nathan

[Doug Tangren]

It's not... I'm thinking of deprecating PassAndThen entirely, unless someone objects.

Go for it! I added it as an experiment for a potential use case but probably not a common one. I look back now an realize it was more of a hack than a solid solution and there are much better ideas on the list now for these kinds of problems. I say nix it and cut the fat.

[n8han]

Now part of the core library, unfiltered.kit.GZip:
https://github.com/n8han/Unfiltered/blob/master/library/src/main/scala/kit/gzip.scala

Tests:
https://github.com/n8han/Unfiltered/blob/master/library/src/test/scala/GzipSpec.scala

It will probably serve gzipped content to ancient versions of IE that lie about supporting it, so if anybody wants to make it smarter be my guest. I'm going to look at the scalate module whose tests I recently broke, and see if I can get it into the happy new output stream path.

Nathan

[Dustin Whitney]

Zounds! that was fast!

[Gerd Riesselmann]

Thumbs up!