Should http.Client.Do decompress gzipp'ed replies?

[Elazar Leibovich]

It seems that client.Do is supposed to return a response whose Body is wrapped with gzip reader if the Content-Encoding:gzip header is received. However look at the following code (see https://gist.github.com/2050445 for the full example)

        req,err := http.NewRequest("GET","http://bash.cyberciti.biz/wp-content/themes/bashshell/custom/images/headerbg.gif",nil)

        if err != nil {log.Fatal(err)}

        req.Header.Set("Accept-Encoding","gzip")

        if err != nil {log.Fatal(err)}

        client := &http.Client{}

        resp,err := client.Do(req)

        if err != nil {log.Fatal(err)}

        r,err := gzip.NewReader(bytes.NewReader(b))

        if err == nil {

                log.Println("Hold it, resp.Body should be plain text!")

                if _,err := r.Read([]byte{0}); err == nil {

                        log.Println("And we can even read the first byte!")

                }

        }

I'm sending a request to an arbitrary server, accepting gzip encoding, and I expect `client.Do` to wrap the body response with gzip reader. That's the job of the Transport, which is below the client, and that's what should happen if I use client.Get/Post/Head. I can't find in the documentation reference to whether or not response.Body should be compressed, but I think it makes sense it will.

Am I correct in this case? If not, how can I easily do something like http.DecompressResponseBodyAccordingToHeaders(resp)?

[Mike Rosset]

> It seems that client.Do is supposed to return a response whose Body is
> wrapped with gzip reader if the Content-Encoding:gzip header is received.
> However look at the following code

Do you have some documentation pointing to this? as far as I know you
need to handle the reader yourself which is pretty easy.

I forked your gist and provided an example. see

https://gist.github.com/2050623

[Patrick Mylund Nielsen]

Ah--beat me to it. You can just replace the body reader itself, too:

package main

import (
"compress/gzip"
"net/http"
)

func main() {
req, err := http.NewRequest("GET",

"http://bash.cyberciti.biz/wp-content/themes/bashshell/custom/images/headerbg.gif",
nil)
if err != nil {

panic(err)

}
req.Header.Set("Accept-Encoding", "gzip")

res, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
if res.Header.Get("Content-Encoding") == "gzip" {
res.Body, err = gzip.NewReader(res.Body)
if err != nil {
panic(err)
}
defer res.Body.Close()
}
// ...
}

(Don't forget to close both readers!)

[Kyle Lemons]

I believe the behavior, last time I was trolling the source, was to only transparently gzip decode when the "accept: gzip" was added by the library.  If you set it yourself, it expects you to also do the decoding yourself.

[Elazar Leibovich]

Some points:

1. These things should be in the documentaion (eg http.Get is free to add headers to your request, as long as they would not affect the response)
2. What about chunked encoding, should Client.Do let you handle it as well?
3. I think that the functionality "Read a request, and parse whatever transport headers you can" should be exposed by the http package. No reason to implement it twice.
   Will you accept a patch that expose this functionality? http.PreprocessResponse?

[Brad Fitzpatrick]

Please file a bug requesting more documentation and feel free to assign it to me.

Thanks!

[Elazar Leibovich]

Filed http://code.google.com/p/go/issues/detail?id=3359 feel free to use the content of the

(I can't assign it to anyone, so you'll have to do that).

Thanks