Cookies not being set after content has been written to http.ResponseWriter

[tomwilde]

Hi

I just noticed that if I try to http.SetCookie on an http.ResponseWriter that has already printed some content it will fail silently.

Is this ment to be this way? As far as I understand the "Transfer-Encoding: chunked" header signifies that headers might be set after the message body...

This is on Go v1.0.2.

I also noticed that the "Content-Type" header randomly changes to "application/octet-stream" (instead of the default "text/plain; charset=utf-8") triggering a download on my Google Chrome (v20.0) instead of showing the page (which has text in the message)

What is going on?

As a workaround; how can I disable the chunked transfer behavior? 

[tomwilde]

OK I just figured out that the "application/octet-stream" is set if I use w.Write([]byte("Hello world")) as opposed to fmt.Fprintln(w, "Hello world"). Is this ment to work like this? Don't mess with my content-type!

[tomwilde]

Code to reproduction (won't run on playground for obvious reasons): http://play.golang.org/p/DFasbD_rwD (Note comments in cookie)

The "application/octet-stream" problem seems to occur when the output contains non-ASCII characters or something... I couldn't reproduce it cleanly.

[Chris Broadfoot]

If you care about the Content-Typr header, how come you're not setting it yourself?

[tomwilde]

Believe me, I am trying. Even if I set it as the last statement in my handler it gets overwritten in the final output (and there's absolutely no more code of mine running after this):

w.Header().Set("Content-Type", "text/plain; charset=utf-8")

Am Samstag, 14. Juli 2012 02:10:28 UTC+2 schrieb Chris Broadfoot:

If you care about the Content-Typr header, how come you're not setting it yourself?

[tomwilde]

OK so to reproduce the Content-Type header issue do:

fmt.Fprintln(w, "Hello world!" + string(2))

The content-type detection system is cool and all but if I explicitly set the header right above or beyond that line I expect that to be respected.

You might be asking yourself "why would you possibly be printing low-ASCII to your users?" answer is I'm not; I'm debugging my application and I printed a bson.ObjectId (labix.org/v2/mgo/bson) which happens to be string-based.

Point is, this is unexpected behavior and wrong in accordance with the documentation, see doc for http.ResponseWriter:

// If the Header does not contain a 

// Content-Type line, Write adds a Content-Type set to the result of passing 

// the initial 512 bytes of written data to DetectContentType.

I explicitly set a Content-Type header BEFORE writing to the ResponseWriter and my content still got DetectContentType-ed.

Cookie topic still open...

[Patrick Mylund Nielsen]

package main

import (

        "fmt"

        "net/http"

)

func Handler(w http.ResponseWriter, req *http.Request) {

        w.Header().Set("Content-Type", "text/plain; charset=utf-8")

        fmt.Fprintln(w, "Hello world!" + string(2))

}

func main() {

        http.Handle("/", http.HandlerFunc(Handler))

        http.ListenAndServe(":8080", nil)

}

# go run foo.go &

# curl -I localhost:8080/

HTTP/1.1 200 OK

Content-Type: text/plain; charset=utf-8

Date: Sat, 14 Jul 2012 00:55:07 GMT

[tomwilde]

You're right... it's Chrome detecting the content and not Go, sorry for the noise.

Cookie thing still open.

PS: With chunked transfer encoding, even if w.Write() sets a detected Content-Type header, a later call to w.Header().Set(...) should still be able to overwrite that header... so it comes down to the same issue as the cookie not being set.

Am Samstag, 14. Juli 2012 02:57:25 UTC+2 schrieb Patrick Mylund Nielsen:

package main

import (

        "fmt"

        "net/http"

)

func Handler(w http.ResponseWriter, req *http.Request) {

        w.Header().Set("Content-Type", "text/plain; charset=utf-8")

        fmt.Fprintln(w, "Hello world!" + string(2))

}

func main() {

        http.Handle("/", http.HandlerFunc(Handler))

        http.ListenAndServe(":8080", nil)

}

# go run foo.go &

# curl -I localhost:8080/

HTTP/1.1 200 OK

Content-Type: text/plain; charset=utf-8

Date: Sat, 14 Jul 2012 00:55:07 GMT

[David Symonds]

On Sat, Jul 14, 2012 at 11:09 AM, tomwilde <sedevel...@gmail.com> wrote:

> PS: With chunked transfer encoding, even if w.Write() sets a detected
> Content-Type header, a later call to w.Header().Set(...) should still be
> able to overwrite that header... so it comes down to the same issue as the
> cookie not being set.

No, it can't. It's impossible. As soon as you've done a Write that
data has possibly already been sent over the wire, which means the
header has been written too, which means it's too late to change the
header.


Dave.

[tomwilde]

http://en.wikipedia.org/wiki/Chunked_transfer_encoding

"Chunked encoding allows the sender to send additional header fields after the message body. This is important in cases where values of a field cannot be known until the content has been produced such as when the content of the message must be digitally signed."

"The final chunk is a regular chunk, with the exception that its length is zero and may be followed by an optional trailer of additional entity header fields that are normally delivered in the HTTP header to allow the delivery of data that can only be computed after all chunk data has been generated. The sender may indicate in a Trailer header field which additional fields it will send in the trailer after the chunks." 

Maybe sending the Content-Type header in the trailer would be a little awkward for the browser, I agree. The cookie though, must be set.

Am Samstag, 14. Juli 2012 03:12:32 UTC+2 schrieb David Symonds:

[Patrick Mylund Nielsen]

This is one of those things, like pipelining, that is both tricky to implement and very poorly supported. The pain vs. gain factor is usually not worth it.

net/http sends the headers before the first write, by design. This is the documented behavior:

// A ResponseWriter interface is used by an HTTP handler to                                                          

// construct an HTTP response.                                                                                       

type ResponseWriter interface {

        // Header returns the header map that will be sent by WriteHeader.                                           

        // Changing the header after a call to WriteHeader (or Write) has                                            

        // no effect.                                                                                                

        Header() Header

...

You could write a wrapper that accomplishes what you are looking for. You could submit it as a CL, but--implementation difficulties aside--keep in mind that the documented behavior of the Go 1 API can't change.

[David Symonds]

So by header you mean trailers, right? I think Brad was thinking about
them, but almost no-one cares. You might just be an edge case. Why
can't you arrange for the cookie to be sent before the body is
written?


Dave.

[tomwilde]

Hi,

I can but I don't want to. That's the last thing I wanna worry about when building an application.

Why implement chunked transfer encoding as the default response type if it's not completely functional?

For dynamic application these days it's almost a requirement to be able to abort a response and show an error page instead. With the current implementation this becomes a pain to the user because we have to buffer our output somewhere and keep that where accessible from all our methods and whatnot, while (hopefully) 90+% of the request-response-cycles will be successful (without exceptions occurring).

I'll just make another response writer myself. I honestly expected more of the standard lib =/

Cheers and good night

[Patrick Mylund Nielsen]

> For dynamic application these days it's almost a requirement to be able to abort a response and show an error page instead.

[Citation needed]

Do you know of any large website that uses chunked transfer-encoding like you describe?

As soon as you start doing any kind of authentication and templating, trailers are pretty much useless. And nobody supports them--or the different implementations vary.

http://code.google.com/p/chromium/issues/detail?id=33325

https://issues.apache.org/bugzilla/show_bug.cgi?id=49921

http://connect.microsoft.com/VisualStudio/feedback/details/618853/net-4-0-discards-trailer-headers-on-chunked-encoded-http-responses

http://osdir.com/ml/cherrypy-users/2010-09/msg00022.html

...

> I'll just make another response writer myself. I honestly expected more of the standard lib =/

A good example of how not to sell an argument.

[Brad Fitzpatrick]

On Fri, Jul 13, 2012 at 6:39 PM, tomwilde <sedevel...@gmail.com> wrote:

Hi,

I can but I don't want to. That's the last thing I wanna worry about when building an application.

Sorry, that's how HTTP in the wild works.

You'll have to change your app.  This isn't a Go issue.

 

Why implement chunked transfer encoding as the default response type if it's not completely functional?

Nobody in the wild supports HTTP Trailers, sorry.  The Go HTTP package supports them, but that assumes you have a peer that also supports them (likely only Go).

[tomwilde]

Yes, you are right. That's how HTTP works and I wasn't aware there was so poor support for trailers.

Maybe the best solution for Go 1 would be to document this more prominently and centrally in a single chunk of text in the net/http package. For later versions this issue could eventually be revisited and maybe simplified as Peter suggests.

For people coming from interpreted web-languages (like me, from PHP) this stuff is being taken care of behind the scenes and it's confusing when we suddenly encounter that cookies are not being set for some reason.

The good thing to the story is how easy it is to just create another response writer. And that's why Go still rocks!

[Kyle Lemons]

On Sat, Jul 14, 2012 at 3:13 AM, tomwilde <sedevel...@gmail.com> wrote:

Yes, you are right. That's how HTTP works and I wasn't aware there was so poor support for trailers.

Maybe the best solution for Go 1 would be to document this more prominently and centrally in a single chunk of text in the net/http package. For later versions this issue could eventually be revisited and maybe simplified as Peter suggests.

For people coming from interpreted web-languages (like me, from PHP) this stuff is being taken care of behind the scenes and it's confusing when we suddenly encounter that cookies are not being set for some reason.

Does PHP do this for you now?  Last time I used the language (admittedly awhile ago) you also had to set cookies before any content was written.

[tomwilde]

To be completely honest I have always used MVC frameworks in PHP and I'm not certainly sure if it was the framework or the parser taking care of it.

One of the reasons I moved away from PHP is the level of magic and abstraction involved - which makes Go feel very controllable and steady for me.

I just have to find a Go job now and I'll be completely done with PHP :-)

[Dave Cheney]

> Does PHP do this for you now? Last time I used the language (admittedly
> awhile ago) you also had to set cookies before any content was written.

ISTR that most php installations buffer the first few kb of output,
allowing you to set headers after the page has started to render
(which may occur unexpectedly if there is whitespace at the top of
your .php file). It's crufty at best.