http server chunked encoding question

2,009 views
Skip to first unread message

Cenk Altı

unread,
Jan 9, 2014, 4:38:27 PM1/9/14
to golan...@googlegroups.com
Hello,

I don't understand the http server behavior when no content length header is specified by the handler function.
The docs states that "The http package adds chunking automatically if handlers don't set a Content-Length header".

http://golang.org/pkg/net/http/httputil/#NewChunkedWriter

The "Transfer-Encoding: chunked" header is not set in the following code:


What may be the problem?

Thanks in advance.

Jesse McNelis

unread,
Jan 9, 2014, 4:48:57 PM1/9/14
to Cenk Altı, golang-nuts
It's set by the server, but the client removes it because reading from the Body of the response is automatically unchunked.



 
--
=====================
http://jessta.id.au

Cenk Altı

unread,
Jan 9, 2014, 4:57:34 PM1/9/14
to Jesse McNelis, golang-nuts
I understand this. 
But when I make the same request with curl, the "Transfer-Encoding" header appears but the content is not chunked.
Isn't this wrong?

cenk@tardis ~ (master) $ curl -i http://127.0.0.1:1234/
HTTP/1.1 200 OK
Date: Thu, 09 Jan 2014 21:55:29 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

abc


Dave Cheney

unread,
Jan 9, 2014, 5:24:41 PM1/9/14
to Cenk Altı, Jesse McNelis, golang-nuts
Maybe curl is unchunking the data for you. It is designed so you can do curl $URL > $FILE or something, so including the http transport data would be a big fail. 

If you want to know for sure use ngrep/wireshark or telnet to the port and issue the request yourself.


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Cenk Altı

unread,
Jan 9, 2014, 5:32:34 PM1/9/14
to Dave Cheney, Jesse McNelis, golang-nuts
No, curl does not do auto-unchunking. It prints the stream as it is received.
You can see the chunk size (a2) is printed before each chunk below:

HTTP/1.1 200 OK
Content-Type: application/json
X-Etcd-Index: 869
X-Raft-Index: 7160
X-Raft-Term: 23
Date: Thu, 09 Jan 2014 19:12:22 GMT
Transfer-Encoding: chunked

a2
{"action":"set","node":{"key":"/a","value":"1","modifiedIndex":870,"createdIndex":870},"prevNode":{"key":"/a","value":"1","modifiedIndex":869,"createdIndex":869}}
a2
{"action":"set","node":{"key":"/a","value":"1","modifiedIndex":871,"createdIndex":871},"prevNode":{"key":"/a","value":"1","modifiedIndex":870,"createdIndex":870}}
a2
{"action":"set","node":{"key":"/a","value":"1","modifiedIndex":872,"createdIndex":872},"prevNode":{"key":"/a","value":"1","modifiedIndex":871,"createdIndex":871}}

Dave Cheney

unread,
Jan 9, 2014, 5:50:11 PM1/9/14
to Cenk Altı, Jesse McNelis, golang-nuts
Hmm, I cannot reproduce that with my version of curl. Anyway, the problem appears related to the Go http server. If so, can you please paste a small example server that demonstrates the problem.

Kyle Lemons

unread,
Jan 9, 2014, 5:58:56 PM1/9/14
to Dave Cheney, Cenk Altı, Jesse McNelis, golang-nuts
It's not a bug that I can see.  Curl behaves for me in every case I tried.   You can even validate that your original example works fine on the playground:

Jesse McNelis

unread,
Jan 9, 2014, 6:22:29 PM1/9/14
to Cenk Altı, Dave Cheney, golang-nuts
On Fri, Jan 10, 2014 at 9:32 AM, Cenk Altı <cenk...@gmail.com> wrote:
No, curl does not do auto-unchunking. It prints the stream as it is received.
You can see the chunk size (a2) is printed before each chunk below:

HTTP/1.1 200 OK
Content-Type: application/json
X-Etcd-Index: 869
X-Raft-Index: 7160
X-Raft-Term: 23
Date: Thu, 09 Jan 2014 19:12:22 GMT
Transfer-Encoding: chunked

a2
{"action":"set","node":{"key":"/a","value":"1","modifiedIndex":870,"createdIndex":870},"prevNode":{"key":"/a","value":"1","modifiedIndex":869,"createdIndex":869}}
a2
{"action":"set","node":{"key":"/a","value":"1","modifiedIndex":871,"createdIndex":871},"prevNode":{"key":"/a","value":"1","modifiedIndex":870,"createdIndex":870}}
a2
{"action":"set","node":{"key":"/a","value":"1","modifiedIndex":872,"createdIndex":872},"prevNode":{"key":"/a","value":"1","modifiedIndex":871,"createdIndex":871}}

It looks like the problem is actually etcd.
etcd is double chunking the data.


etcd is chunking the response body data itself and then passing it on to the http response which is chunking it again.

The chunks you're seeing are not the chunks at the http level.
curl is infact unchunking the data.



--
=====================
http://jessta.id.au

minux

unread,
Jan 9, 2014, 6:33:50 PM1/9/14
to Cenk Altı, Dave Cheney, Jesse McNelis, golang-nuts
On Thu, Jan 9, 2014 at 5:32 PM, Cenk Altı <cenk...@gmail.com> wrote:
No, curl does not do auto-unchunking. It prints the stream as it is received.
You can see the chunk size (a2) is printed before each chunk below:
curl is dechunking the data here. You can verify that with telnet.

$ telnet localhost 1234
Trying ::1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Date: Thu, 09 Jan 2014 23:32:04 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

1
a
1
b
1
c
0


Alex Skinner

unread,
Jan 9, 2014, 6:43:50 PM1/9/14
to golan...@googlegroups.com, Cenk Altı, Dave Cheney, Jesse McNelis
This is correct.  From curl's docs -

CURLOPT_HTTP_TRANSFER_DECODING
Pass a long to tell libcurl how to act on transfer decoding. If set to zero,
transfer decoding will be disabled, if set to 1 it is enabled
(default). libcurl does chunked transfer decoding by default unless this
option is set to zero. (added in 7.16.2)


Thanks,
Alex

Cenk Altı

unread,
Jan 10, 2014, 5:45:42 AM1/10/14
to Alex Skinner, golang-nuts, Dave Cheney, Jesse McNelis
I have mistaken, sorry. etcd was doing double chunking. Thank you for your answers.
Reply all
Reply to author
Forward
0 new messages