How to disable 'Transfer-Encoding: chunked' header?

[andrei]

Hi!

I've got an issue with node.js v0.1.20 on Webfaction hosting. I try to
run node.js hello world app at a custom port behind their proxy.

And instead of returning "Hello World" it appends weird chars to the
body like this:

b
Hello World
0

Their administrator told me that this is a result of the 'chunked'
Transfer-encoding set by the node.js server.

Is there any way to disable it, how should I change the source code of
hello world app?

[Christopher Lenz]

On 03.12.2009, at 10:37, andrei wrote:
> Their administrator told me that this is a result of the 'chunked'
> Transfer-encoding set by the node.js server.

He's right :) Proper HTTP clients should have no problem handling this, though.

> Is there any way to disable it, how should I change the source code of
> hello world app?

Set a Content-Length header in sendHeader().

Cheers,
--
Christopher Lenz
cml...@gmail.com
http://www.cmlenz.net/

[andrei]

Does it mean I have to calculate Content-Length of every response, or
is there any other option?

[Christopher Lenz]

On 03.12.2009, at 11:08, andrei wrote:
> Does it mean I have to calculate Content-Length of every response, or
> is there any other option?

Yes, afaik the only other option in node.js is to use chunked encoding. Not sure why you were having problems with that, it should normally work just fine.

[andrei]

It works well without their proxy. If ssh to server and run "curl
http://127.0.0.1:40740" then I will receive the right "Hello World",
but the problem appears when I map this app to public domain via their
proxy.

[Hagen]

chunked is a known problem with all mainstream reverse proxies. fixing
them would be best, giving node the capability to disable chunking the
most pragmatic option. this option must be transparent to code running
ontop of node.

5 sentences or less

> --
>
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en
> .
>
>

[Christopher Lenz]

On 03.12.2009, at 12:40, Hagen wrote:
> chunked is a known problem with all mainstream reverse proxies. fixing
> them would be best, giving node the capability to disable chunking the
> most pragmatic option. this option must be transparent to code running
> ontop of node.

Actually I think this is a bug in the node.js HTTP server. Assuming those proxies specify the HTTP version as 1.0 in their requests, node should not be doing chunked encoding, but rather stream out the body chunks normally and then close the connection.

[Felix Geisendörfer]

We are using the pound load balancer and it does an excellent job at
handling http. It supports streaming for incoming as well as outgoing
http requests and also handles chunked encoding with grace. However,
it seems like you have no control over this, so yes, specifying the
Content-Length header *is* your only option : /.

-- Felix Geisendörfer

On Dec 3, 12:40 pm, Hagen <six...@gmail.com> wrote:
> chunked is a known problem with all mainstream reverse proxies. fixing  
> them would be best, giving node the capability to disable chunking the  
> most pragmatic option. this option must be transparent to code running  
> ontop of node.
>
> 5 sentences or less
>

[Felix Geisendörfer]

I think you are right, RFC 2616 says: "A server MUST NOT send transfer-
codings to an HTTP/1.0 client."

I'll have a look, should be easy enough to patch.

-- Felix Geisendörfer

[Christopher Lenz]

On 03.12.2009, at 13:15, Felix Geisendörfer wrote:
> I think you are right, RFC 2616 says: "A server MUST NOT send transfer-
> codings to an HTTP/1.0 client."
>
> I'll have a look, should be easy enough to patch.

This change seems to work for me:

<http://github.com/cmlenz/node/commit/4fe22f583345a3e6cdf222fb7b8418053a4694f1>

[Felix Geisendörfer]

> This change seems to work for me:

Yip, good solution. I would suggest going even simpler so:

Pretty: http://github.com/felixge/node/commit/0e081775147bd6b7917904933567bd3e2340c732
Patch: http://github.com/felixge/node/commit/0e081775147bd6b7917904933567bd3e2340c732.patch

(Also added a unit test)

-- Felix Geisendörfer aka the_undefined

On Dec 3, 1:28 pm, Christopher Lenz <cml...@gmail.com> wrote:
> On 03.12.2009, at 13:15, Felix Geisendörfer wrote:
>
> > I think you are right, RFC 2616 says: "A server MUST NOT send transfer-
> > codings to an HTTP/1.0 client."
>
> > I'll have a look, should be easy enough to patch.
>
> This change seems to work for me:
>

> <http://github.com/cmlenz/node/commit/4fe22f583345a3e6cdf222fb7b841805...>

[andrei]

Thanks! The patch helped my problem.

On Dec 3, 3:45 pm, Felix Geisendörfer <fe...@debuggable.com> wrote:
> > This change seems to work for me:
>
> Yip, good solution. I would suggest going even simpler so:
>

> Pretty:http://github.com/felixge/node/commit/0e081775147bd6b7917904933567bd3...
> Patch:http://github.com/felixge/node/commit/0e081775147bd6b7917904933567bd3...

[Tautologistics]

Are you using telnet to test? "b" is hex for 11 and "Hello world"
is... 11 characters. Those characters are not being sent by the
server.

[Tim Caswell]

Actually I've also seen this error once when running a node server behind an Nginx reverse proxy. When the browser hit the node app directly the extra characters weren't there. I'm glad we're finally making sense of this one.

[Christopher Lenz]

On Dec 3, 1:45 pm, Felix Geisendörfer <fe...@debuggable.com> wrote:
> > This change seems to work for me:
>
> Yip, good solution. I would suggest going even simpler so:
>

> Pretty:http://github.com/felixge/node/commit/0e081775147bd6b7917904933567bd3...
> Patch:http://github.com/felixge/node/commit/0e081775147bd6b7917904933567bd3...

>
> (Also added a unit test)

Cool, here's a follow-up that does the same for "HTTP/0.9" mode, just
to be safe:

http://github.com/cmlenz/node/commit/49e8d44db5d2b98332534b0f50a1b6dac3c7ae63

This gets triggered when you send a request that lacks a HTTP version,
e.g. just "GET /"

[eugene]

That's what I'm trying to do now and I can't get around this. I've got
nginx reverse proxying to my node.js server.

My node.js server is on port 8000 and returns:

HTTP/1.1 200 OK
Content-Type: text/plain
Connection: keep-alive
Transfer-Encoding: chunked

Hello World

My nginx returns:

HTTP/1.1 200 OK
Server: nginx/0.7.64
Date: Fri, 04 Dec 2009 11:30:33 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Transfer-Encoding: chunked

5
Hello
6
World
0

And when I open this (the nginx front-end server) in safari I get:

5
Hello
6
World
0

Very annoying.

How do I get this to work properly :-(

Did you have any luck resolving it?

Cheers,

Eugene

On Dec 4, 1:54 am, Tim Caswell <t...@creationix.com> wrote:
> Actually I've also seen this error once when running a node server behind an Nginx reverse proxy.  When the browser hit the node app directly the extra characters weren't there.  I'm glad we're finally making sense of this one.
>
> On Dec 3, 2009, at 8:51 AM, Tautologistics wrote:
>
>
>
> > Are you using telnet to test? "b" is hex for 11 and "Hello world"
> > is... 11 characters. Those characters are not being sent by the
> > server.
>
> > On Dec 3, 4:37 am, andrei <andre...@gmail.com> wrote:
> >> Hi!
>
> >> I've got an issue with node.js v0.1.20 on Webfaction hosting. I try to
> >> run node.js hello world app at  a custom port behind their proxy.
>
> >> And instead of returning "Hello World" it appends weird chars to the
> >> body like this:
>
> >> b
> >> Hello World
> >> 0
>
> >> Their administrator told me that this is a result of the 'chunked'

> >> Transfer-encodingset by the node.js server.

[eugene]

Hi Christopher,

This fixed my problem with the reverse nginx proxy issues, but then
basic HTTP servers just hung without returning.

Ie. When I go to a server with HTTP 1.1 client (eg. curl or safari)
then it would not get any data and not return. When I killed the
node.js server the rest of the data flushed out. Weird!

Any ideas?

Cheers,

Eugene

On Dec 3, 11:28 pm, Christopher Lenz <cml...@gmail.com> wrote:
> On 03.12.2009, at 13:15, Felix Geisendörfer wrote:
>
> > I think you are right, RFC 2616 says: "A server MUST NOT send transfer-
> > codings to an HTTP/1.0 client."
>
> > I'll have a look, should be easy enough to patch.
>
> This change seems to work for me:
>

> <http://github.com/cmlenz/node/commit/4fe22f583345a3e6cdf222fb7b841805...>

[Christopher Lenz]

On 04.12.2009, at 13:08, eugene wrote:
> This fixed my problem with the reverse nginx proxy issues, but then
> basic HTTP servers just hung without returning.
>
> Ie. When I go to a server with HTTP 1.1 client (eg. curl or safari)
> then it would not get any data and not return. When I killed the
> node.js server the rest of the data flushed out. Weird!
>
> Any ideas?

That's with the reverse proxy inbetween or not?

If yes, I'd guess node.js is not closing the connection, so the proxy (not having received a content-length) just sits there waiting for the connection to get closed. However, in my testing node.js does automatically close the connection for HTTP/1.0 clients, so I'm not sure why you'd see this behavior. Can you log the request method, version and headers the node server gets sent by the reverse proxy and post them?

[eugene]

OK.

When I make the change in http.js this is what I get when I hit nginx.
Config is nginx is listening on 80 and reverse proxying to a node.js
server on port 8000.

So when I hit nginx:

[eugene@eugenes-macbook /usr/local/lib/node/libraries]$ curl -i
http://localhost

HTTP/1.1 200 OK
Server: nginx/0.7.64

Date: Sat, 05 Dec 2009 03:41:42 GMT

Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive

Hello World = 8000

---

Here are the headers received by the node.js server:

[eugene@eugenes-macbook ~/Documents/work/nodedev/example] master$ ./
example.js -p 8000
Server running at http://127.0.0.1:8000/
user-agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4
OpenSSL/0.9.8k zlib/1.2.3
host: localhost:8000
accept: */*
host: 127.0.0.1:8000
connection: close
user-agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4
OpenSSL/0.9.8k zlib/1.2.3
accept: */*

----

no problems there. When I now go straight to the node.js server
listening on port 8000, however:

[eugene@eugenes-macbook /usr/local/lib/node/libraries]$ curl -i
http://localhost:8000

HTTP/1.1 200 OK
Content-Type: text/plain
Connection: keep-alive

--- (it just hangs - actually it waits for a few minutes and then
returns) ---

The headers received by the node.js server:

[eugene@eugenes-macbook ~/Documents/work/nodedev/example] master$ ./
example.js -p 8000
Server running at http://127.0.0.1:8000/
user-agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4
OpenSSL/0.9.8k zlib/1.2.3
host: localhost:8000
accept: */*


--- (just hanging - but the http version is 1.0 - i've verified this)
---


Here is my example node.js server:

#!/usr/bin/env node

var sys = require('sys'),
http = require('http'),
optparse = require('optparse');;

var SWITCHES = [
['-p', '--port-number [PORT_NUMBER]', "Portnumber to listen to"],
['-H', '--help', "Shows this help section"],
];

// Internal variable to store options.
var options = {
port: 8000
};

// Create a new OptionParser with defined switches
var parser = new optparse.OptionParser(SWITCHES);
parser.banner = 'Usage: example.js [options]';

// Handle the --print switch
parser.on('port-number', function(value) {
options.port = parseInt(value);
});

// Handle the --help switch
parser.on('help', function() {
sys.puts(parser.toString());
});

// Parse command line arguments
parser.parse(process.ARGV);

http.createServer(function (req, res) {
for (var i in req.headers) {
sys.puts(i + ": " + req.headers[i]);
}

res.sendHeader(200, {'Content-Type': 'text/plain'});
res.sendBody('Hello');
res.flush();

setTimeout(function () {
res.sendBody(' World = ' + options.port);
res.finish();
}, 2000);
}).listen(options.port);

sys.puts('Server running at http://127.0.0.1:' + options.port + '/');



Any ideas?

[eugene]

One more thing. The problem only occurs when I request directly to the
node.js server with http 1.1. It works fine with 1.0.

Here's the trace again (now printing out the http version):

SERVER:

[eugene@eugenes-macbook ~/Documents/work/nodedev/example] master$ ./
example.js

Server running at http://127.0.0.1:8000/

http version: 1.1

user-agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4
OpenSSL/0.9.8k zlib/1.2.3
host: localhost:8000
accept: */*

CLIENT:

[eugene@eugenes-macbook /usr/local/lib/node/libraries]$ curl -i
http://localhost:8000
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: keep-alive

Hello World = 8000


It hangs for about 30 seconds before returning (guess cause it's
missing a content-length?) Worked before. It's almost like it's not
closing the connection on the server side because of the keep-alive?
But it worked fine before the patch. Hmm... very strange.

Thanks for your help!

On Dec 5, 2:46 pm, eugene <eug...@noblesamurai.com> wrote:
> OK.
>
> When I make the change in http.js this is what I get when I hit nginx.
> Config is nginx is listening on 80 and reverse proxying to a node.js
> server on port 8000.
>
> So when I hit nginx:
>

> [eugene@eugenes-macbook /usr/local/lib/node/libraries]$ curl -ihttp://localhost

> HTTP/1.1 200 OK
> Server: nginx/0.7.64
> Date: Sat, 05 Dec 2009 03:41:42 GMT
> Content-Type: text/plain
> Transfer-Encoding: chunked
> Connection: keep-alive
>
> Hello World = 8000
>
> ---
>
> Here are the headers received by the node.js server:
>
> [eugene@eugenes-macbook ~/Documents/work/nodedev/example] master$ ./
> example.js -p 8000

> Server running athttp://127.0.0.1:8000/

> user-agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4
> OpenSSL/0.9.8k zlib/1.2.3
> host: localhost:8000
> accept: */*
> host: 127.0.0.1:8000
> connection: close
> user-agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4
> OpenSSL/0.9.8k zlib/1.2.3
> accept: */*
>
> ----
>
> no problems there. When I now go straight to the node.js server
> listening on port 8000, however:
>

> [eugene@eugenes-macbook /usr/local/lib/node/libraries]$ curl -ihttp://localhost:8000

> HTTP/1.1 200 OK
> Content-Type: text/plain
> Connection: keep-alive
>
> --- (it just hangs - actually it waits for a few minutes and then
> returns) ---
>
> The headers received by the node.js server:
>
> [eugene@eugenes-macbook ~/Documents/work/nodedev/example] master$ ./
> example.js -p 8000

> Server running athttp://127.0.0.1:8000/

> sys.puts('Server running athttp://127.0.0.1:'+ options.port + '/');

[ry]

On Dec 3, 1:11 pm, Christopher Lenz <cml...@gmail.com> wrote:
> On 03.12.2009, at 12:40, Hagen wrote:
>
> > chunked is a known problem with all mainstream reverse proxies. fixing  
> > them would be best, giving node the capability to disable chunking the  
> > most pragmatic option. this option must be transparent to code running  
> > ontop of node.
>
> Actually I think this is a bug in the node.js HTTP server. Assuming those proxies specify the HTTP version as 1.0 in their requests, node should not be doing chunked encoding, but rather stream out the body chunks normally and then close the connection.

Yes, it's a bug. Fixed in c5d8238, hopefully.

[eugene]

Thanks heaps. This fixes my nginx AND going direct to node.js problem.
Fixed forever! :-)

-- Eugene

[Ben Noordhuis]

On Tue, Oct 30, 2012 at 6:09 PM, Nishad Virkud <nishad...@gmail.com> wrote:
> Hi,
>
> I am also having
> Transfer-Encoding:
> chunked in Response headers.
> Please help me out to solve this.
> I am using Apache-2.2.19, with Jboss-4.0.2 and Centos 5.5 64 Bit Operating
> System
>
> Please find below Response headers:
>
> Connection:
> Keep-Alive
> Content-Encoding:
> gzip
> Content-Language:
> en
> Content-Type:
> text/html;charset=UTF-8
> Date:
> Tue, 30 Oct 2012 17:13:44 GMT
> Keep-Alive:
> timeout=5, max=98
> Pragma:
> no-cache
> Server:
> Apache
> Set-Cookie:
> com.anm.octashop.lang=en, com.anm.octashop.orgId=0
> Transfer-Encoding:
> chunked
> Vary:
> Accept-Encoding
> X-Powered-By:
> Servlet 2.4; JBoss-4.0.2 (build: CVSTag=JBoss_4_0_2
> date=200505022023)/Tomcat-5.5

Is this related to node.js or are you simply spamming Usenet in the
hope that something pops up?

If it's option #1, note that you're a) reviving a topic that's been
dead for nearly three years, and b) your post is scarce on details.

If it's option #2, please fuck off and never come back.

[Gaitho Bernard]

Hi Eugene, I have a problem with the response from my Nginx which is hosting a client side server. My backend is NodeJs. Randomly i keep getting error 500 which i believe is associated with the transfer encoding as chunked. Can you help me?