am i seeing ghosts: "Can't render headers after they are sent to the client" error with Node.js 0.10.24?

[ming]

Hi,
i've been running a reverse proxy (with the http-proxy module) on Node.js 0.10.24 for a while after i migrated from 0.8.22.    To my surprise, i again ran into the annoying
   Error: Can't render headers after they are sent to the client.
so my Node.js server crashed (complete stack trace below in <1>):

According to ChangeLog of 0.9.3 at
   http://nodejs.org/changelog.html
there is the following:
   http: add response.headersSent property (Pavel Lang)

Should this error be squashed after 0.9.3 release with the headersSent check?   What am i missing?

<1>
---------------------------------------------------------------
  Error: Can't render headers after they are sent to the client.
    at ServerResponse.OutgoingMessage._renderHeaders (http.js:733:11)
    at ServerResponse.writeHead (http.js:1150:20)
    at ClientRequest.proxyError (/.../node_modules/http-proxy/lib/node-http-proxy/http-proxy.js:213:9)
    at ClientRequest.g (events.js:180:16)
    at ClientRequest.EventEmitter.emit (events.js:95:17)
    at Socket.socketErrorListener (http.js:1547:9)
    at Socket.EventEmitter.emit (events.js:95:17)
    at net.js:441:14
    at process._tickDomainCallback (node.js:459:13)
---------------------------------------------------------------

The pertinent code snippet is as follows:
---------------------------------------------------------------
  var proxyDomain = domain.create();
  proxyDomain.on('error', ...);
  proxyDomain.on('proxyError', ...);

  spdy.createServer
  (
    ...
    proxyDomain.run
    (
      function()
      {
        var proxy = new httpProxy.HttpProxy({target: {host: ..., port: ...}});
        ...
        proxy.proxyRequest(req,res);
      }
    );
  );
---------------------------------------------------------------


Anything blatantly wrong? 

Thanks for reading.

[Pedro Narciso García Revington]

You are probably triggering response.end() more than once.

http://nodejs.org/api/http.html#http_class_http_serverresponse

[Hseu-Ming Chen]

Hi Pedro,
Thank you for your input.

> You are probably triggering response.end() more than once.

Yes, that would be my guess as well.   However, in the code snippet below:

   var proxy = new httpProxy.HttpProxy({target: {host: ..., port: ...}});
   ...
   proxy.proxyRequest(req,res);

i simply proxy the HTTP request on via  "proxy.proxyRequest(req,res);" so i don't explicitly invoke "response.end()".   That's where i'm stuck.  In general, what is a surefire way to track down similar problems deterministically?

A little background info here: my Node.js server is a reverse proxy (with the http-proxy module) as stated earlier and has been running fine for more than one year.  The recent crashes happened when someone carried out some [stress|load] test against my server.  The server crashed in the middle of the load test.  

What i also noticed is when my Node.js server crashed, the response from the target server is 405 (not sure that's true in all crashes since the crash happened so rarely).   Since the crash rarely happens (only in the load test), i can't conveniently resort to [Node.js | Google V8] debuggers.

Thanks.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/FWcUXFAKqiY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pedro Narciso García Revington]

Server response is a stream. You maybe have better look by listening the finish event

[greelgorke]

i don'T know for sure, but from the stacktrace: could it be possible, that the proxy module write some headers already, and then your code in the proxyError event handler again?

[David Stodolsky]

I have been seeing a related error. 

Here is the terminal output:

Last login: Tue Dec 24 19:20:19 on console

MacPro-2:~ davidold$ cd /Users/davidold/Downloads/MyTurn-master 

MacPro-2:MyTurn-master davidold$ node nodejs_server.js

Express server listening on port 3000 in development mode

   info  - socket.io started

http.js:704

    throw new Error('Can\'t set headers after they are sent.');

          ^

Error: Can't set headers after they are sent.

    at ServerResponse.OutgoingMessage.setHeader (http.js:704:11)

    at ServerResponse.res.setHeader (/Users/davidold/Downloads/MyTurn-master/node_modules/express/node_modules/connect/lib/patch.js:62:20)

    at /Users/davidold/Downloads/MyTurn-master/node_modules/express/node_modules/connect/lib/middleware/static.js:168:11

    at Object.oncomplete (fs.js:107:15)

MacPro-2:MyTurn-master davidold$ 

Console:

24-12-13 19:24:24 [0x0-0x1a01a].com.google.Chrome[347] [353:263:1224/192424:ERROR:base_feature_provider.cc(122)] manifestTypes: Allowing web_page contexts requires supplying a value for matches.

24-12-13 19:24:24 [0x0-0x1a01a].com.google.Chrome[347] [351:263:1224/192424:ERROR:base_feature_provider.cc(122)] manifestTypes: Allowing web_page contexts requires supplying a value for matches.

24-12-13 19:24:25 [0x0-0x1a01a].com.google.Chrome[347] [347:26883:1224/192425:ERROR:base_feature_provider.cc(122)] manifestTypes: Allowing web_page contexts requires supplying a value for matches.

24-12-13 19:24:35 [0x0-0x1a01a].com.google.Chrome[347] [357:263:1224/192435:ERROR:base_feature_provider.cc(122)] manifestTypes: Allowing web_page contexts requires supplying a value for matches.

24-12-13 19:26:27 [0x0-0x1a01a].com.google.Chrome[347] [361:263:1224/192627:ERROR:base_feature_provider.cc(122)] manifestTypes: Allowing web_page contexts requires supplying a value for matches.

dss

[Hseu-Ming Chen]

Hi greelgorke,

Thank you for your input.

i think you're right on the money.  However, i need to re-run the load test (which may take a while) to see if that is really the case.

Two questions:

>  ...  that the proxy module write some headers already, and then your

>  code in the proxyError event handler again?

Where did you get the clue about "then your code in the proxyError event handler"?    Did you get from the following line in the stack trace:

   at process._tickDomainCallback (node.js:459:13)

After i read up on:
   https://github.com/nodejitsu/node-http-proxy/issues/264
   https://github.com/nodejitsu/node-http-proxy/blob/master/examples/http/custom-proxy-error.js
currently the way i handle the proxyError event is as follows:

   proxyDomain.on
   (
     'proxyError',
     function(err,req,res)
     {
       try
       {
         res.writeHead(500, {'Content-Type': 'text/plain'});
         res.end('   !!!!!   something went really wrong');
       }
       catch(err)
       {
         console.log(err.stack);
         process.exit(17);
       }
     }
   );

Anything blatantly wrong?   It appears that the line
   res.writeHead(500, {'Content-Type': 'text/plain'});
is the culprit for

   Error: Can't render headers after they are sent to the client.

Thanks again.

[Hseu-Ming Chen]

Hi David,

My personal experience for such errors

    Error: Can't set headers after they are sent.

is oftentimes they are manifests of problems somewhere else, instead of the origin of the problem.  

For example, i once ran into these errors due to DNS resolution failure.    It usually takes some time to pinpoint the real culprit.

[greelgorke]

inline

Am Dienstag, 11. März 2014 03:44:16 UTC+1 schrieb ming:

Hi greelgorke,
Thank you for your input.

i think you're right on the money.  However, i need to re-run the load test (which may take a while) to see if that is really the case.

Two questions:

>  ...  that the proxy module write some headers already, and then your
>  code in the proxyError event handler again?

Where did you get the clue about "then your code in the proxyError event handler"?    Did you get from the following line in the stack trace:
   at process._tickDomainCallback (node.js:459:13)

indeed from the stacktrace and the fact that your code seems to handle errors (but does not show the handling code itself)

After i read up on:
   https://github.com/nodejitsu/node-http-proxy/issues/264
   https://github.com/nodejitsu/node-http-proxy/blob/master/examples/http/custom-proxy-error.js

as the comments in the issue say, the proxy already writes headers. what you have to do in your error handler, is to check if the headers are already written and only write yourself headers, if they're not.  http://nodejs.org/api/http.html#http_response_headerssent

another way seems to be to add an 'error' handler on your proxy yourself (not on the domain object) and set the headers yourself: https://github.com/nodejitsu/node-http-proxy#listening-for-proxy-events . the difference is, that the domain catches errors and handles them, but the proxy module has it's own logic for handling errors and expects a custom error handler for that.