node.js program crash due to "Can't render headers ..." exception

[ming]

Hi,
i'm running a node application which crashes probably once a week due to an uncaught "Can't render headers ..." exception.   The node.js version in use is 0.8.22 and the OS distro is RHEL 6.2.  i did not find the definitive pattern of sequence of events which led to the crash though.

The stack trace looks like:

     http.js:715
     throw new Error('Can\'t render headers after they are sent to the client.'
     ^
     Error: Can't render headers after they are sent to the client.
     at ServerResponse.OutgoingMessage._renderHeaders (http.js:715:11)
     at ServerResponse.writeHead (http.js:1059:20)
     at Socket.proxyError (/opt/share/node_modules/http-proxy/lib/node-http-proxy/http-proxy.js:193:9)
     at Socket.g (events.js:192:14)
     at Socket.EventEmitter.emit (events.js:126:20)
     at Socket.connect.require.lookup.addressType (net.js:699:16)
     at process.startup.processNextTick.process._tickCallback (node.js:245:9)

According to my web surfing, several developers also ran into similar problem in the past but it is not clear to me what the culprit is.  i know the node.js version i used (0.8.22) has some age but it seems that similar problems also exist in 0.10.10 so i'm not sure upgrading node.js helps in this case.  Moreover, to upgrade node.js in our environment is somewhat painful and not really under my control.

Any suggestion how i can track down and rectify the problem?   

Thanks.

[Bruno Fuster]

It's probably because some callback is being called twice and you are trying to send a response twice.

That problem is not easy to find, though. 

Increase your log level and make sure that you have proper logging and request/response logging. When that error occurs, check which requests came before and you should be able to check where the problem is coming from.

It might be some silly error like this:

if (err) {

  callback(err);

}

callback(null,result);

Which causes the callback to be called twice.

Good luck

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bruno Fuster]

Ps: that error is not a node.js version issue

Don't even bother trying to upgrade or downgrade

On Dec 9, 2013, at 9:01 PM, ming <hseu...@gmail.com> wrote:

[Bruno Fuster]

Ps: use process.on uncaughtException and the server will not crash

Be careful with that :) 

On Dec 9, 2013, at 9:01 PM, ming <hseu...@gmail.com> wrote:

[Hseu-Ming Chen]

Hi Bruno,
Thank you for your input.

> if (err) { callback(err); }
> callback(null,result);
> Which causes the callback to be called twice.

i searched for similar pattern in my code without finding any

> Increase your log level and make sure that you have proper logging
> and request/response logging.

Yes, i'll ask Winston to be more chatty.

> Ps: that error is not a node.js version issue

agreed

My node.js program is basically a reverse proxy and at the same time a [private|local] CA that authenticates client-side X.509 certs.
i'm using http-proxy but i don't think i can conveniently blame the underlying "battle-hardened" http-proxy either.

> Ps: use process.on uncaughtException and the server will not crash
> Be careful with that :)

Instead of
   process.on('uncaughtException')
are you thinking of something like "domain"?   i need to read up a bit on it since
* it is still "unstable"
* unsure whether i can use it in node 0.8.22

Thanks again.

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/RK5jfu5cOhY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[Bruno Fuster]

I have no experience with domains given that's still unstable. 

If you use uncaughtException it will fix your problem temporarily: the server will not shutdown, this error will not cause any problems for the users, but you still have to find it and fix it because something is wrong.

Can you also check that you are not calling res.send() or res.json() or res.end() twice?

After you find which endpoint is causing the issue by enabling logs or creating some of them you should be able to track it down.

Cheers

Bruno Fuster

[ming]

Hi Bruno,
The skeleton of my code (again, my node application is a reverse web proxy) is like:
 
   require('spdy').createServer
   (
      ...
      var proxy = new httpProxy.HttpProxy({target: {host: ..., port: ...}});   //  [1]
      proxy.on('proxyError', function(err,req,res) ...);
      proxy.proxyRequest(req,res);    //  [2]
   );

It turns out that there is a "proxyError" event emitted in the HttpProxy constructor call below thanks to a DNS failure for the host of the target service/application:
   var proxy = new httpProxy.HttpProxy({target: {host: ..., port: ...}});   //  [1]

i believe my node server crashed thanks to the default error handler so i just added the listener for "proxyError".

That means the following statement should be skipped completely:
   proxy.proxyRequest(req,res);    //  [2]
if the constructor call (HttpProxy) at [1] fails. 

i'll sleep on a good way for error handling in this case.

Thanks again.