Socket hang up problems

[easternbloc]

Hi,

I've been getting this issue every now and then on a long poll
request.
It doesn't happen consistently so I'm finding it very hard to track
down.
Has anyone else had this error?

node.js:116
throw e; // process.nextTick error, or 'error' event on first
tick
^
Error: socket hang up
at Client.<anonymous> (http.js:1440:26)
at Client.emit (events.js:42:17)
at Array.<anonymous> (net.js:800:12)
at EventEmitter._tickCallback (node.js:108:26)

Any help would be greatly appreciated.

[Laurent Eschenauer]

Error: socket hang up

I had the same issues lately and switching to the new http API in 0.4 solved it all. Instead of using http.createClient(), I now use http.request().

http://nodejs.org/docs/v0.4.1/api/http.html#http.request

No clue however on what changed and why using createclient produces these socket hang up.

Hope it helps,

-Laurent

http://github.com/eschnou

 

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Tim Caswell]

FWIW I also found the compat code unusable and wrote my own that wrapped the new APIs.  Here is my quick and dirty monkey patch to allow apps to continue to use the old API without breaking.

var http = require('http');

var https = require('https');

var EventEmitter = require('events').EventEmitter;

http.createClient = function (port, host, secure) {

var module = secure ? https : http;

var client = new EventEmitter();

var options = {

port: port,

host: host

};

client.request = function (method, path, headers) {

var request = new EventEmitter();

options.method = method;

options.path = path;

options.headers = headers

var request = module.request(options, function (response) {});

return request;

};

return client;

};

[easternbloc]

Cheers guys,

I'll migrate to the new API calls and see if I get more stability.

Thanks, Ben

On Mar 2, 6:48 pm, Tim Caswell <t...@creationix.com> wrote:
> FWIW I also found the compat code unusable and wrote my own that wrapped the
> new APIs.  Here is my quick and dirty monkey patch to allow apps to continue
> to use the old API without breaking.
>
> var http = require('http');
> var https = require('https');
> var EventEmitter = require('events').EventEmitter;
> http.createClient = function (port, host, secure) {
> var module = secure ? https : http;
> var client = new EventEmitter();
> var options = {
> port: port,
> host: host};
>
> client.request = function (method, path, headers) {
> var request = new EventEmitter();
> options.method = method;
> options.path = path;
> options.headers = headers
> var request = module.request(options, function (response) {});
> return request;
>
> };
> return client;
> };
>
> On Wed, Mar 2, 2011 at 12:53 AM, Laurent Eschenauer

> <laur...@eschenauer.be>wrote:

[Randy Fay]

I'm seeing the same issue and baffled by it. Help would be appreciated also.

It looks to me like the http.request() creation times out, and this is a bogus error message. However, I haven't been able to sort out why it would be timing out, or what might be wrong with my request ceation.

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

--
Randy Fay
Drupal Module and Site Development
ra...@randyfay.com
+1  970.462.7450

[Randy Fay]

The problem in my case was that I hadn't fully completed the metaphor change to http.request().

I had failed to issue the req.end() after creating the http.request(). Adding the req.end() fixed it for me.

The doc is quite clear about this, but I had lost it when translating old code to new.

-Randy

[felixcriv]

I'm facing the same problem, already know how to solve this?.

Thanks.

[Łukasz Mielicki]

I'm also facing this issue and I'm wondering if it isn't connected to recent update of some system libraries in Debian, since I cannot get rid of this by downgrading node...

[Matt Ranney]

This is causing a lot of problems for me too.  I seems to be from the HTTP client, with or without an agent.

What I notice in my server is that this doesn't seem to happen until the server has been running for several hours, or it could be related to the number of connections it has made.

2011/3/7 Łukasz Mielicki <bo...@wp.pl>

I'm also facing this issue and I'm wondering if it isn't connected to recent update of some system libraries in Debian, since I cannot get rid of this by downgrading node...

[Mikeal Rogers]

To clarify, there is *always* an agent. Even setting {agent:false} means that you node will just create a new Agent instance for every request, so you opt out of pooling but the Agent logic and code is still in play.

[Ryan Dahl]

It's a bug - but we need to get a reproducible test case.

[Ryan Dahl]

On Wed, Mar 2, 2011 at 10:48 AM, Tim Caswell <t...@creationix.com> wrote:
> FWIW I also found the compat code unusable and wrote my own that wrapped the
> new APIs.  Here is my quick and dirty monkey patch to allow apps to continue
> to use the old API without breaking.
> var http = require('http');
> var https = require('https');
> var EventEmitter = require('events').EventEmitter;
> http.createClient = function (port, host, secure) {
> var module = secure ? https : http;
> var client = new EventEmitter();
> var options = {
> port: port,
> host: host
> };
> client.request = function (method, path, headers) {
> var request = new EventEmitter();
> options.method = method;
> options.path = path;
> options.headers = headers
> var request = module.request(options, function (response) {});
> return request;
> };
> return client;
> };
>

Unusable in what way?

[Matt Ranney]

On Tue, Mar 8, 2011 at 1:14 PM, Ryan Dahl <r...@tinyclouds.org> wrote:

It's a bug - but we need to get a reproducible test case.

I've almost got one.  I need to whittle it down to just the necessary parts, but it's pretty close.

[Mikeal Rogers]

I'm just kicking this thread.

Some patches landed, i'm wondering if mjr is still seeing these errors in production.

[naneau]

If you also control the server that handles the request, you might try
adding connection: keep-alive. We faced the same issue when connecting
to an ExpressJS server, that did not add keep-alive by default (as it
should, by the HTTP 1.1 spec). There's also a bug in node.js's
connection handling that does not gracefully fail when this happen.

https://github.com/joyent/node/blob/v0.4.2/lib/http.js#L1160 is where
it goes wrong, offending line being 1174. See also this issue:
https://github.com/joyent/node/issuesearch?state=open&q=emit#issue/784

[Matt Ranney]

On Tue, Mar 15, 2011 at 2:12 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:

Some patches landed, i'm wondering if mjr is still seeing these errors in production.

I think we are going to update production tomorrow, so I can report back then.

It is interesting that we mostly see this when talking to CouchDB.  We do sometimes see this when talking to other node HTTP servers, but much less so.  Of course,they are doing very different things and thus are used in very different ways.  However, I have the maxSockets set to 4,000, and I'm nowhere near that, which would sort of seem to rule out the recent fix around maxSockets.

But of course I'll try the new version and report back.  :)

[naneau]

I am finally able to reproduce this error reliably:

Server: https://gist.github.com/874392
Client: https://gist.github.com/874398

Steps to reproduce

1. Start the server: coffee server.coffee
2. Start the client: coffee client.coffee
3. Kill the server, see the client's ECONNREFUSED to indicate it tried
to make a request but failed
4. Start the server again. The client will now get error messages like
"Error making ping request to 127.0.0.1:10000: Error: Parse Error" and
"Error making ping request to 127.0.0.1:10000: Error: socket hang up"


On Mar 16, 10:54 pm, Matt Ranney <m...@ranney.com> wrote:

[naneau]

Compiled client/server to js for convenience:

Server: https://gist.github.com/874462
Client: https://gist.github.com/874467

[Matt Ranney]

On Thu, Mar 17, 2011 at 5:00 AM, naneau <mau...@angrybytes.com> wrote:

Compiled client/server to js for convenience:

Server: https://gist.github.com/874462
Client: https://gist.github.com/874467

This looks great.  I can confirm that this client reproduces the problem.  There's nothing special about the server.  I get the same results from this client with starting/stopping CouchDB, apache httpd, or even with ssh port redirection to a remote server.

 

[Stéphan Kochen]

On Mar 17, 8:36 pm, Matt Ranney <m...@ranney.com> wrote:
> This looks great.  I can confirm that this client reproduces the problem.
>  There's nothing special about the server.  I get the same results from this
> client with starting/stopping CouchDB, apache httpd, or even with ssh port
> redirection to a remote server.

Good to hear others can reproduce this as well.

It appears to be a problem in the reuse of parsers in http.js. Replace
the FreeList with a dummy implementation, and the problem disappears.
For example, I've added the following to http.js:

//var FreeList = require('freelist').FreeList;
var FreeList = function(name, max, alloc) {
this.alloc = alloc;
this.free = function() {};
};

Maurice's server may seem ordinary, but I've encountered problems on
node HTTP servers as well, so it's not exclusive to the client
implementation. While building a stress test for our app, I find when
rapidly creating connections, the server gets 'request' events for
different HTTP connections that reference the same file descriptor.
Wireshark traces clearly show two separate connections, with node
accidentally sending data to the wrong socket. This problem also
disappears when I remove the FreeList.

--
Stéphan

[Stéphan Kochen]

We created a pull request with a patch that solved the problem for us.
Find it at:
https://github.com/joyent/node/pull/803

--
Stéphan Kochen

[Matt Ranney]

2011/3/18 Stéphan Kochen <ste...@angrybytes.com>

We created a pull request with a patch that solved the problem for us.
Find it at:
https://github.com/joyent/node/pull/803

FYI to anybody following along, this has been fixed here:

https://github.com/joyent/node/commit/66570c196429f01786c79f177c643b07f678c32c

We've put this change into production.  I'll report back if this does not fix it.

[Smog]

We still see this problem even with 0.4.7, which has the patch.
Very simple use case, http.request(), req.write(), req.end().

[Renault]

Same here. I have the socket hang up error and I'm running 0.4.8. Any
progress on this bug?

[murvinlai]

I also have socket hang up problem.

What i do is that, my program use http.createClient to connect to
another server's REST API. when I do load testing, I believe, some
client although successfully return the response data, it is still
hanging there until socket times out.

I don't know what to do now...

On May 24, 4:57 am, Renault <renault.j...@gmail.com> wrote:
> Same here. I have thesockethang up error and I'm running 0.4.8. Any

[murvinlai]

I am using 0.4.8 and still see the hang up problem.

What i did: 100-500 concurrent threads sent from my node.js server to
another client server using http.request with agent.maxSockets=1024
and Connection:keep-alive. After running for 50+ iterations, then I
will start seeing socket hang up error, especially when all my batches
have done.
The error is (sorry.. json stringified) :

{"err":"{\"stack\":\"Error: socket hang up\\n at Socket.<anonymous>
(http.js:1285:45)\\n at Socket.emit (events.js:64:17)\\n at
Array.0 (net.js:831:12)\\n at EventEmitter._tickCallback (node.js:
126:26)\",\"message\":\"socket hang up\"}"}


It has something to do with the socket closed.

if (socket._httpMessage) {
if (socket._httpMessage.res) {
socket._httpMessage.res.emit('aborted');
socket._httpMessage.res.emit('close');
} else {
if (!socket._httpMessage._hadError) {
socket._httpMessage.emit('error', new Error('socket hang
up'));
}
}
}


I'm wondering if it is a bug, or what. Thanks.

[murvinlai]

I kinda solve the Socket Hang up problem. At least ,now I don't see
any error message.

Here is what I've done:

1) for http.request options, set Connection:keep-alive
2) set Agent.maxSockets = 1024 (so more connection to play around
with )
3) very critical: DO a timeout for the http.request.

e.g.
var responseHdr = function (clientResponse) {
if (clientResposne) {

} else {
clientRequest.abort();
}
};

var timeoutHdr = setTimeout(function() {
clientRequest.emit('req-timeout');
}, 5000); // timeout after 5 secs

//cfg.http_client_response_timeout
clientRequest.on("req-timeout", responseHdr);


clientRequest.on('error', function(e) {
clearTimeout(timeoutHdr);
console.error('Ok.. clientrequest error' + myCounter);
next({err:JSON.stringify(e)});
});

4) do the clientRequest.abort(); when timeout (see code above).

What I found out is that, if you let the timeout socket sitting there,
it will issue timeout hang up.
Even you have your own timeout routine (as above) to handle timeout
issue, the socket associated with the clientRequest is actually still
alive. So, in order to do that, you have to do
clientRequest.abort() when you manually handling timeout (as above).

5) Better to set the timeout be some reasonable but NOT large number.
Because, what I found out is that, even set maxSockets to 1024, if ONE
socket is stucked, your node.js will stall and cannot connect to any
new socket until that one is timeout or released. (but if you have
two node.js instance, the other one will not be affected). So,
better not set it to more than 10 secs.. my experience tells me. :)

On May 26, 7:52 pm, murvinlai <murvin...@gmail.com> wrote:

[murvinlai]

Still having problem.

I left my program running (but idle) during the weekend. Then, when I
come back, I got the socket hang up error, and can't execute.

It complains on http.js: 1285:45 at Socket.emit (events.js: 64:17).



On Jun 9, 4:36 pm, murvinlai <murvin...@gmail.com> wrote:
> I kinda solve the SocketHangupproblem.  At least ,now I don't see

> > > Same here. I have thesockethanguperror and I'm running 0.4.8. Any

[Luca922]

Any news on this issue?

The socket hang up seems to happen on 0.5pre as well, usually in a
situation where a "no route to host" occurs.

The problem can be reproduced by us as we have one EC2 box constantly
doing http.requests to our own server. This worked perfectly fine
until we replaced our server with a more powerful box in a different
data center. For some reason, our monitoring reports that - only on
two or three requests a day - the EC2 cannot find the route to our new
server. No problems like that with the old server. Anyway, this is
when Node gets stuck.

Luca

[murvinlai]

I haven't heard any news yet. Here is my post in github:
https://gist.github.com/1032413
however, I get one of the core developers to take a look into this
problem. :)

What is "no route to host"?

Please see my code in the link I provide. I have a very controlled
testing. I think when doing remote call, no matter how many socket
you set (e.g. maxSockets= 1000), socket will break / hang up because
of the client's side response being timeout by whatever reason. Then
once that socket has problem, it will never be recovered. If you use
up all the good sockets, then your program cannot make any
http.request and you can declare your node die.

There is a get around solution I'm using before they come up with the
correct solution.
1) use Agent, and set Agent.maxSockets = big number that matches with
your ulimit. And http.options.headers.Connection:'keep-alive'
2) Make your own timeout route. If it times out, make sure to call
http.clientRequest.abort(); But, your socket still die, just that it
won't timeout and the socket timeout again.
3) When timeout or on.('error') with error message is "socket hang up"
more than certain number of times, (e.g. 100), then kill and restart
node. I don't know if just restart http.server will work, but I'm
sure restarting node will work.
4) To restart node inside code, I use pgte/fugue . https://github.com/pgte/fugue
I have done some testings and it looks good.

This workaround seems to be promising. But, if you use it, please
let me know if you find out anything. :) Thanks.

On Jun 18, 11:55 pm, Luca922 <arne%steingraeber....@gtempaccount.com>
wrote:

> Any news on this issue?
>

> The sockethangupseems to happen on 0.5pre as well, usually in a

[Kiran]

Has anybody else solved this problem. I am having a similar issue.
When I am using Apache Bench to test my node server connecting to a
remote server, i occasionally get a hang of the entire Node process
for 20 seconds. THe number 20 is key because it is the default socket
timeout in Node so somehow having a bad socket is causing a real
problem with nodes ability to be fully async and performant. Is there
a way to reduce the timeout and/or is there a way to get this issue to
be resolved or at least a patch to the native code that shows were
this might be a problem.

Kiran

On Jun 20, 2:27 pm, murvinlai <murvin...@gmail.com> wrote:
> I haven't heard any news yet.  Here is my post in github:https://gist.github.com/1032413
> however, I get one of the core developers to take a look into this
> problem. :)
>
> What is "no route to host"?
>
> Please see my code in the link I provide.   I have a very controlled
> testing.   I think when doing remote call, no matter how many socket
> you set (e.g. maxSockets= 1000), socket will break /hangupbecause
> of the client's side response being timeout by whatever reason.  Then

> once that socket has problem, it will never be recovered.   If you useupall the good sockets, then your program cannot make any

> http.request and you can declare your node die.
>

> There is a get around solution I'm using before they comeupwith the

> correct solution.
> 1)  use Agent, and set Agent.maxSockets = big number that matches with
> your ulimit.   And http.options.headers.Connection:'keep-alive'
> 2) Make your own timeout route.  If it times out, make sure to call
> http.clientRequest.abort();   But, your socket still die, just that it
> won't timeout and the socket timeout again.
> 3) When timeout or on.('error') with error message is "sockethangup"
> more than certain number of times, (e.g. 100), then kill and restart
> node.  I don't know if just restart http.server will work, but I'm
> sure restarting node will work.

> 4) To restart node inside code, I use   pgte/fugue  .https://github.com/pgte/fugue

[Murvin Lai]

You can try  (temporary ) edit the node.js library.   lib\http.js   

search for socket.setTimeout    the default timeout should be 2 * 60 * 1000 = 2 minutes , instead of 20 seconds.  You can change that value.

however, if your remote client has a slow response, then you will get a lot of timeout.  

Also, a warning is that, if you let your process run for long time, and the socket opened (for connection pool) is over your maxSockets, you may not be able to establish any new http.request.   

before a formal fix, I suggest the followings:

1) increase your maxSocket (and also your OS one.  ulimit -a   and use ulimit -n   to increase it , or change the /etc/security/limits.conf  Incraese soft and hard.  (where hard > soft). e.g   * soft  nofile  10000     * hard nofile 12000

2) catch http.request error.  i.e.:
var clientRequest = http.request (... );
clientRequest.on ('error', function() {  clientRequest.abort() } );

Although socket timeout will kill and free up the socket, but I find out doing that will save you more socket.

3) lower your socket timeout. change the http.js  and then "Make" again. 

[Arindam]

I am using node 0.6.10 version and I got the error today while uploading a file to amazon S3.

I am using knox(@0.0.9) module with request(@2.9.153).

[2012-03-06 22:33:34.251] [ERROR] choicebeat - Error uploading to S3: Error: soc

ket hang up

    at createHangUpError (http.js:1104:15)

    at Socket.<anonymous> (http.js:1207:27)

    at Socket.emit (events.js:88:20)

    at Array.0 (net.js:320:10)

    at EventEmitter._tickCallback (node.js:192:40)

Looks like the issue was fixed. Am I using any older version ?

To unsubscribe from this group, send email to nodejs+unsubscribe@googlegroups.com.

[Arindam]

Also, I checked that I have http.js in the following modules and none of them have setTimeout (did a grep).

./express/lib/http.js

./express/node_modules/connect/lib/http.js

./socket.io/lib/transports/http.js

[Mark Essel]

I've seen socket hangups putting and getting to and from s3 with Knox as well. Haven't had much luck recreating the problem in small test scripts -ie 1000 100mbyte file gets are all flawless (sha1'd)

Anyone else bump into this? It's making processing fail in tough to recover from ways.

[Mark Essel]

heads up we patched up our communication with s3

https://github.com/fastsociety/node-knoxed-up

md5sum puts, sha1 compares on gets

and retries on hangups

[Oleg Slobodskoi]

I have this error reproducible very stable using node v0.8.21

And I think I have nailed the issue: it happens if the maxSockets of the agent is lower than the amount of requests we are doing.

If I set https.globalAgent.maxSockets = 50; and do 50 parallel requests - after some seconds the error will be there.

If I do 40 parallel requests - I am able to download thousands of files from S3.

Possible solutions:

1. I think first of all it is a documentation issue on node as well as knox. Both of them should mention, that the default Agent has maxSockets == 5. Node should mention this not only in place where maxSocket option is described but also in 3-4 other places where the users read how to create requests.

2. Knox could set for its engine the maxSockets value to something much more higher than 5, f.e. 500, because knox will be often used with multiple connections per host. Also knox could expose and document maxSockets option which is then forwarded to the Agent.

On Tuesday, March 1, 2011 2:31:07 PM UTC+1, easternbloc wrote:

Hi,

I've been getting this issue every now and then on a long poll
request.
It doesn't happen consistently so I'm finding it very hard to track
down.
Has anyone else had this error?

node.js:116
        throw e; // process.nextTick error, or 'error' event on first
tick
        ^
Error: socket hang up
    at Client.<anonymous> (http.js:1440:26)
    at Client.emit (events.js:42:17)
    at Array.<anonymous> (net.js:800:12)
    at EventEmitter._tickCallback (node.js:108:26)

Any help would be greatly appreciated.

[Ryan Schmidt]

The message you replied to was two years old...

Have you tried with node 0.10 yet?

[Oleg Slobodskoi]

the issue persists on node 0.8 ...

https://github.com/LearnBoost/knox/issues/116

haven't tried 0.10 yet.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines

You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com

To unsubscribe from this group, send email to

nodejs+un...@googlegroups.com

For more options, visit this group at

http://groups.google.com/group/nodejs?hl=en?hl=en

---

You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.