Doing 1000 concurrent http requests, and getting ETIMEDOUT.

[Dan]

Hey,

I'm coding a proxy server checker.

In order to check a proxy, I need to connect to it using "request" module, and perform several requests.

The goal is to be able to check thousands of proxies in short amount of time.

I was hoping to test 1,000 proxies concurrently. I did the following configs:

http.globalAgent.maxSockets = 100000;

events.EventEmitter.prototype._maxListeners = 100000;

and I execute 1,000 requests using async.queue.

I also set the open file descriptors to a really high limit... Another thing I do is I set a timeout for each proxy for 30 secs...

Anyway, what happens is that the first requests are performed well and then I start getting a lot of ETIMEDOUT & ESOCKETTIMEDOUT.

I checked the amout of open files for my node process:

pidof node

lsof -p 3730 | grep TCP | wc -l

And I check it periodically during the execution. It looks like I always get stuck on average of 30 sockets. It seems like something is limiting my requests to work. And then, when a turn of a request comes, it simply refuses to work because it's already timedout.

Any suggestions?

Thanks.

[Alex Kocharin]

Request can be limiting it, see pool.maxSockets.

I'd suggest to use 'net' module for that and implement http protocol manually. Request module is a bit too heavy for that.

[Jarrett Cruger]

Hey Dan,

I'd try using hyperquest and see if it solves your issue. It will prevent you from being limited by socket pooling and you will just be at the mercy of the operating system. It seems like you already took care of your ulimits so you should be fine. Let me know how it goes.

Cheers,

Jarrett

[James Nylen]

The trick that hyperquest uses to disable the node core pooling behavior is to pass `agent: false` to the options.  The same thing works with request.

More discussion here:  https://github.com/request/request/issues/1231

[Jaret Flores]

What would be the difference between

1. using the global agent with `maxSockets = Inifinity` 
2. using a new agent for each request (so called `agent: false` to request package)

Why is 2 a fix and not 1?