Please explain how to use the new http.Agent

[Rambo]

I'm trying to make hundreds of requests to different hosts and I don't know which is the best approach to handle a request queue.

I can implement a "task manager" which handles up to X concurrent requests at a time or I can use the native http.Agent which can be configured to use up to X requests per host.

I would like to implement the latter, but I saw that 0.5+ has a new http module that works differently from 0.4 and reading the documentation didn't help me understand how to create a new Agent instance and how to limit concurrent requests per host.

More specifically:

• How to create a new http.Agent? which are the parameters?
  
• http.globalAgent is the only instance node uses for all the client requests? if so, how can I set maxSockets for different hosts? do I need to create a new Agent instance per host and set maxSockets property?
  
• I really don't understand which is the purpose of setting {agent:false} when making a request. How does this "pool" thing work?
  

Thanks for your help.

[Mikeal Rogers]

It's actually simpler than the previous Agent, I'll explain.

An Agent instance is now a pool for *many* hosts, not a single host. The http module has a default instance available at

require('http').globalAgent

So if you just want to change the default maxSockets per client you just:

require('http').globalAgent.maxSockets = 500

If you want to keep different pools with different maxSockets you can can create them

x = new http.Agent()

And then use them in your http requests:

http.request({agent:x})

-Mikeal

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Rambo]

Ok thanks but one more thing.

When I create a new Agent instance and set the maxSockets property, for what hosts does that work? I need to establish a max amount of concurrent requests per host. Is it automatically detected from the http.request options?

And what about the {agent:false} option?

Thanks.

[Ted Young]

You will need to create an Agent per host, and use that agent when you make a request.

agents = {

  'foo.com': new http.Agent(),

  'bar.com': new http.Agent()

}

http.request({host:'foo.com', agent: agents['foo.com']});

agent:false opts that request out of any connection pooling.  

Ted 

[Demián Andrés Rodriguez]

So basically, setting agent:false automatically creates an agent for me instead of using globalAgent?

[Ted Young]

If you set agent to false, it will immediately attempt to make the
request, and keep-alive will be set to false. If you are making a lot
of connections to the same host, and want to turn keep-alive on, it
would be better to use an Agent. If you don't want to limit the
number of connections, you could set maxSockets on that agent to an
impossibly high number.

With the new http module, I'm not sure what the benefit would be to
using agent:false, vs using an agent with a high maxSockets.

Ted

[Floby]

For a ridiculously high number you can use Infinity, which is a
special value of Number. You can get it with Math.min().
(yeah Math.max() seems to return negative infinity). I really thought
there was another way to get it.

[Jann Horn]

Maybe “1/0“?

Am 12.09.2011 10:10 schrieb "Floby" <floren...@gmail.com>:

For a ridiculously high number you can use Infinity, which is a
special value of Number. You can get it with Math.min().
(yeah Math.max() seems to return negative infinity). I really thought
there was another way to get it.

On Sep 11, 6:56 pm, Ted Young <t...@radicaldesigns.org> wrote:

> If you set agent to false, it will...

[Demián Andrés Rodriguez]

Ok thanks for the advise but it doesn't make any sense to set infinity connections, I was looking to limit them like... 3 per host :P

--

[Isaac Schlueter]

Math.min returns Infinity, because it is implemented something like this:

function min () {
var m = Infinity
for (var i = 0; i < arguments.length; i ++) {
if (arguments[i] < m) m = arguments[i]
}
return m
}

function max () {
var m = -Infinity
for (var i = 0; i < arguments.length; i ++) {
if (arguments[i] > m) m = arguments[i]
}
return m

[Mikeal Rogers]

On Sep 10, 2011, at September 10, 20119:33 AM, Demián Andrés Rodriguez wrote:

So basically, setting agent:false automatically creates an agent for me instead of using globalAgent?

In 0.4.x yes, a new Agent will be created when passing agent:false.

In 0.5.3+ no agent is created and a new socket is passed to onSocket() and Connection is set to "close". 

-Mikeal

[Ted Young]

Actually now that I look at the Agent code, it looks like it sets maxSockets on a per domain basis in an Agent:

Agent.prototype.addRequest = function(req, host, port) {
  var name = host + ':' + port;
  if (!this.sockets[name]) {
    this.sockets[name] = [];
  }
  if (this.sockets[name].length < this.maxSockets) {
    // If we are under maxSockets create a new one.
    req.onSocket(this.createSocket(name, host, port));
  } else {
    // We are over limit so we'll add it to the queue.
    if (!this.requests[name]) {
      this.requests[name] = [];
    }
    this.requests[name].push(req);
  }
};

What is the expected behavior of maxSockets on an Agent?  Should maxSockets represent the total number of sockets available to that agent, or the total number of sockets available to each host:port in that agent?  My vote is for the former, since it's possible to build the later on top of it, but not vice versa.

Ted 

[Dean Landolt]

On Mon, Sep 12, 2011 at 2:07 PM, Isaac Schlueter <i...@izs.me> wrote:

Math.min returns Infinity, because it is implemented something like this:

function min () {
 var m = Infinity
 for (var i = 0; i < arguments.length; i ++) {
   if (arguments[i] < m) m = arguments[i]
 }
 return m
}

function max () {
 var m = -Infinity
 for (var i = 0; i < arguments.length; i ++) {
   if (arguments[i] > m) m = arguments[i]
 }
 return m

}

tl;dr: you can get a reference to Infinity by typing `Infinity` -- it's a free variable.

FWIW `Infinity` can be overwritten in local scope, like `undefined` and other free vars, so Math.min may be a better way to get a handle. but this can be overwritten globally, so really the safest way to spell Infinity is probably 1/0, which also happens to be the shortest. So that's nice, and makes this (slightly) more than a pedantic jswtf rant :)

[Floby]

yeah, I don't know why I didn't even try that :)

[Demián Andrés Rodriguez]

A bit of a silly question, but obviously the socket limits are valid per node process, right?

[Mikeal Rogers]

More precisely they are per-Agent. An instance of the Agent determines how the socket limit for sockets attached to that agent.

It's also worth noting that web sockets requests get pulled out of the pool on "upgrade" and don't could against the pool limit.

-Mikeal