Throttling intensive IO task in node.js

[Gollum]

I'm playing around with node.js, trying to re-write a particularly
poorly designed part of my production system at work. So far, so good,
I use rabbitmq for messaging, and my node.js part of the system runs
ghostscript command line tool to convert tiff files to pdf. Obviously
I need to make sure I'm not running more than some fixed amount of
conversions at a time. What would be the best way to do this with
node? I understand that maybe node.js isn't really about running heavy
disk IO stuff, but I'm having too much fun with it to quit. I was
considering just using a blocking call to execute command line
utilities but the thing is that some messages don't require this
conversion and there's no need to delay their processing.

[Tim Caswell]

Node can handle this.  You just need to write the logic to not allow more than X child processes be running at once.  Here is some simple code to give a rough idea.  Please don't block your node process while waiting for child processes.

var queue = [];  // or if the queue gets huge, use a smarter queue.  Array.prototype.shift can get expensive in the hundreds

var running = 0;

var maxRunning = 10;

function doWork(options, callback) {

  if (running >= maxRunning) {

    queue.push({options:options,callback:callback});

    return;

  }

  running++;

  doRealWork(options, function () {

    running--;

    checkQueue();

    return callback.apply(this, arguments);

  });

}

function checkQueue() {

  while (running < maxRunning) {

    var next = queue.shift();

    doWork(next.options, next.callback)

  }

}

  

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Roly Fentanes]

Use a job manager

https://github.com/flatiron/neuron

[Gollum]

Thank you, Tim, apparently it's not so hard with "thread safe" global
vars.