Throttling intensive IO task in node.js

Showing 1-4 of 4 messages
Throttling intensive IO task in node.js Gollum 3/6/12 7:10 AM
I'm playing around with node.js, trying to re-write a particularly
poorly designed part of my production system at work. So far, so good,
I use rabbitmq for messaging, and my node.js part of the system runs
ghostscript command line tool to convert tiff files to pdf. Obviously
I need to make sure I'm not running more than some fixed amount of
conversions at a time. What would be the best way to do this with
node? I understand that maybe node.js isn't really about running heavy
disk IO stuff, but I'm having too much fun with it to quit. I was
considering just using a blocking call to execute command line
utilities but the thing is that some messages don't require this
conversion and there's no need to delay their processing.
Re: [nodejs] Throttling intensive IO task in node.js Tim Caswell 3/6/12 8:06 AM
Node can handle this.  You just need to write the logic to not allow more than X child processes be running at once.  Here is some simple code to give a rough idea.  Please don't block your node process while waiting for child processes.

var queue = [];  // or if the queue gets huge, use a smarter queue.  Array.prototype.shift can get expensive in the hundreds
var running = 0;
var maxRunning = 10;
function doWork(options, callback) {
  if (running >= maxRunning) {
    queue.push({options:options,callback:callback});
    return;
  }
  running++;
  doRealWork(options, function () {
    running--;
    checkQueue();
    return callback.apply(this, arguments);
  });
}
function checkQueue() {
  while (running < maxRunning) {
    var next = queue.shift();
    doWork(next.options, next.callback)
  }
}
  



--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: Throttling intensive IO task in node.js Roly Fentanes 3/6/12 4:01 PM
Re: Throttling intensive IO task in node.js Gollum 3/7/12 6:09 AM
Thank you, Tim, apparently it's not so hard with "thread safe" global
vars.

On Mar 6, 6:06 pm, Tim Caswell <t...@creationix.com> wrote:
> Node can handle this.  You just need to write the logic to not allow more
> than X child processes be running at once.  Here is some simple code to
> give a rough idea.  Please don't block your node process while waiting for
> child processes.
>
> var queue = [];  // or if the queue gets huge, use a smarter queue.
>  Array.prototype.shift can get expensive in the hundreds
> var running = 0;
> var maxRunning = 10;
> function doWork(options, callback) {
>   if (running >= maxRunning) {
>     queue.push({options:options,callback:callback});
>     return;
>   }
>   running++;
>   doRealWork(options, function () {
>     running--;
>     checkQueue();
>     return callback.apply(this, arguments);
>   });}
>
> function checkQueue() {
>   while (running < maxRunning) {
>     var next = queue.shift();
>     doWork(next.options, next.callback)
>   }
>
>
>
>
>
>
>
> }