Batch jobs in Node

[Guilherme Pim]

Hi guys,

Having the need to synchronize some data with external sources, I have written some tasks on Grunt to do so. I wonder if Grunt is actually the tool to write batch jobs in, or there actually is some other alternative.

I have come across Node.io, but I rather stick to Grunt in this case. Do you guys know about some alternatives, please?

Thanks!

Guilherme

[Nuno Job]

Can you share some more details on what you are trying to accomplish?

You have talked about tools (both of which I am unaware of) and said "batch", but apart from that there wasn't much more that could help anyone make an informed suggestion.

Let me know, I'll try to help :)

Nuno

Guilherme

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Guilherme Pim]

Okay, my bad!

I am crawling some web pages to populate a database, and currently doing that with Grunt (http://gruntjs.com/), which is a task-based 'build tool' (that gets me confused, is it supposed to solely build?). 

So, I have created some scripts in the following manner: 

========== Task file ==========

module.exports = function (grunt) {

    grunt.registerTask('courses/update-info', 'Updates all courses info on the database', ... )

}

============================

So then I may have a somewhat controlled access to these jobs from the terminal, with ability to list all existing jobs and their descriptions. Also, I may pass some parameters to each task, although it's not very readable because they can't be associated with a name:

grunt courses/update-info:4:5:2 (these will be passed as the tasks's function arguments)

I guess I could go about creating node scripts and using a parameters library along with console colors and stuff, but I would rather find a framework, if such exists.

Did I make myself clearer at all? haha

Thanks for you time,

Guilherme

2012/10/5 Nuno Job <nunojo...@gmail.com>

[greelgorke]

use registerMultiTask.

define your websites in the main gruntfile.

may be a better way is to use plain node script for it.

[Ted Young]

Sounds like you just want a node service that does the scraping, and provides an http api for adding tasks and querying what tasks are currently running.  Then have a separate command-line app that can talk to the web service.  And maybe also an express app that can talk to it via a browser.  I use that pattern a lot, but haven't yet written an abstraction for it.  An abstraction for easily banging out command-line apps with tab-complete, etc that just talk to web services would be a useful thingie though.

I'm not sure if there's a generic node-based job-processor out there, but you might like https://github.com/defunkt/resque.  It's ruby, but each job could just be a system call to start a node process.

Ted

[Ted Young]

Actually, kue looks like a resque-like thing: http://learnboost.github.com/kue/

Ted