NodeJs application architecture - Need suggestion

[Mouli Kumar]

I am working on a NodeJs based application primarily a Web app written using AngularJs/HTML/CSS.

I have a need for running a worker process in my server which will process background jobs. Like, sending emails to 100s of staff at the end of the day, or processing billing related records etc. Sometimes I also want this process to listen to the main NodeJS process that is serving the webui, so that the main process can handover the work to the worker for immediate processing.

Things I have considered

1) Worker process will run as another NodeJs app (entirely different app, no forking, no clustering from the main process).

2) Worker process will open a server socket for listening to the main nodejs process for any immediate job delegation.

3) Worker process will poll the database (mongodb) periodically to pick up unprocessed records (say, every 5 sec)

Your expert suggestions needed

1) Is the approach explained above viable? 

2) Communicating between the 2 nodejs processes using sockets, is it a recommended approach. I may have to move this worker process to a different server and may even scale out if required.

3) Is mongodb a good option for implementing a workload queue. My web application uses MongoDB as database, so I did not consider moving to a different database for the queue. Any suggestions for in-memory queue.

Thank you very much in advance.

Mouli

[Aria Stewart]

On May 21, 02014, at 5:06, Mouli Kumar <mouli...@gmail.com> wrote:

> I am working on a NodeJs based application primarily a Web app written using AngularJs/HTML/CSS.
> I have a need for running a worker process in my server which will process background jobs. Like, sending emails to 100s of staff at the end of the day, or processing billing related records etc. Sometimes I also want this process to listen to the main NodeJS process that is serving the webui, so that the main process can handover the work to the worker for immediate processing.
>
> Things I have considered
> 1) Worker process will run as another NodeJs app (entirely different app, no forking, no clustering from the main process).
> 2) Worker process will open a server socket for listening to the main nodejs process for any immediate job delegation.
> 3) Worker process will poll the database (mongodb) periodically to pick up unprocessed records (say, every 5 sec)
>
> Your expert suggestions needed
> 1) Is the approach explained above viable?

Yes.

> 2) Communicating between the 2 nodejs processes using sockets, is it a recommended approach. I may have to move this worker process to a different server and may even scale out if required.

Sounds decent. Consider Gearman instead though.

> 3) Is mongodb a good option for implementing a workload queue. My web application uses MongoDB as database, so I did not consider moving to a different database for the queue. Any suggestions for in-memory queue.

I love Gearman for queueing. It gives limited guarantees (design for actions to be repeatable in the case of failures detected). It’s the right mix for a robust, simple app, I think.

Aria

[// ravi]

These are good options. As you may know, Cluster provides a built-in channel for the master to communicate with the workers but since you are looking to scale out that wouldn’t be the path for you. What about job resilience, persistence, etc? What happens when there’s no worker responding? A database-based solution might serve you better if these issues are important. Redis, Couch, RethinkDB (and others) all provide in-memory+persistence capabilities if you are looking for alternatives. Then there are the tools built specifically for job spooling, e.g., beankstalkd (http://kr.github.io/beanstalkd/), Kue (http://learnboost.github.io/kue/), etc.

—ravi