NodeJS clustering

[Dirk Hoving]

Hi,

We are trying to cluster a level deeper than is usual.

NodeJS seems to developed to be able to run a code than can be accessed by multiple users and every instance can be run by a different worker.

Now we have a case, where we have only one user with a very complicated backend process. This single instance of the code causes the CPU to overload at times so we want to split up the work over different CPU cores. We have tried multiple approached but haven't been able to make it work yet, any insight to this?

Thanks, Dirk

[boby rahmawan]

Hi Dirk. May be u can split into child process. So it can split usage resource CPU. CMIIW

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/cf3a1c41-df02-4888-b713-4784ec65c4ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Boutell]

What approaches have you tried? Is the complicated backend process itself written in node?

This sounds like a good use for child_process.spawn. You could keep track of how many instances are running easily if there is a single web interface node process, or with a little more work if you need to use an external scoreboard like redis.

[Oleg Verych]

Hi,

 

We are trying to cluster a level deeper than is usual.

NodeJS seems to developed to be able to run a code than can be accessed by multiple users and every instance can be run by a different worker.

IMHO Node.JS can handle particular request (be it user, be it some URL route, etc.) by a different worker

Now we have a case, where we have only one user with a very complicated backend process. This single instance of the code causes the CPU to overload at times so we want to split up the work over different CPU cores.

 

We have tried multiple approached but haven't been able to make it work yet, any insight to this?

What are they?

If that user doing many requests to particular URL of your app and those request one by one eat CPU very much, then maybe you can handle that URL by own cluster running on another TCP port and have nginx as router pointing to that URL? There must be some splitting boilerplate for that api controllers (js files) to load config, connect to db, etc as usual app does. 

Boby Rahmawan says "split into child process". Maybe it is the same.

[Kevin Ingwersen (Ingwie Phoenix)]

Hello Dirk,

I dont want to sound rude, but you provided rather little detail, leaving us with barely anything to work with. I myself have had clustering issues, and there are a variety of things you can do - spreading from using RPC methods like hprose over to using a cluster solution like SocketCluster or PowerHouse, over to simply running the more independent parts as separate child_process.spawn’s.

The more you can tell, the better we can answer :)

Kind regards,
Ingwie.

[Dirk Hoving]

Thanks for all the replies, lost the topic so didn't reply back earlier.

It is a case where only server side is running (one instance that does a lot of calculations on data it retrieves from websites and performs actions based on these calculations (e.g. database entries or other asynchronized requests)).

One good solution we have found is to start up different parts of the code in another node.js instance and have the several node.js instances communicate with each other over sockets internally.

[klrumpf]

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/cf3a1c41-df02-4888-b713-4784ec65c4ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

there was also an interesting article outlining a similar strategy at
http://techblog.spanning.com/2015/10/12/Long-running-Node-processes/

Karl