What's the limit of spawning child_processes?

[simo]

I have to serve a calculation via algorithm, I've been advised to use a child process per each opened socket, what I am about to do is something like that:

var spawn = require('child_process').spawn;
var child = spawn('node', ['algorithem.js']);

I know how to send argument to the algorithm process and how to receive results.

What I am concerned about, is how many socket (each socket will spawn a process) I can have? How can I resolve this with my cloud hosting provider? so that my app gets auto scaled?

What's the recommended node js cloud hosting provider?

Finally, is this a good approach in using child processes?

[Alan Hoffmeister]

I think that the maximum number of opened files is the only unix's limitation for the number of sockets, and this can be changed. 

--
Att,
Alan Hoffmeister

2013/3/24 simo <as3...@gmail.com>

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Samir Sabri]

So, is it a recommended approach to spawn a new process for each opened socket?

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/qP_hsLZMq1A/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

--
Kind Regards,

--------------------------------------------- 

Samir Sabri
Software Architect& Developer

www.dcaclab.com

Jordan-Middle East

[Floby]

no. Definitely not.

Spawn a predefined number of workers (using the cluster api) and queue jobs in the master process then send them to the workers when they are ready to process.

[greelgorke]

a better option is spawning a separate cluster of algorthm workers as service and using something like https://github.com/substack/dnode for communication. spawning a new process takes tens of miliseconds, which probably is to long for short living processes. 

[Samir Sabri]

Thanks Floby, I appreciate your advice, do you know a good blog or source that talks about workers and jobs queuing?

[Samir Sabri]

You idea is great! but how can I manage load balance on the algorthm workers? if I have 100 algorthm workers for example?

[greelgorke]

http://nodejs.org/api/cluster.html#cluster_how_it_works

[Samir Sabri]

Yes, I already found it, amazing tech

[Matteo Collina]

Hi Samir,

I solved that problem for CPU intensive jobs. I ended up with my own control flow library, because no one has been gone in the route of "queuing" task with a work-in-progress limit in a sane way before: https://github.com/mcollina/kanban. There is async.queue, but it lacks some flexibility, if you have a processing pipeline with different constraint.

The number of worker processes you want to spawn in a given period of time is strictly dependent to the number of concurrent process you might want to run in your system. With node.js you can end up prettly quickly with load > 1, and then your performance gets worse as more processes are added.

Let me know if you need more help on this subject.

Cheers,

Matteo

2013/3/25 Samir Sabri <as3...@gmail.com>

[Matt]

On Mon, Mar 25, 2013 at 11:14 AM, Matteo Collina <matteo....@gmail.com> wrote:

The number of worker processes you want to spawn in a given period of time is strictly dependent to the number of concurrent process you might want to run in your system. With node.js you can end up prettly quickly with load > 1, and then your performance gets worse as more processes are added.

Remember "load" should be divided by number of CPUs. It's generally a bad figure to use to see if you have a problem.

[greelgorke]

there is also threads-a-gogo lib which can be used to spawn threads. i haven't use it yet (never really had to), but it is another option.

[Samir Sabri]

@Matteo Thanks, but as I am still at the beginning of this, I will need to find a way to simulate lots of users, and to find a way to profile performance, then I can decide what to do, if you can help me how to simulate virtual users and how to profile performance of my node app in real time, it will be a great help!

@Mat you said ("load" should be divided by number of CPUs) isn't load automatically divided when I node cluster? 

Kind Regards,

Samir

[Matt]

On Mon, Mar 25, 2013 at 2:17 PM, Samir Sabri <as3...@gmail.com> wrote:

@Mat you said ("load" should be divided by number of CPUs) isn't load automatically divided when I node cluster? 

The load we were referring to is "load average": http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

[Samir Sabri]

Isn't there a cloud hosting provider who would auto scale my app? would it be possible to stay away from load avarage details so that my cloud hosting provider handle it? if so, who do you recommend?

--