child_process and scaling

[Ω Alisson]

I need to develop a service where the user sends an array of values, I generate a chart with d3.js and convert it to png with a child_process calling imagemagick to send it back to the user. Are there any caveats for child_process when scaling for thousands of requests?

[Zlatko]

You have multiple things there. Thousands requests (per second, I assume) is a lot even without spawning a child shell. Thousands of spawned shells, even without image magick, can bring a busy system down. But then again, you will know best with a test. Try it and see if and where you get stuck.

Even if the system itself handles it, depending on how fast you get to your saturation limits, your response time may get increasingly longer. If that's acceptable and you know there will be downtimes when the jobs can catch up, you might do well with this.

The bigger problem I see there is that you cannot scale horizontally there. You wanna be able to send those convert jobs to other machines as well, probably. And for that, a simple child_process won't do, it would be better if you had some sort of a job queue (MQ, redis pubsub, something of the sort depending if you can allow failures and lost jobs). Then your service that listens for reqs only shoots them to the queue, and listens for a matching response on the response queue. And your actual png generation service can listen on the other side and only take as many requests simultaneously as it makes sense - e.g. maybe per num of CPUs or similar, test there and see what makes sense. And when you have a lot of work, just spawn another png generator on a new machine and you're good, no blocking.

The best thing to do is simply measure. Try to run several processes at once and shoot a thousand requests, see how long it takes for all of them to finish. Thn try to fire 10 waves of 1k reqs, each wave 2 seconds apart, and see how that goes. (Of course, adjust the numbers to your environment.).

Lastly, from my personal experience, these kinds of things regularly grow out of hand and original specs, and it's better to just do job queues from the get-go.

Hope that helps.

Zlatko

[Ω Alisson]

Interesting. I wonder if it is better to use phantomjs for the PNG rendering. Although it is slower, I don't need child_process

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zlatko Đurić]

You should try it. But the bottom line is, in a thousand-reqs-per-sec scenario, you'll almost always hit a bottleneck and need to scale the processing job over multiple machines. So it's a tradeoff - how important is it to you to finish every request, how fast you want results, is the 1k/sec throughput constant or peak... your imagemagick (or phantom binary, whatever you use) can only run so many instances on a single machine. 

So again - test your load such as it is. If you already have a sample JSON, write a node script to make a png out of it - parsing json and calling imagemagick should not take very much time. Then run the script 1k times in a row and see how fast it goes. Then run it in thousand parallel shells and see if that goes well, too. 

testing, testing, testing. Hope I'm not too boring with it.

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/1Oi4uzV2TGw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

To post to this group, send email to nod...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACZE8Y6cqWvVTFF8Ee9SdvsBZTk-DKraTACCs_hq%2BWciTGbRcA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Zlatko

[Matt]

Check out node-canvas for the rendering, then you won't need any child processes.

https://github.com/Automattic/node-canvas

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CADu3pbzKt2Z85t1Of8a2GHw5VA8%3DHWzcxU%3Dm_9Nh%3Dhvx3VMBxw%40mail.gmail.com.

[Ω Alisson]

That's actually a great idea!

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAPJ5V2YnNJ3-T6wHkGMSL7DomNbDqchQ9AM%3DP-HB5wkYSgszDQ%40mail.gmail.com.