highly asynchronous app gets stuck

[kiril...@gmail.com]

Is there such a thing as a maximum number of functions you can put in the event loop?

I have a highly asynchronous app, which results in a ton of async callbacks getting pushed into the event loop. One "operation" could result in several tens of async functions. Everything runs fine at low concurrency (like 5000 concurrent operations or so), but when I crank it up (20,000), it runs fine for a little while (less than a second), and then gets stuck. It queues up a bunch of callbacks, and they never get resolved. The app it just at a standstill. Is there any technical reason this would be happening?

The app in question is basically a task runner: https://github.com/catdad/grandma/

At first, I suspected that IPC was the issue (because multiple threads), so I wrote a whole custom IPC implementation using named sockets. Still the same issue. The IPC bits work just fine, more process.nextTick instances get called, but the function never executes once the concurrency is high enough.

I have tried setting --max_old_space_size to a high number (thinking it might be a memory issue) and that did not help. I have also set UV_THREADPOOL_SIZE to 128, 256, and 512, thinking that maybe the libuv thread pool is getting saturated, and that doesn't seem to be helping either.

I have found that when using longer async tasks (process.nextTick vs. setTimeout(cb, 10) vs. an http request) that longer tasks get stuck at a much lower concurrency. Assuming there is no bug in the code (I know, everyone hates doing that), is there any technical reason for such an issue?

[Ryan Graham]

I only had a few minutes to look for obvious things, so I don't have a solution, but I do have some observations.

1. You've done a pretty good job at organizing the code from what I've see, so +1 to that.
2. You seem to be under the impression that child_process.fork() spawns threads, when it is actually spawning entire new node processes.

In light of #2, you could be hitting OS enforced limits for things like the number of processes per user. Given that each process creates a new IPC channel, you could also be hitting a file descriptor limit. It's not clear which concurrency parameter you are referring to in your post, but I would suggest keeping your "thread" parameter tied to the number of CPU cores (0.5x-2x depending on the type of code being run).

~Ryan

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/267a1ccf-27cc-4a82-8009-d3794c56fcf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

~Ryan