looping through a large array - forEach vs process.setImmediate vs process.nextTick

[krishnan venkat]

I am currently looping through large arrays in a child thread, I know that the thread blocks until the loop ends but is it a good idea to use process.setImmediate or nextTick instead of the default for loop.

There is no async code in the for loop and I am worried that process.nextTick might starve the event thread.

Is this a good use case for using the latter 2 options?

[Angel Java Lopez]

First, it depends on your context. Now, I presume you are in a web app (in a console application, I don't worry about the long array in many cases).

First ideas:

Process 100 (or 1000 or what you decide) items and then setImmediate

Measure the impact in your application (block other clients?)

Measure the time of the different approaches

After "make it work", maybe an step towards "make it right": instead of 100 items, start to investigate: I process x items, to spent n millisecons, then setImmediate

(ummm... the last time I used nextTick, some error message started to appear, circa 0.8 to 0.10)

(recently, I realized that setImmediate is not defined in some browser, so if your library should run in client-side too, use setTimeout(fn, 0)

Other comments?

Angel "Java" Lopez

@ajlopez

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/752ff9b8-8e0d-4150-9b93-f87602630431%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[// ravi]

On May 13, 2014, at 2:42 AM, krishnan venkat <krishna...@gmail.com> wrote:

I am currently looping through large arrays in a child thread, I know that the thread blocks until the loop ends but is it a good idea to use process.setImmediate or nextTick instead of the default for loop.

There is no async code in the for loop and I am worried that process.nextTick might starve the event thread.

Don’t you mean “I am worried that without process.nextTick…”? To answer your question, I’d guess it depends on the typical execution time for the loop and how dependent other parts of your logic flow are on what occurs within the loop.

—ravi

[Francesco Mari]

nextTick and setImmediate can help to convert your synchronous code to an asynchronous model of execution.

The difference between the two is subtle. nextTick schedules a function to be executed at the very next tick in the event loop. If some I/O operation is pending, they will be delayed if you use nextTick. setImmediate, instead, schedules a callback to be executed after any I/O operation in the event loop, thus preventing your I/O callbacks to starve. For your case you should use setImmediate instead of nextTick.

How you organize the rest of your code depends on how you process the input array. Are you converting your input array to another array containing the processed elements? Are you aggregating the input array to produce another piece of data? Can the elements in the array be processed in chunks, or the processing must happen one element after the other? Is order important when traversing the array?

[Aria Stewart]

Just how big are you talking? ns? us? ms duration?

setImmediate is probably what you want if you want to yield.

Aria

[Matt]

How large an array?

[greelgorke]

you said it's a child_process. so it depends on what is it for the main process. if the whole purpose of this child_process is to loop over the array as fast as it can, then just loop, send the result back and fetch the next message. if the purpose is to get messages from outside and present them anwers, even if they are partial, then use setImmediate after some elements or even use a stream.