Priority level of asynchronous operations

[Felix Geisendörfer]

Hey Ryan,

I ran into a problem where I think node.js asynchronous model is
failing me.

I'm trying to stream large HTTP file uploads to disc. So basically I
got a 'body' listener for the ServerRequest. Every time that fires, I
do some parsing to the stream, and then I pass the current chunk on to
File.write().

This works fine for small files. However, it seems like the
ServerRequest 'body' event fires so fast that it takes complete
priority before the File.write() queue. So for a large file (~270mb)
the entire file contents queue up in memory until the last 'body'
event has fired, and then the file queue gets flushed.

I think I was able to verify that I wasn't doing anything overly
stupid on my end by simply removing the File.write() call which caused
node.js to use a consistent amount of memory during the entire upload
process.

Anyway, let me know what you think. One thing I could imagine is to
tell node.js to not read from the ServerRequest unless some sort of
continue() callback is triggered from the users code.

-- Felix Geisendörfer aka the_undefined

[r...@tinyclouds.org]

Yes this is a problem. Please try out the new methods
request.pause() and request.resume()
which were added in 0638a3a.

2009/8/9 Felix Geisendörfer <fe...@debuggable.com>:

[r...@tinyclouds.org]

request.pause() stops the tcp socket from consuming more data. It is
possible that you will receive a small number of further events after
calling request.pause(), as node must finish parsing the last packet
received. That is, be prepared to get another "body" event or two -
especially if the request is chunked.

[Felix Geisendörfer]

Hey Ryan,

thanks a lot for looking into this so quickly. Unfortunately I'm
having issues with it:

var self = this;
this.req.addListener('body', function(chunk) {
self.req.pause();
self.handleChunk(chunk);
node.debug(chunk.length+' bytes chunk');
setTimeout(function() {
node.debug('resume body');
self.req.resume();
});
});

The idea is that v8 gets to the timeout callback as soon as there are
free cycles again. Unfortunately all I get is a long list of:

DEBUG: 8192 bytes chunk
DEBUG: 8192 bytes chunk
DEBUG: 8192 bytes chunk
DEBUG: 8192 bytes chunk

followed by all the resume calls:

DEBUG: resume body
DEBUG: resume body
DEBUG: resume body
DEBUG: resume body
DEBUG: resume body
DEBUG: resume body
DEBUG: resume body

So it seems like request.pause() is not kicking in at all.

Let me know if you need me to isolate this further.

Best Regards,
-- Felix Geisendörfer

[ry]

We talked about this already on IRC but for reference I fixed this
problem in 738d20f.

[Felix Geisendörfer]

Awesome, works like a charm!

Q: Do you think I'll run into the same problem with process stdout /
stderr events?

-- Felix Geisendörfer aka the_undefined

[ryan dahl]

2009/8/10 Felix Geisendörfer <fe...@debuggable.com>:

>
> Awesome, works like a charm!
>
> Q: Do you think I'll run into the same problem with process stdout /
> stderr events?

Yes. The pause/resume functionality comes from
evcom_stream_read_pause(). Processes currently do not use that. The
main thing I'm working now on is to generalize evcom to use pipes
(currently they only support duplex sockets) and that will allow a
more generalized interface to all input events and pause/resume will
be part of that.