Re: [nodejs] NodeJS ReadStream not reading bufferSize bytes at a time

[Ben Noordhuis]

On Thu, Aug 2, 2012 at 9:21 PM, Gill <taran...@gmail.com> wrote:
> I have a code where the NodeJS server reads a file and streams it to
> response, it looks like:
>
> var fStream = fs.createReadStream(filePath, {'bufferSize': 128 * 1024});
> fStream.pipe(response);
>
> The issue is, Node reads the file exactly 40960 bytes a time. However, my
> app would be much more efficient (due to reasons not applicable to this
> question), if it reads 131072 (128 * 1024) bytes at a time.
>
> Is there a way to force Node to read 128 * 1024 bytes at a time from the
> stream?
>
> Thanks in advance!

No. bufferSize is a hint, not an imperative. It's up to the operating
system to honor it.

[Nathan Rajlich]

You may want to check out "block-stream" if you require buffers in specific chunk sizes: https://npmjs.org/package/block-stream

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Gill]

Ben, thanks for the reply. I have a doubt that its just a hint, because how come it is exactly 40960 bytes every time. The underlying filesystem is a custom coded one, which WILL return the exact number of bytes that were asked for. Line 38 for /lib/fs.js says:

var kPoolSize = 40 * 1024;

Do you think changing it to 128 * 1024 will change anything?

 - Gill

[Gill]

Nathan,

Its not about how the bytes are written, its more about how they are read. I guess the BlockStream just writes them one block at a time, but the underlying fs.createreadstream will still read them at 40960 bytes a time.

 - Gill

nodejs+unsubscribe@googlegroups.com

[Gill]

I think this is related: https://github.com/joyent/node/issues/2098

 - Gill

[Dominic Tarr]

Gill, the kPool is a large buffer that ReadStream cuts buffers out of,
I think, to minimize the allocation of memory...

I was reading through this the other day, and noticed that the default
buffer size is larger than the default kPoolSize. that seems wrong.

you'd need to recompile node, of course, but making kPoolSize larger
should work.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+un...@googlegroups.com

[Gill]

In this particular application, I do not have to worry too much about memory usage. The focus is to read as much data from the file system as possible, in one read operation.

I will try increasing the kPoolSize and recompile Node. I will update this thread with results.

Thanks guys.

 - Gill

> nodejs+unsubscribe@googlegroups.com

[Gill]

I changed kPoolSize to 128 * 1024, and compiled node again.

Now, Node tries to read larger chunks in each read. Although it never reaches 131072 bytes, but its usually in the range of 128000 to 130000. It reads each 128*1024 chunk in two reads, first one about 128000 bytes, and the second one the remaining number of bytes. Although not totally efficient, two reads is better than the 4 reads we were getting earlier. Also, this is very consisted with Java's stream.read function, which also behaves in the same way.

 - Gill