new node module: fd-slicer

[Andrew Kelley]

Safe fs.ReadStream and fs.WriteStream using the same fd.

Let's say that you want to perform a parallel upload of a file to a remote server. To do this, we want to create multiple read streams. The first thing you might think of is to use the {start: 0, end: 0} API of fs.createReadStream. This gives you two choices:

1. Use the same file descriptor for all fs.ReadStream objects.

2. Open the file multiple times, resulting in a separate file descriptor for each read stream.

Neither of these are acceptable options. The first one is a severe bug, because the API docs for fs.write state:

Note that it is unsafe to use fs.write multiple times on the same file without waiting for the callback. For this scenario, fs.createWriteStream is strongly recommended.

fs.createWriteStream will solve the problem if you only create one of them for the file descriptor, but it will exhibit this unsafety if you create multiple write streams per file descriptor.

The second option suffers from a race condition. For each additional time the file is opened after the first, it is possible that the file is modified. So in our parallel uploading example, we might upload a corrupt file that never existed on the client's computer. We want multiple streams to use the same file descriptor.

This module solves this problem by providing createReadStream and createWriteStream that operate on a shared file descriptor and provides the convenient stream API while still allowing slicing and dicing.

https://npmjs.org/package/fd-slicer

https://github.com/andrewrk/node-fd-slicer

[Matt]

Can't you just use a stream multiplexer (AKA "Tee") for this?

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/9cdaf49e-bb2f-47d9-9beb-65c1de1d3ab2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mscdex]

On Friday, August 8, 2014 4:38:21 PM UTC-4, Andrew Kelley wrote:

Note that it is unsafe to use fs.write multiple times on the same file without waiting for the callback. For this scenario, fs.createWriteStream is strongly recommended.

Is this because the Buffer passed to `fs.write()` cannot be (re)used until the callback is executed or does it have to do with something at the OS level?

[Andrew Kelley]

On Saturday, August 9, 2014 8:34:10 AM UTC-7, mscdex wrote:

Is this because the Buffer passed to `fs.write()` cannot be (re)used until the callback is executed or does it have to do with something at the OS level?

I think because multiple threads would each try to seek at the same time, so you might get something like this:

Thread 1 seek to position 10

Thread 2 seek to position 50

Thread 1 write 
Thread 2 write

Oops.

On Saturday, August 9, 2014 8:24:44 AM UTC-7, Matt Sergeant wrote:

Can't you just use a stream multiplexer (AKA "Tee") for this? 

 How does this address the problem in any way?

[Matt]

On Sun, Aug 10, 2014 at 1:49 AM, Andrew Kelley <super...@gmail.com> wrote:

Can't you just use a stream multiplexer (AKA "Tee") for this? 

 How does this address the problem in any way?

You create one read stream, then multiplex it to two writers. Isn't that the problem you were trying to solve?

Matt.