flushing file streams in streamline 1.0 / ez-streams 1.2

[Phil Sung]

Hello,

I'm not sure if I am missing some important API documentation, or am just doing something wrong, but I ran into the following issue when upgrading to streamline 1.x / ez-streams 1.2 (which I tried to do simultaneously).

The following program, which is supposed to write some lines to a file, typically exits without having written anything to the file (leaving a file of length 0).

#!/usr/bin/env _node

var ez = require('ez-streams');

function main(_) {
  var outputFile = ez.devices.file.text.writer('testoutput.txt');
  var i;
  for (i = 0; i < 100; ++i) {
    outputFile.write(_, 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'utf-8');
    outputFile.write(_, '\n', 'utf-8');
  }
  outputFile.write(_, undefined); // Not sure if this is needed, but it doesn't work without this, either
  outputFile.end();
}

main(function (err) {
  if (err) {
    console.error(err.stack);
    process.exit(1);
  }
  process.exit();
});

This is on:

* streamline 1.0.13

* ez-streams 1.2.1

* streamline-runtime 1.0.16

I presume it's a race condition between the flushing of the stream data and the exit of the program, because if I add a setTimeout(_, 1000) after the call to end(), the data is written.

In the past, under streamline 0.10.17 and ez-streams 0.1.7, what I did was to invoke end() with a callback:

outputFile.end(_);

which seems to cause the data to be flushed reliably. But this method no longer takes a callback argument. Can you recommend how in general I can wait to ensure that the data has been flushed?

Thanks in advance for your help,

Phil Sung

[Bruno Jouhier]

Hi Phil,

Yes this is a known race condition with process.exit. 

Writing undefined is sufficient to end the stream. You don't need an extra end call.

The problem comes from the node apis. Ez-streams considers that the stream is flushed when emitter.write returns true or when the drain event arrives but it looks like data is still in the air when that happens with file streams. I was thinking of reimplementing the file reader and writer directly on top of low level fs api. That should be more efficient and it should fix it.

I won't work on it till Jan 15 because i'm on vacation without computer.

Bruno

Envoyé de mon iPhone

---

The contents of this e-mail and any attachments are confidential and only for use by the intended recipient. Any unauthorized use, distribution or copying of this message is strictly prohibited. If you are not the intended recipient please inform the sender immediately by reply e-mail and delete this message from your system. Thank you for your co-operation.

--
You received this message because you are subscribed to the Google Groups "streamline.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to streamlinejs...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Phil Sung]

Good to know. Is this a known issue in node.js or do you think that the way forward is really to change ez-streams? Is there a bug tracker issue anywhere that tracks this?

Thanks for the quick reply, and enjoy your vacation!

Phil

[Bruno Jouhier]

I'd rather fix in ez-streams. The node api has lots of events and it is not very clear which one signals a complete flush. I was assuming that drain was the one. It works well for flow control but it does not correspond to a full flush. It's difficult to say that it is a node bug because there are other events at the end of a stream and i should probably wait on a different one to ensure flush. Also i wrote ez-streams at a time where node streams were still a moving target. Now things have stabilized and it's probably the right time to investigate and fix it. 

I have two options to fix: 1) find the right event and wait for it in the last write (the one with undefined) or 2) re-implement file streams with the low level api.

So i propose that you log an issue in ez streams github and i'll follow up from there when i get back.

Bruno

Envoyé de mon iPhone

[Bruno Jouhier]

A priori the event that comes after flush is 'finish'. 

Envoyé de mon iPhone

[Phil Sung]

Thanks for the diagnosis-- I filed https://github.com/Sage/ez-streams/issues/41

Cheers,

Phil