Re: [node-webkit] node fs writefile doesn't write whole file to disk

[Walter Purvis]

I'm clueless, but a similar problem was reported not long ago:

http://stackoverflow.com/questions/21807778/fs-writefile-only-saving-part-of-string

So you're not the only one encountering this.

On Sun, Apr 13, 2014 at 4:49 PM, Rick H. <cub...@gmail.com> wrote:

Hello, all.  I'm not sure whether this behavior of node's is only in a node-webkit context, or not.  

I have the following code: 

            function saveFileAs() {

                var inputSaveAsFile = document.querySelector(inputSaveAsFileId);

                inputSaveAsFile.addEventListener('change', function(evt) {

                    var textBox = document.querySelector('#theText');

                    var data = textBox.innerHTML;

                    var path = this.value;

                    fs.writeFile(path, data, {encoding: 'ascii'}, function(err) {

                        if (err) throw err;

                        console.log('writeFile succeeded.');

                    });

                }, false);

                inputSaveAsFile.click();

            }

I can type something into my contenteditable div that has the id "theText," and use my file menu to come to the saveFileAs function.  It opens the file dialog as expected, and I come to the fs.writeFile call.  (I have an input with type file and the attribute nwsaveas.)  Right before the call: 

path

"C:\node-webkit\experiment\Inigo_Sunday.txt"

data

"Hallo.<div>My name is Inigo Montoya.</div><div>You killed my father.</div><div>Prepare to die.</div>"

After the call I type out the file, and it isn't all the data:

Rick@RicksLaptop /cygdrive/c/node-webkit/experiment $ cat Inigo_Sunday.txt Hallo.<div>My name is Inigo Montoya.</div><div>You Rick@RicksLaptop /cygdrive/c/node-webkit/experiment $

I am mystified. Does anyone know what's going on?

Regards, Rick

--
You received this message because you are subscribed to the Google Groups "node-webkit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-webkit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[LZAntal]

Hi,

Try changing the encoding to utf-8

Laszlo

http://twitter.com/LZAntal

[Walter Purvis]

Yes, I bet it is an encoding issue.

The part that's getting written is exactly half the size of the data that's supposed to be written

"Hallo.<div>My name is Inigo Montoya.</div><div>You" = 50 bytes

"Hallo.<div>My name is Inigo Montoya.</div><div>You killed my father.</div><div>Prepare to die.</div>" = 100 bytes

[Bastian KG]

This is a asynch method.Without going into details, try to use synchronized version :

fs.writeFileSync

[Rick H.]

Hello, all.  I tried a variety of encodings, and utf16le seemed to work.  But now I am concerned because my callback from writeFile gets called more than once.  

I refreshed the page, and saved the contenteditable div to a file three times.  The first time the callback was called once.  The second time the callback was called two times.  And the third time the callback was called three times.  This can be most easily seen by looking at the console log below.  I also included the code, and the result of typing the file out to the screen. 

I wonder whether I screwed something up by putting everything into one index.html file.  It would be tidier to put the javascript stuff into .js files and refer to them.  The node-webkit documentation is trying to tell me something about contexts for node, and for each window.  Maybe I'm running afoul of contexts. 

Ideas? 

The way it appears in the console of the debugger:

save as index.html:90

evt: index.html:51

evt.type = changeevt.defaultPrevented = falseevt.eventPhase = 2 index.html:53

encoding = utf16le index.html:56

path = 'C:\node-webkit\experiment\140414.2050.txt' index.html:57

data = ' Hello there.<div>I'm just typing some stuff.</div>' index.html:58

err: index.html:60

null index.html:61

writeFile succeeded. index.html:63

// That was stored just fine.

undefined

save as index.html:90

evt: index.html:51

evt.type = changeevt.defaultPrevented = falseevt.eventPhase = 2 index.html:53

encoding = utf16le index.html:56

path = 'C:\node-webkit\experiment\140414.2051.txt' index.html:57

data = ' Hello there.<div>I'm just typing some stuff.</div><div>I should add some stuff,&nbsp;</div><div>and see what happens.</div>' index.html:58

evt: index.html:51

evt.type = changeevt.defaultPrevented = falseevt.eventPhase = 2 index.html:53

encoding = utf16le index.html:56

path = 'C:\node-webkit\experiment\140414.2051.txt' index.html:57

data = ' Hello there.<div>I'm just typing some stuff.</div><div>I should add some stuff,&nbsp;</div><div>and see what happens.</div>' index.html:58

err: index.html:60

null index.html:61

writeFile succeeded. index.html:63

err: index.html:60

null index.html:61

writeFile succeeded. index.html:63

// That one was also stored fine. But look. The writeFile called the callback twice.

undefined

save as index.html:90

evt: index.html:51

evt.type = changeevt.defaultPrevented = falseevt.eventPhase = 2 index.html:53

encoding = utf16le index.html:56

path = 'C:\node-webkit\experiment\140414.2053.txt' index.html:57

data = ' Hello there.<div>I'm just typing adding some stuff&nbsp;</div><div>in the middle. &nbsp;And deleting the rest.</div>' index.html:58

evt: index.html:51

evt.type = changeevt.defaultPrevented = falseevt.eventPhase = 2 index.html:53

encoding = utf16le index.html:56

path = 'C:\node-webkit\experiment\140414.2053.txt' index.html:57

data = ' Hello there.<div>I'm just typing adding some stuff&nbsp;</div><div>in the middle. &nbsp;And deleting the rest.</div>' index.html:58

evt: index.html:51

evt.type = changeevt.defaultPrevented = falseevt.eventPhase = 2 index.html:53

encoding = utf16le index.html:56

path = 'C:\node-webkit\experiment\140414.2053.txt' index.html:57

data = ' Hello there.<div>I'm just typing adding some stuff&nbsp;</div><div>in the middle. &nbsp;And deleting the rest.</div>' index.html:58

err: index.html:60

null index.html:61

writeFile succeeded. index.html:63

err: index.html:60

null index.html:61

writeFile succeeded. index.html:63

err: index.html:60

null index.html:61

writeFile succeeded. index.html:63

// This worked too. Note now it revisited three times.

undefined

The resulting contents of the files:

Rick@RicksLaptop /cygdrive/c/node-webkit/experiment

$ cat 140414.1842.txt

<div>Hallo.</div><div>My name is Inigo Montoya.</div><div>You killed my father.</div><div>Prepare to die.</div>

Rick@RicksLaptop /cygdrive/c/node-webkit/experiment

$ cat 140414.2050.txt

            Hello there.<div>I'm just typing some stuff.</div>

Rick@RicksLaptop /cygdrive/c/node-webkit/experiment

$ cat 140414.2051.txt

            Hello there.<div>I'm just typing some stuff.</div><div>I should add some stuff,&nbsp;</div><div>and see what happens.</div>

Rick@RicksLaptop /cygdrive/c/node-webkit/experiment

$ cat 140414.2053.txt

            Hello there.<div>I'm just typing adding some stuff&nbsp;</div><div>in the middle. &nbsp;And deleting the rest.</div>

Rick@RicksLaptop /cygdrive/c/node-webkit/experiment

$

The code, from index.html:

44            function saveFileAs() {

45                var inputSaveAsFile = document.querySelector(inputSaveAsFileId);

46                inputSaveAsFile.addEventListener('change', function(evt) {

47                    var textBox = document.querySelector('#theText');

48                    var data = textBox.innerHTML;

49                    var path = this.value;

50                    var encoding = 'utf16le';

51                    console.log('evt:');

52                    // console.log(JSON.stringify(evt) );

53                    console.log('evt.type = ' + evt.type + 

54                                'evt.defaultPrevented = ' + evt.defaultPrevented + 

55                                'evt.eventPhase = ' + evt.eventPhase);

56                    console.log('encoding = ' + encoding);

57                    console.log('path = \'' + path + '\'');

58                    console.log('data = \'' + data + '\'');

59                    fs.writeFile(path, data, {encoding: encoding}, function(err) {

60                        console.log('err:');

61                        console.log(err);

62                        if (err) throw err;

63                        console.log('writeFile succeeded.');

64                    });

65                }, false);

66                inputSaveAsFile.click();

67            }

[Anatoly Pashin]

You just create new event listener on every click. This way you got it incrementing each time. You've to attach it once or create new <input> each click instead of  var inputSaveAsFile = document.querySelector(inputSaveAsFileId);

--
You received this message because you are subscribed to the Google Groups "node-webkit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-webkit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Пашин Анатолий,
эникейщик.

[Rick H.]

Hi, Anatoly. 

At first I thought "That's it!"  Then I realized the event handler isn't being called.  If it were, we would see the output from the console logs on lines 57 & 58.  Only the callback from the call to fs.writeFile is being called. 

Also, I found this in Mozilla's Developer Network: 

If multiple identical EventListeners are registered on the same EventTarget with the same parameters, the duplicate instances are discarded. They do not cause the EventListener to be called twice, and since the duplicates are discarded, they do not need to be removed manually with the removeEventListener method.

It was an oversight on my part to set the event handler over and over, and I'll take it out, but I don't think it's the cause of the trouble. 

[Anatoly Pashin]

If it were, we would see the output from the console logs on lines 57 & 58

Sure, I see output from lines 57 and 58 one, then two and then three times. 

--
You received this message because you are subscribed to the Google Groups "node-webkit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-webkit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Пашин Анатолий,
эникейщик.

[Laszlo Z. Antal]

Hi,

On Apr 14, 2014, at 20:18, "Rick H." <cub...@gmail.com> wrote:

Hi, Anatoly. 

At first I thought "That's it!"  Then I realized the event handler isn't being called.  If it were, we would see the output from the console logs on lines 57 & 58.  Only the callback from the call to fs.writeFile is being called. 

Also, I found this in Mozilla's Developer Network: 

If multiple identical EventListeners are registered on the same EventTarget with the same parameters, the duplicate instances are discarded. They do not cause the EventListener to be called twice, and since the duplicates are discarded, they do not need to be removed manually with the removeEventListener method.

It was an oversight on my part to set the event handler over and over, and I'll take it out, but I don't think it's the cause of the trouble. 

To avoid all those event issues you could use this https://github.com/lzantal/LZADialog

I can't wrap my head around that encoding. Could you post a sample of your index.html to test?

Also try setting an encoding on the index.html document it self to utf-8 and see if that helps. That way nw will use that encoding on the entire content instead of auto picking one. 

Laszlo

http://twitter.com/LZAntal

On Monday, April 14, 2014 9:26:25 PM UTC-5, Anatoly Pashin wrote:

You just create new event listener on every click. This way you got it incrementing each time. You've to attach it once or create new <input> each click instead of  var inputSaveAsFile = document.querySelector(inputSaveAsFileId);

[Rick H.]

Aaaaaa!  You're right!  The output is just not synchronized.  The handler and the callback do not happen in sequence.  But the handler is getting called once, then twice, then three times. 

Excellent!  Thank you, Anatoly. 

[Anatoly Pashin]

You're welcome. Calling Roger Wang — will we ever see http://www.html5rocks.com/en/tutorials/developertools/async-call-stack/ in node-webkit?

[Rick H.]

Hi, Laszlo. 

I responded to your questions below. 

          Regards, Rick

On Monday, April 14, 2014 10:30:03 PM UTC-5, LZAntal wrote:

Hi,

 

To avoid all those event issues you could use this https://github.com/lzantal/LZADialog

That looks really good!

I can't wrap my head around that encoding. Could you post a sample of your index.html to test?

I assume you mean attach it as a file right here.  Well, I'm not proud of the file, but humility is a virtue, right? 

 

Also try setting an encoding on the index.html document it self to utf-8 and see if that helps. That way nw will use that encoding on the entire content instead of auto picking one. 

Do you mean this?

<html>
    <head> 

                <meta http-equiv="Content-Type" content="text/html; Charset=UTF-8" /> 

[Rick H.]

Hello, Laszlo. 

I have solved the multiple-handlers problem.  The remaining problem is the encoding, and getting the whole string written to disk. 

I added this to my <head> section, per your suggestion. 

        <meta charset="utf-8"/>

It didn't appear to affect the outcome. 

Something I don't understand that might be involved is that when I ask the file command to evaluate my html file, it reports an error. 

Rick@RicksLaptop /cygdrive/c/Abe/node-webkit/experiment

$ file index.html

index.html: ERROR: line 163: regex error 17, (illegal byte sequence)

I'm using Brackets to edit my files.  Could it write its files in some strange format? 

I could just use utf16le and call it good.  But I hate to got forward when I don't understand.  It might bite me later, in unexpected ways. 

1.  

Is the Raptor editor itself open source?  Is the code available for inspection? 

The dialogs are nice.  I am past the getting the file name from the dialogs now, and want to write my data to disk.  I suspect the way that your editor handles encoding will illuminate things for me. 

2.  

The only encoding that works is utf16le.  If I use utf8, or ascii, the whole file doesn't get written. 

That doesn't make sense to me.  It seems to me that once I have my data in a Javascript string, Javascript knows how it's encoded.  If fs.writeFile writes the data to disk using a particular encoding, it should appear on the disk in that encoding.  

Thanks for any ideas. 

              Rick

[LZAntal]

Hi,

On Apr 17, 2014, at 6:16 AM, Rick H. <cub...@gmail.com> wrote:

Hello, Laszlo. 

I have solved the multiple-handlers problem.  The remaining problem is the encoding, and getting the whole string written to disk. 

I added this to my <head> section, per your suggestion. 

        <meta charset="utf-8"/>

It didn't appear to affect the outcome. 

Are you using an editor like ace or codemirror or just plain textarea?

Something I don't understand that might be involved is that when I ask the file command to evaluate my html file, it reports an error. 

Rick@RicksLaptop /cygdrive/c/Abe/node-webkit/experiment

$ file index.html

index.html: ERROR: line 163: regex error 17, (illegal byte sequence)

I'm using Brackets to edit my files.  Could it write its files in some strange format? 

Oh ok so if I remember good brackets uses codemirror. That very well could be. Why don't you open your file in an other editor or create one from scratch.

As always I would recommend going super simple. Create an index.html file with nothing but a textarea in it. Then use the same dialog and file save functions to save the content of the textarea and see if everything worked as expected. 

I could just use utf16le and call it good.  But I hate to got forward when I don't understand.  It might bite me later, in unexpected ways. 

That would be my worry as well.

1.  

Is the Raptor editor itself open source?  Is the code available for inspection? 

The dialogs are nice.  I am past the getting the file name from the dialogs now, and want to write my data to disk.  I suspect the way that your editor handles encoding will illuminate things for me. 

RaptorGL Editor is not complete just yet. I'm hoping to release it with the game library by June(fingers crossed) I don't know about the exact license but it will be free ;)

Right now it uses ace editor for the code editing and everything else is written around it. So I let ace(or maybe codemirror in the future) handle the editing of the text but. But as you point out below when I grab the file contents and write it back or create new one I always set the encoding to utf-8. 

2.  

The only encoding that works is utf16le.  If I use utf8, or ascii, the whole file doesn't get written. 

That doesn't make sense to me.  It seems to me that once I have my data in a Javascript string, Javascript knows how it's encoded.  If fs.writeFile writes the data to disk using a particular encoding, it should appear on the disk in that encoding.  

Thanks for any ideas. 

              Rick

As I wrote above it's strange. Where the data comes from that you are wiring to file? Are you typing it in or is it from a file? Are you setting the encoding as well when you read the file in?

Laszlo

http://twitter.com/LZAntal

On Monday, April 14, 2014 10:49:02 PM UTC-5, Rick H. wrote:

Hi, Laszlo. 

I responded to your questions below. 

          Regards, Rick

On Monday, April 14, 2014 10:30:03 PM UTC-5, LZAntal wrote:

Hi,

 

To avoid all those event issues you could use this https://github.com/lzantal/LZADialog

That looks really good!

I can't wrap my head around that encoding. Could you post a sample of your index.html to test?

I assume you mean attach it as a file right here.  Well, I'm not proud of the file, but humility is a virtue, right? 

 

Also try setting an encoding on the index.html document it self to utf-8 and see if that helps. That way nw will use that encoding on the entire content instead of auto picking one. 

Do you mean this?

<html>
    <head> 

                <meta http-equiv="Content-Type" content="text/html; Charset=UTF-8" /> 

[Anatoly Pashin]

2014-04-18 6:45 GMT+11:00 LZAntal <lza...@gmail.com>:

$ file index.html

index.html: ERROR: line 163: regex error 17, (illegal byte sequence)

Very, very strange. Please make sure you save your file in UTF-8 without BOM. You can do this using Sublime text: File → Save with encoding. Or use your preferred editor if it can do this.

--
Пашин Анатолий,
эникейщик.

[Rick H.]

Hello, Laszlo and Anatoly. 

Okay.  Apparently version 5.17 of Cygwin "file" utility has a problem.  http://cygwin.com/ml/cygwin/2014-03/msg00266.html  

I installed version 5.18, and the mysterious error message went away.  Whew!  One problem down. 

Laszlo, I am writing from a div with the contenteditable attribute.  I attached my index.html file earlier in this thread. 

I have to go to bed now, but on the weekend I'm move that code to a straight node environment, and simplify it.  That should make the cause clear. 

I'll let you know what I find out. 

               Regards, Rick

[Anatoly Pashin]

What file utility outputs now? Output should end with `UTF-8 Unicode text` or `ASCII text`

--

You received this message because you are subscribed to the Google Groups "node-webkit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-webkit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Пашин Анатолий,
эникейщик.

[Rick H.]

"file" output looks normal now: 

Rick@RicksLaptop /cygdrive/c/Abe/node-webkit/experiment

$ file index.html

index.html: HTML document, ASCII text

I'll still set up a simplified situation in node, not node-webkit, tonight or tomorrow.  Off to my day job!

[Anatoly Pashin]

OK, check it. Looks like you'd better show all your sources so we all can test this very very strange case.

18 апр. 2014 г. 23:51 пользователь "Rick H." <cub...@gmail.com> написал:

[radu....@gmail.com]

I've encounter the same problem with partial saving. When I've changed from node-webkit 0.9.2 to 0.8.5 all was ok. So my guess is that in 0.9.2 is a problem.

[radu....@gmail.com]

Unfortunately 0.8.6 truncate the files at save, as 0.9.2. With 0.8.5 I had no problem.

So fs.writeFile seems not to work with latest releases. I can send files for testing if needed.

[Rick H.]

Radu, Anatoly, Laszlo, 

I have created an issue on github in the node-webkit project. 

https://github.com/rogerwang/node-webkit/issues/1794

I guess I'll use 0.8.5, and let the maintainers do their work. 

Wow!  I had a bad version of Cygwin's file utility, and then I had a bad version of node-webkit.  Maybe karma is trying to tell me I shouldn't do this project. 

Thank you guys for your help.  You are awesome! 

[radu....@gmail.com]

Oh, here is the solution for Windows.

text='\ufeff'+text

This will force the UTF8 encoding. More tests needs to be done, but it seems this trick will make everything ok even for 0.9.2 and 0.8.6

http://stackoverflow.com/questions/13859218/nodejs-how-to-make-function-fs-writefile-write-with-bom

[radu....@gmail.com]

The following code will make UTF8 without BOM to be sure. At least that's what I use. I repeat: applying this on the data before writing on the disk will write all the content, with 0.9.2 and 0.8.6 also.

 

function utfTrick(text){

text=(0xFEFF==text.charCodeAt(0)?text.substring(1):text);

text='\ufeff'+text;

text=(0xFEFF==text.charCodeAt(0)?text.substring(1):text);

return text;

}

fs.writeFile(file,utfTrick(data),{encoding: "utf8"}); 

[Rick H.]

Thanks for the pointer, Radu. 

I did the searches grep   -r -l -i feff *   and   grep -r -l -i bom *   in my copy of the node-webkit source, and didn't find anything interesting.  I was trying to find the place where the code had an error.  Or where the byte order mark was used.  

           Rick