Lost two weeks of work - what I did wrong and what you can do to avoid it

[Andy Redfern]

Folks,

Started using TiddlyWiki around four weeks ago. I love the friendliness of the community and the product itself. As a long time developer, the ability to tweak and tinker is very attractive. I can create the work flow I want.

Started running in a single file mode, added in using TiddlyDesktop and then in an effort to make it work with my phone too I switched to the NodeJS. Read a couple of the guides to using NodeJS and TiddlyWiki, and I was off.

Then on Friday night disaster struck and I lost 2 weeks of data – everything since I had switched to NodeJS. So after a day sobbing quietly in the corner this is my effort to figure out where I went wrong with my set up so you too don’t make the mistake I do.

So following this guide
https://www.npmjs.com/package/tiddlywiki

and everything worked as expected apart from
tiddlywiki mynewwiki --init server

which errored:
Error: Error creating directory 'testlife'

Doh! Permissions I thought. So added in Sudo
sudo tiddlywiki mynewwiki --init server

Created fine and started
tiddlywiki mynewwiki --init server

However it errored too:
Sync error while processing save of '$:/StoryList': Error: ENOENT: no such file or directory, open '/var/www/html/..fullpath../tiddlers/$__StoryList.tid'

Ah tiddlers folder not created – manually created and I was off.

All worked as expected except I get having to start tiddlywiki every time my Putty session ended. Not what I wanted.
Read around nodejs advice sites and people suggest using forever.
So:
forever start --spinSleepTime 10000 /usr/local/bin/tiddlywiki /var/www/html/..fullpath../mynewwiki --listen host=0.0.0.0 username=**** password=****

Everything works as I am expecting.

And yet I have made an error – I created the Tiddler folder using Root as I used sudo – and the new folder doesn’t have write permissions for the process running in with forever. Without forever it has all the permissions it needs, but with forever it doesn’t.

So everything looked fine – but actually Tiddlywiki was syncing to the server fine but all my data was in memory. So when after 2 weeks I stopped the process to make some tweaks ready for a proposed PR on the core code, I lost everything. <FX: SOUND OF HEAD BANGING ON TABLE>


So in summary - What should I have done differently to avoid the permissions based errors in the first place?

Tiddlywiki --init failed – so I used sudo – what should I have done?

Tiddlywiki --init failed to create tiddlers sub folder – permissions?

Forever runs as the correct user but without the permissions. I solved the problem by giving the folder 777 full write permissions, but what should I have done?

Final thought. The permissions error in writing data to the server drive surfaced no errors in TW – why? Shouldn’t it have alerted me that no data was being saved. The only place I could find the error was in the difficult to interpret forever logs which I was scouring to figure where I had gone wrong.

Anyway, sorry for the long read but I hope my experience can help others avoid the same issue and potentially see TW amended to report when it can’t save tiddlers to the server drive.

Andy

[TonyM]

ANdy,

Thanks for sharing this experience, lets hope it saves some one a lot of time. No need for to state the first rule of tiddlywiki (backup), especially since it is so simple for single files. Your transition to the server solutions changes this somewhat.

Yours in sympathy, Best Wishes

Tony

[Andy Redfern]

Thanks Tony.

Of course it does highlight a fundamental issue with back ups - the directory was being backed up everyday on normal server back up process - but the data had never been written to the drive, so the back up was unchanged for that folder.

Perhaps I was unlucky but I was surprised that Tiddlywiki report nothing of the multiple failed tiddler saves everyday.

Andy

[Mark S.]

What's missing from your report is your operating system. Linux? Which flavor? Mac? Or ??

I run node and node-like products on xubuntu, and have never had to run as root. But maybe other systems are fussier?

[Andy Redfern]

Good point Mark.

I am currently running Ubuntu 16.04.5 LTS on a Linode instance.

It is entirely possible that I have misconfigured something else which was a contributing factor.

I am normally a LAMP developer and this is the first time I have spun up NodeJS.

Andy

[Arlen Beiler]

I'm sorry to hear this. As I deal with data folders a lot, I'm investigating this and will open a PR shortly.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/af0978d5-310a-4094-b800-78e4267bace1%40googlegroups.com.

[Arlen Beiler]

It seemed strange to me that there would be no logging for that error, and that would definitely have been a bug, however in trying to reproduce this, I am getting an error message.

Sync error while processing save of '$:/StoryList': Error: EACCES: permission denied, open '/Users/arlen/Desktop/GitHub/TiddlyWiki5-Arlen22/data/tiddlers/$__StoryList.tid'

And it comes every second. 

If the directory cannot be created, there is no error, but then of course the tiddler fails to save. 

Can you verify that you were running tiddlywiki version 5.1.21 or 5.1.22?

[Arlen Beiler]

Do you see any messages like this?

 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList or any other tiddler title

[Andy Redfern]

Hi Arlen,

This was on 5.1.22

The issue was that running directly:

tiddlywiki mynewwiki --listen

had no issues. It was saving tiddlers without issue.

It was when I instantiated using forever that the permissions were wrong.

The error message you show was present in the forever logs, but I only saw that once I had lost the data and begun the investigation.

Andy

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[Arlen Beiler]

This is what my console looks like after a very little bit of editing. Your log file should have similar lines.

[Andy Redfern]

Correct.

Having given the tiddler folder 777 permissions even when running with forever I see;

 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList

 syncer-server-filesystem: Dispatching 'save' task: $:/_sq/Stories/Story2HistoryList

 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList

 syncer-server-filesystem: Dispatching 'save' task: Untitled

 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList

Andy

On Saturday, May 9, 2020 at 11:10:05 PM UTC+1, Arlen Beiler wrote:

This is what my console looks like after a very little bit of editing. Your log file should have similar lines.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[Arlen Beiler]

Ok, I understand now. That makes sense. I'm sorry to hear about this. 

There is currently no indication in the browser if the server is having trouble saving. That's not necessarily for me to decide, but if there are others here that would be interested in that please comment here with your suggestions. I can see where there could be value in that. 

By the way, I haven't used forever, but I recommend using PM2, which has an epic management interface and you can tell it to install itself as a service. Just make sure you run `pm2 save` once you have everything configured the way you want. It's a little more complex, but if you're already used to working with servers, it should be fine. It also has pretty epic logs.

Arlen

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/d616fddc-2c8e-43d8-84fb-338acc809b0d%40googlegroups.com.

[Andy Redfern]

Cheers Arlen.

I'll check out PM2. I guess it would have been helpful if on of the Tiddlywiki NodeJS server docs (https://www.npmjs.com/package/tiddlywiki) had pointed to PM2 (or even forever as it does work when used correctly).

Perhaps once I have found my way around the world of TW, I will contribute to the docs too.

Once again thanks for reviewing the issue.

Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/d616fddc-2c8e-43d8-84fb-338acc809b0d%40googlegroups.com.

[Mark S.]

If you run without "forever" does it run OK? I just launch from a command line or an I3WM menu.

[Andy Redfern]

So the issue as I see it was.

Logged in as my username (andyr) on my server then:

tiddlywiki testwiki --init server

generates the error:

Error: Error creating directory 'testwiki'

So I then used sudo to run as root:

sudo tiddlywiki testwiki --init server

This then works as it has the permissions it needs.

Then when I run the listen:

tiddlywiki testwiki --listen

it doesn't have the permission to create tiddler file as testwiki folder is owned by root, so it errors with:

Sync error while processing save of '$:/StoryList': Error: ENOENT: no such file or directory, open '/var/www/html/locationdata.uk/public_html/testwiki/tiddlers/$__StoryList.tid'

So I manually created the tiddlers sub folder (as andyr) and re-ran:

tiddlywiki testwiki --listen

It works fine and save the tiddlers as it goes because tiddlywiki is running as andyr.

However, when I run it with forever it runs as a different user and so doesn't have access to the tiddlers folder as that was set to write only for andyr. The errors are getting sent to obscure log file which I didn't find until I realised my mistake.

So my personal view is that TW cant legislate for the user and permissions structure that has been set up on a particular Linux instance.

However I think two things would help:

* an example in the docs of using nodejs running as a service so that it is always available - for example in this thread PM2 has been suggested as a better solution than forever

* a sync error visible in the TW client if the server is unable to save. This could occur for a variety of reasons - permissions (as in my case), lack of server disk space, server disk write error. Knowing that your data was unsaved you could then take action to resolve the issue after first securing your data

I fully appreciate that the exact nature of "why" there was an error was my fault, but having a little fault tolerance so server disk failures are available to the client side.

Andy

PS I'm not running Linux as a desktop environment so I'm not sure that I3WMsi relevant or am I missing something?