Lost work two times with tiddlywiki node version

921 views
Skip to first unread message

Bruno Loff

unread,
Dec 8, 2016, 5:53:43 AM12/8/16
to TiddlyWiki
I have been using tiddlywiki for taking notes for my work for the last couple of months. I was fascinated with the extensibility of it, and have spent many hours customizing it to suit my purpose.

But in this week I mysteriously lost my work two times while running tiddlywiki on node. That's over fourteen hours of work that I now have to redo, hopefully I'll remember everything. I'm pretty pissed off.

If you don't pay attention to something as fundamental as "saving", your whole software is standing on a shaky foundation.

Instead of recommending the use of tiddlywiki for someone writing a novel (god spare them), you should put a disclaimer in your webpage.

I guess that as a piece of software, it just isn't reliable enough for serious work. After burning myself in this way I will be very hesitant in using it for anything else ever again.

I thought I would leave the criticism here, in case the developer is willing to learn from it.

Jeremy Ruston

unread,
Dec 8, 2016, 6:02:11 AM12/8/16
to tiddl...@googlegroups.com
Hi Bruno

I’m sorry to hear of your experience; like most of us I know how frustrating it is to lose your own work.

That’s why tiddlywiki.com does go to some trouble to highlight the “First rule of using TiddlyWiki”, to take cautious backups and guard against data loss. As much as possible, we try to learn from feedback so that we can engineer improvements to the product to reduce the likelihood of data loss. But ultimately data security requires attention from the user; TiddlyWiki gives enough power to shoot oneself in the foot very easily.

Can you shed any light on what went wrong for you? It would be great to understand any underlying issues so that we can attempt to address them.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/31bfe421-f4ac-49a4-a4ba-b7e04ace8357%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arlen Beiler

unread,
Dec 8, 2016, 8:42:55 AM12/8/16
to TiddlyWiki
I'm sorry about all that lost work. Man, that's frustrating!

First think to check is whether there are any tiddlers which have
names that start with Draft. If there are, open them and see if any of
them contain the lost work.

Would you be able to give any details about what you were doing at the
time? Did you leave it open in your browser overnight? (Probably not a
problem, but could be a corner case that we need to check.) Were there
other interesting pieces of information that we might be able to use
that could point us toward any possible bug or situation we haven't
thought of?

There is a save button, which is a checkmark in a circle, which turns
red if there are unsaved changes. I guess all the changes were saved?
It should have warned you, though, if you try to close the window
without all the changes being saved.

In TiddlyWiki, when you click the settings gear icon, a tiddler comes
up. One of the first things that it says there is Tiddlywiki Version.
What is that version number? Also, what browser do you normally use to
access it.

Please understand, I can't access your computer, so if you could help
me to know a few things I would be very grateful. The reason behind
these questions is that I am trying to figure out if there is a
certain platform or browser that we haven't accounted for and is
causing TiddlyWiki to behave in unpredictable ways.

Also, I don't know if this helps or not, but it is possible that some
of your files are still in the data folder. That's the "current
directory" folder or the one specified as the first command line
argument. Again, not sure if that helps, but just wanted to mention it
in case.

Sorry for your trouble, hopefully something here helps. But if not, I
wish you the best.
God bless,
-Arlen
> https://groups.google.com/d/msgid/tiddlywiki/3CFE4E0A-04A0-4A16-906F-5E6AE83C2C1C%40gmail.com.

Hiru Yoru

unread,
Dec 8, 2016, 10:06:37 AM12/8/16
to TiddlyWiki
Hi,

That sucks. I've been there. Have you tried the Classic version? I personally prefer that one and it has options that you can set to auto-save any changes and to create backups. It is located here: http://classic.tiddlywiki.com -- There are also a lot of plugins and themes already out there because it's been around for a while.

You can set the auto-save and save-backups settings on the "options »" panel on the right. Click it and it will show you a panel, then check the boxes. To specify a backup folder, you'd hover your mouse over the top right-hand corner of the screen and it will show you a "backstage" button. Click it and it will open a bar. Click on "tweak" and then at the top of the table, you can see the backups folder name as the third option down. You can also access the aforementioned auto-save and save-backups options here.

It's very hard to lose work when using these methods because a copy of your wiki is saved every time you click "done" when editing a tiddler. It's also saved whenever you manually click "save." If your current wiki somehow went wrong, you can even go back to the backups and check them for previous information. You can open them just as you would your main wiki file.

There are themes for it here: http://themes.tiddlywiki.com/
Plugins/extensions are here: http://tiddlyvault.tiddlyspot.com/

I hope this helps. I'm yet to lose anything with this method, and have great trust in it.

Regards,
Hiru

Bruno Loff

unread,
Dec 12, 2016, 5:56:34 AM12/12/16
to TiddlyWiki
Sorry for the angry vent, that was uncalled for. I was really upset.

The bug, wherever it lies, is hard to reproduce.

I am running a node.js instance, and accessing it via the browser (actually via a custom-made electron app which I had eventually planned to release publicly).

When I usually write a draft and close the app, the draft will still be there when I return, provided that I wait enough for the "save" button to turn from red to gray. So I became used to this feature. It seemed very clever since TW will autosave very often while in the middle of editing, so I had a false sense of security: "even if the latest edit I've made is lost, that's at most one or two sentences, so no big deal".

But at some point, maybe the application was closed in some abnormal way, maybe it was killed, maybe it was closed before the red "save" button turned back to gray, something like that, I couldn't reproduce the problem, so I'm not sure when it occurs. Upon opening the wiki again, the draft tiddler still exists, but it is no longer in "editing" mode when I return, and, worst of all, its contents are completely empty. So I didn't just lose the last few sentences I had written since the last successful autosave, I lost all the changes that I had made since last clicking the "confirm changes to this tiddler" button. When searching through the .tid files in the wikifolder, indeed the "Draft of ... .tid" file was there, but it was also empty.

The behavior I expected was that, even if the changes I last made were not properly saved, the draft should be at whichever state it was since the last autosave happened. So for example, when the Sync machinery is working to auto-save the contents of a draft, it should ensure that the operations are "atomic", in some way, i.e., that the promise holds that it should either overwrite the draft with the new edits, or preserve the draft's contents since the last time it autosaved. *Never* should it happen that the entire draft just disappears.

The feeling of insecurity I get from having this basic assumption broken makes me very uncomfortable in using TW for work, as if the ground beneath my feet wasn't solid.

Respectfully, Jeremy, your statement that TW gives enough power to shoot oneself in the foot is not convincing, because while I do see that TW is extremely powerful and versatile, I honestly don't think that a reliable saving mechanism gets in the way of that.

The reason I chose TW in the first place was out of admiration for its highly self-reflective design. That admiration has not abated in the slightest.

Yours truly,
Bruno

PMario

unread,
Dec 12, 2016, 10:59:51 AM12/12/16
to tiddl...@googlegroups.com
Hi Bruno,

Some questions:

 - Which OS do you use on your client-browser? OS and Version?
    - electron version?
 - Is there a network connection in between, or is it localhost?
    - Where does the node.js server run? OS, V?
    - node.js version?
 - tiddlywiki version?
    - installed with npm or
    - via github?

that's it for the moment.

-m

Mark S.

unread,
Dec 12, 2016, 11:10:22 AM12/12/16
to TiddlyWiki
You still haven't told us anything about your set-up. What browser are you using? What operating system? Is node.js running on the same machine or a different machine than the machine where you are working?

You say that you've lost 14 hours worth of work. But from your description, it sounds like only the current draft tiddler was lost. That would suggest that you're putting massive amounts of writing into a single tiddler.  We have been told that using a single tiddler for a large document is not "the tiddlywiki way." I would worry that a large tiddler might get dropped by the node.js server, which I don't think was really ever meant for intensive server loading.

Mark

Bruno Loff

unread,
Dec 12, 2016, 12:13:45 PM12/12/16
to TiddlyWiki
Tiddlywiki was running on electron v1.4 on Arch Linux; the server side was running on electron's server (a node.js process), and the webpage was running on electron's chromium process. So all of it was localhost.

I was working on a slightly modified fork (https://github.com/bloff/TiddlyWiki5), that included a minimal change to detect when TW was running under electron.

It also included various plugins that I wrote/modified to serve my purposes (customized katex plugin, a plugin for managing bibtex citations, and a plugin to interface with electron), which is roughly keeping track of my mathematical writeups, lecture-notes, etc. but I don't think these were to blame.

14 of work amounted to roughly 3-5 pages. That's not much.

Mark S.

unread,
Dec 12, 2016, 1:46:54 PM12/12/16
to TiddlyWiki
But, was all that work -- 3 to 5 pages -- in a single tiddler?

Mark

Bruno Loff

unread,
Dec 12, 2016, 2:24:55 PM12/12/16
to TiddlyWiki
Yes it was.

Arlen Beiler

unread,
Dec 22, 2016, 7:30:15 AM12/22/16
to TiddlyWiki
Nevertheless, this is a bug. TiddlyWiki should never lose changes no matter how big the Tiddler is. Thanks for the info, though. I hope we can find any problems that are causing it. 

I have frankly never just used the draft idea before, I always save the Tiddler when I am done writing. And I would recommend doing that normally. However, it seems you are supposed to be able to save a draft and come back to it later.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscribe@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

Mark S.

unread,
Dec 22, 2016, 9:51:05 AM12/22/16
to TiddlyWiki
Not necessarily a bug in TW per se. No one ever promised (that I know of) that it would work on Electron. Definitely no one promised that it would work with a custom version of Electron and if it's on a custom branch of TW5 that depends on sensing its platform  --- all bets are off.

Just depending on small desktop web/file  servers (like node.js) has always been risky in my experience. The hardware and the software may not be robust enough in all situations. Usually, there are time lags. Your machine may prioritize your email or web browsing over the web server.  If you shut down your machine or server before a save has completed (which might happen if you have a 3 or 5 page tiddler) then the chances are even greater. The trick in that situation would be to either save your own copy of current work locally or to break the work into smaller chunks. The editing tools in TW5 make working with smaller chunks easier than before.

 If you check the forum, you will see that reports of actual data loss are fairly rare. The file-based TW seems to save itself reliably.  To be an actual bug you would need to see a repeatable set of standard circumstances under which TW fails to save.

Have fun,
Mark



On Thursday, December 22, 2016 at 4:30:15 AM UTC-8, Arlen Beiler wrote:
Nevertheless, this is a bug. TiddlyWiki should never lose changes no matter how big the Tiddler is. Thanks for the info, though. I hope we can find any problems that are causing it. 

I have frankly never just used the draft idea before, I always save the Tiddler when I am done writing. And I would recommend doing that normally. However, it seems you are supposed to be able to save a draft and come back to it later.
On Dec 12, 2016 14:24, "Bruno Loff" <bruno...@gmail.com> wrote:
Yes it was.

On Monday, 12 December 2016 19:46:54 UTC+1, Mark S. wrote:
But, was all that work -- 3 to 5 pages -- in a single tiddler?

Mark

On Monday, December 12, 2016 at 9:13:45 AM UTC-8, Bruno Loff wrote:
Tiddlywiki was running on electron v1.4 on Arch Linux; the server side was running on electron's server (a node.js process), and the webpage was running on electron's chromium process. So all of it was localhost.

I was working on a slightly modified fork (https://github.com/bloff/TiddlyWiki5), that included a minimal change to detect when TW was running under electron.

It also included various plugins that I wrote/modified to serve my purposes (customized katex plugin, a plugin for managing bibtex citations, and a plugin to interface with electron), which is roughly keeping track of my mathematical writeups, lecture-notes, etc. but I don't think these were to blame.

14 of work amounted to roughly 3-5 pages. That's not much.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

Arlen Beiler

unread,
Dec 23, 2016, 4:17:50 PM12/23/16
to TiddlyWiki
I just had an idea. The way the NodeJS server currently works is asynchronous. It syncs with the server and then returns to the browser. Then the server syncs to the file system and returns. If there would have been a time where you closed your draft, waited for the check mark to turn grey, and then immediately exited, it is possible that the server could have missed writing the files. Try to see if you can replicate it like this. If so, that may be the problem. 

The way I dealt with that was to hack tiddlywiki (forget how, but I think involved some of the first code in boot.js or bootprefix.js) so that it would save directly to the file system from the browser. 


For electron you would set the data directory in index.html, then open index.html in a new BrowserWindow. And your done. The file syncer actually takes care of saving changes, so it should stay red until the file actually gets saved.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscribe@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

Dmitry Sokolov

unread,
Dec 24, 2016, 8:47:05 PM12/24/16
to tiddl...@googlegroups.com
I see dropping modified tiddler as a bug.

I am intensively working with PBWorks at the moment (while preparing transfer to TW in future).
PBWorks gives you all indicators of the processes taking place. I know, for example, that saving is finished when a URL with no "edit" appeared. Then, it's safe to close the window.
Another protection measures they have is a warning dialogue on closing page being edited, before the save button pressed.

I think, we have to learn best practices from other platforms, collect them within just one page/workspace, and implement as soon as practical.
Sorry for being persistent with the simple idea:
our team performance depends on how quick particular topics are found/discovered for reuse.
Please let me know if this mechanism is already realised, and that's just me who can't see it.

Thank you,
Dmitry

Arlen Beiler

unread,
Dec 26, 2016, 7:00:25 PM12/26/16
to TiddlyWiki
It may be a bug, and we may need to look into it.

See my link in the last post, or quoted below, showing how to use NodeJS data folders with Electron so it saves correctly. 

Please do take all reasonable precautions in using this as well, and I can't warrent that it is perfect, but it's for your reference.

Enjoy :)
-Arlen

On Dec 24, 2016 20:47, "Dmitry Sokolov" <dmitry.v...@gmail.com> wrote:
I see dropping modified tiddler as a bug.

I am intensively working with PBWorks at the moment (while preparing transfer to TW in future).
PBWorks gives you all indicators of the processes taking place. I know, for example, that saving finished when a URL with no "edit" appeared. Then, it's safe to close the window.

Another protection measures they have is a warning dialogue on closing page being edited, before the save button pressed.

I think, we have to learn best practices from other platforms, collect them within just one platform, and implement as soon as practical.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscribe@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

Arlen Beiler

unread,
Dec 26, 2016, 9:03:45 PM12/26/16
to TiddlyWiki
Ok, I added the electron.js file that I pass to electron as the first argument.



Russell Cosway

unread,
Jan 17, 2017, 9:27:24 AM1/17/17
to TiddlyWiki
I have started to suffer this now.  I am on a MacMini running MacOS Sierra 10.12.2.  I use Firefox 50.1.0 and autosave is enabled; I only used Firefox for TW work.  TW version is 5.1.13 and is on a node.js hosted on AWS - I didn't set that up but could get the server setup details is needed.

I have had no browser crashes and I can see no recreatable pattern of circumstances.  I have literally watched the text of the tiddler disappear in front of my eyes.  Due to the inter-relationships i'm working on i may have 10 tiddlers open at once, maybe 2-5 of those in draft.  However, the loss occurs when I have saved and the TW tick has gone from red to grey - soon after saving a tiddler, either that tiddler or another I have open clears.  There are no "drafts" of the lost tiddler left.  The tags and last saved information remains intact, just the text is cleared.  The json export shows it as "".

I too have now lost hours of work and considering my options to produce my wiki, so any insights and improvements would welcomed.
R
-Arlen

Arlen Beiler

unread,
Jan 18, 2017, 6:48:20 PM1/18/17
to TiddlyWiki
It sounds like server details would be good. This sounds like it could be related to the sync mechanism somehow, but we would need to know how the server is setup.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscribe@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

Chuck R.

unread,
Jan 19, 2017, 10:57:43 AM1/19/17
to TiddlyWiki
Tip: On linux create a cron job to backup your file every n minutes.
To edit YOUR user cron file: crontab -e

To get the file format google: crontab file format

Chuck R.

unread,
Jan 19, 2017, 11:01:58 AM1/19/17
to TiddlyWiki
Append the date and time to the backup file so the backup file is named: tw5.html.20170119-1005.

Arlen Beiler

unread,
Jan 19, 2017, 11:05:39 PM1/19/17
to TiddlyWiki
Also any logs from the server NodeJS would be helpful. Would more than one person be editing at the same time?

On Jan 17, 2017 09:27, "Russell Cosway" <russellc...@gmail.com> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscribe@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

Brian McMullan

unread,
Jan 29, 2017, 12:46:36 PM1/29/17
to TiddlyWiki
First post, sorry for the intrusion in existing thread.  Using TW for about four years, love it, moreover DEPEND on it.  Jeremy you are a mad genius is all I can say.  But, I am experiencing significant "crashes" since 5.1.13.  Even using an empty_5.1.13 shell, navigating around and editing a tiddler or two and IE 11 will go "internet explorer has stopped responding" and all changes lost.  I can use Chrome and prefer it, but the save mechanism in IE is much better with the (whatever the name is IE saver thing from tiddlywiki site), I lose data it seems in Chrome as the save directory takes a notion to wander to downloads, or desktop, or last place I saved a PDF (ornery thing).  Empty shell of 5.1.12 and prior works fine, pre-release (just testing) 5.1.14 works just fine, but there's something with 5.1.13 that definitely different in a bad sort of way.

My best guess is the WYSIWYG editor since 99% of the time it dies when editing a tiddler and extremely seldom when just navigating about.  The savior was TiddlyDesktop, hadn't used it previously but had to find something to work around the issue.  5.1.13 has never, ever, crashed in it.  I prefer the standalone browser version though, less steps to fire up quickly, make a few notes and entries, save and shut back down.  

Config: Dropbox [c users xxxx dropbox tiddly] as repository, updates and edits (not simultaneously of course) from multiple computers (couple home machines, shop machine, office machines, test dev machine),  It started on ALL machines similarly after updating to 5.1.13.  I should have rolled back to 5.1.12 right after update, but loving the editor I kept updating and adding and boxed myself in.  Maybe can "downgrade" to 5.1.12 but never done that so kept forging ahead.  machines: Win 7 x64 pro and enterprise mainly, and one Win 10 pro [I don't trust Win 10 and it's telemetry data gathering propensity, hate to use it but have to for one business case - nuff said].

Just sharing my two cents, I check the releases weekly anxiously awaiting 5.1.14 so can get back to my normal use pattern.  Amazing thing Tiddly wiki so I hate to post a negative experience.

Tobias Beer

unread,
Jan 29, 2017, 2:15:18 PM1/29/17
to TiddlyWiki
Hi Brian,

Do you experience the same issues in Chrome or Firefox?

No matter what version of TiddlyWiki, Internet Explorer has never been a well supported browser.

Personally, I'm very much for not even trying.

Best wishes,

Tobias.

BJ

unread,
Jan 29, 2017, 2:31:59 PM1/29/17
to TiddlyWiki
Hi Brian,
better to start a new thread so that people with experience of using IE can respond.

all the best
BJ

Brian McMullan

unread,
Jan 29, 2017, 10:08:35 PM1/29/17
to TiddlyWiki

I don't use Firefox; most of the corps [I work with at least] disallow firefox, no sense falling in love with something at home that I cannot reliably use elsewhere.

Chrome yes, and no crashes in 5.1.13, but operationally its very subpar to how things work in IE.   Saving: "Thou cannot savest where thy wanteth without a miracle (aka manual intervention) each save".  Chrome also doesn't play nice with ext links that open local or network folders, it opens them, but in it's browser view vs kicking off Windows Explorer like IE does (and I desire it to operate the IE way).    It's considerably better to use TiddlyDesktop and have a three click and 6 second startup penalty but avoid the multitude of "save penalties" and op ugliness behavior.    

No disagreements that IE is in many ways inferior to Chrome, but IE does have killer app - the Tiddly IE add-on that enforces tiddlers be saved where they're supposed to with a single click.  When TiddlyWiki runs reliably in IE, it's the best of all worlds which is why I settled on it.  Seriously, four years of using IE 11 across many TW5 versions and no problem.  But, 5.1.13 and problem exists.  5.1.14 beta, have yet to have a crash, few kinks still being worked out though of course.  

Thanks for the feedback and sorry for posting in the wrong spot, I'll carry on with TiddlyDesktop until 5.1.14 is released and hope for the best.
Reply all
Reply to author
Forward
0 new messages