My Tiddlywiki Backup Scheme

[Mal]

Folks,

I am interested in comments or suggestions on my backup scheme.

I've created a tiddlywiki as a note-keeping system for a medical student.  It currently contains over 1000 tiddlers plus about 200 embedded images.  This was originally created using twc and I have recently converted it to tw5 to get the benefits of embedded images.  Although the file is now a bit over 20MB, it can be viewed on all devices, including an iPad.  (I could not get the twc image links to work on the iPad.)

Although, the file is stored in Dropbox and synchronised on various devices, I'm very keen to make sure that no information can be lost.  Given that the versioning system in Dropbox only keeps old versions for 30 days, there is always the prospect that an accidental deletion could go un-noticed for this time and then be lost on all devices.

To insure against this I wanted to implement a process to automatically save every version of every tiddler, so that it will be possible to go back in time and recover previous versions.

After trying various alternatives, I set up dropbox on our home server to keep another synchronised copy of the tiddlywiki and created an awk script that extracts each tiddler, appends the "modified" timestamp to the tiddler title to create a unique file name and saves it in a separate folder in Dropbox, as long as a file of this name does not already exist.  This folder will contain the original copy of every tiddler plus all subsequent versions.  I then wrote a bash script to monitor the tiddlywiki html file and run the awk backup script every time a new version the tiddlywiki is saved.

I know the process is not really bullet-proof, but it's working.

No doubt there are much better ways of implementing this, so I will be keen to hear any suggestions.

Thanks,

Mal

[Marcel Kaczala]

I using TW5 on node.js in my VPS and Dropbox for the Tiddlers folder. That simple.

[PMario]

Hi Mal,

Reading your post, your number one priority is, to "protect the content". And as a "side effect" you'd like to recreate "older" versions.

I think your usecase asks for a TW5 dev server and an optional VCS system. It seems you are ok to create some (probably tricky) "bash scripts" and you are in control of a server.

- TW5 comes with a "development server" that can be used in local networks.
   - So it would be possible to run this one on the server and directly edit content with TW5. This option will require a network connection.
   - modified tiddlers would be stored as single files.
   - Version controlling could be done with the system described below.

- The file watching mechanism, that you use can be reused to detect the single tiddler changes.
  - It can trigger a build process, that creates single fiel TWs and copy them to the DP sync folder.

-- Version controlling --

My "brainstorming" depends a little bit on some info, that you didn't provide.
 - how many _different_ users do modify / edit the "core" content ?
 - which OS does the server use?
 - which OSes do the "editors" use, if they are different to the server ?
 - Is some type of access management needed?

So as I know git a bit, I use it as an example. Othere VCSes may work too.

 - On your server you could install git.
   - This server is the "master of content".
   - If the "nodejs dev server" (from above) is active, it can be used to modify the content tiddlers.
   - If no dev server is runnung you can modify the tiddlers with any text editor
   - Your file watch script will do the rest

If you have more than one Editor

 - Editors, can have there own "local backup" with a TW5 installation, that is able to build a single file TW using a custom edition [1]
   - TW5 editions basically are "directories" that contain some configuration info and many content tiddlers.
 - The TW5 repo contains some scripts, (that can be optimized) to work with custom editions and build "single file TWs" [2]
 - The build script could build a file TW and store it in the DP sync folder.

So what would be the advantage of such a configuration.

- Every "Editor" has a local backup on there computer. So if the server breaks, everything could be rebuilt using those local backups.
- If your IT department is responsible for the server, I hope they do regular backups of the whole stuff, so everything you need should be backuped allready.
- You can still use dropbox for syncing your single file TWs. So DP is an additional backup
- Every Editor can _offline_ modify the source and it will be possible to do "real merges".
- A TW core update could be done with a pull from the original TW5 repo.

...

just some ideas, to explore
mario

[1] https://github.com/Jermolene/TiddlyWiki5/tree/master/editions
[2] https://github.com/Jermolene/TiddlyWiki5/blob/master/qbld.sh

[Mal]

I accidentally replied directly to Marcel rather than posting, so I am re-submitting my reply below.  Marcel also suggested Seafile as a dropbox alternative, so I will check it out.

Marcel,

I agree, this is a straight-forward approach.  However, I am still concerned about the lack of long-term version history, given that Dropbox only keeps old versions for 30 days.  I guess there is always the possibility of using the Dropbox Packrat add-in to get unlimited version retention, although it is a paid option and requires a paid version of Dropbox.

Regards,

Mal
 

[Mal]

Mario,

Thanks for the detailed response.  See my comments included below.

Regards,

Mal

On Thursday, 23 January 2014 00:58:19 UTC+10, PMario wrote:

Hi Mal,

Reading your post, your number one priority is, to "protect the content". And as a "side effect" you'd like to recreate "older" versions.

Correct - although some other important requirements are: Single TW5 file to allow easy viewing on iPad, phone, etc.; Off-line viewing and editing capability; automated backup without user interaction.

I think your usecase asks for a TW5 dev server and an optional VCS system. It seems you are ok to create some (probably tricky) "bash scripts" and you are in control of a server.

Note that my server is not open to the internet, other than SSH.

- TW5 comes with a "development server" that can be used in local networks.
   - So it would be possible to run this one on the server and directly edit content with TW5. This option will require a network connection.
   - modified tiddlers would be stored as single files.
   - Version controlling could be done with the system described below.

- The file watching mechanism, that you use can be reused to detect the single tiddler changes.
  - It can trigger a build process, that creates single fiel TWs and copy them to the DP sync folder.

-- Version controlling --

My "brainstorming" depends a little bit on some info, that you didn't provide.
 - how many _different_ users do modify / edit the "core" content ?

One user only.
 

 - which OS does the server use?

Ubuntu 12.04

 - which OSes do the "editors" use, if they are different to the server ?

Ubuntu and OS X.
 

 - Is some type of access management needed?

No - other than keeping it private.

So as I know git a bit, I use it as an example. Othere VCSes may work too.

 - On your server you could install git.
   - This server is the "master of content".
   - If the "nodejs dev server" (from above) is active, it can be used to modify the content tiddlers.
   - If no dev server is runnung you can modify the tiddlers with any text editor
   - Your file watch script will do the rest

If you have more than one Editor

 - Editors, can have there own "local backup" with a TW5 installation, that is able to build a single file TW using a custom edition [1]
   - TW5 editions basically are "directories" that contain some configuration info and many content tiddlers.
 - The TW5 repo contains some scripts, (that can be optimized) to work with custom editions and build "single file TWs" [2]
 - The build script could build a file TW and store it in the DP sync folder.

So what would be the advantage of such a configuration.

- Every "Editor" has a local backup on there computer. So if the server breaks, everything could be rebuilt using those local backups.
- If your IT department is responsible for the server, I hope they do regular backups of the whole stuff, so everything you need should be backuped allready.
- You can still use dropbox for syncing your single file TWs. So DP is an additional backup
- Every Editor can _offline_ modify the source and it will be possible to do "real merges".
- A TW core update could be done with a pull from the original TW5 repo.

I can see this type of process working quite well.  I could keep the tiddler files in Dropbox and run tiddlywiki under node.js on the Ubuntu and Mac machines to allow for editing.  The automated script on the server could then re-build the single TW5 file for access on mobile devices and run a version control commit process each time a change is made.  I guess editing on mobile devices would not be possible, but this is not a high priority.

Of course a good option would be to run tiddlywiki under node.js on moblie devices as well - I'm not sure if this will be an possible at some stage.

One piece that is missing is the tiddlywiki automated save functionality, but I understand it's on the near-term roadmap.

[Leo Staley]

giewiki (Poul's version which runs a modified tiddlywiki on Google apps ) automatically keeps a version history of every tiddler. I imagine a plugin for something like that could be created for TW5, stored locally.

I myself use SugarSync, which just automatically syncs with whatever is in any folder I designate, across multiple devices. It syncs with my local backups folder, and I'm fine if i have my local stuff backed up correctly. I access the tiddlywiki locally, and sugarsync detects the changed file and syncs with any changes automatically and quickly.

I use TWC, and I just found http://mptw.tiddlyspot.com/#LessBackupsPlugin. I wonder if something like this, if converted to TW5 would work at all for you?

I also wonder, would it be possible to create a plugin for TW5 that does what giewiki does?

Cheers,
Leo

[PMario]

On Thursday, January 23, 2014 5:28:29 AM UTC+1, Leo Staley wrote:

I use TWC, and I just found http://mptw.tiddlyspot.com/#LessBackupsPlugin. I wonder if something like this, if converted to TW5 would work at all for you?

IMO LessBackupsPlugin doesn't work anymore, due to increased browser security restrictions. ... If you use a latest browser version and the plugin works, I'd like to know about your settings.

-m

[Danielo Rodríguez]

I'm really interested in this kind of implementation but I don't understand how users can edit offline their own copy. I would like to carry my TW file in my pendrive for editing and consulting it while working and merge changes after. I would be glad if you explain that a bit further. I have a NAS server and I think I can implement this scenario on it.

Thanks un advance

[PMario]

On Thursday, January 23, 2014 4:42:22 PM UTC+1, Danielo Rodríguez wrote:

I'm really interested in this kind of implementation but I don't understand how users can edit offline their own copy.

Saving from a file TW back to an URL TW is blocked by the browser vendors, due to security concerns. They removed some browser specific functions, that where needed to access the computer file system (the way TW needs it) and if a html page is loaded from a file:// URL they blocked upload to http:// URLs ...

Removing these functions caused and is causing problems with TW. eg: Jeremy had to create the TiddlyFox addOn for Firefox, to be able to save TW to the disk.
With TW5, the TiddlyDesktop app uses a similar mechanism to save file TWs ...

I would like to carry my TW file in my pendrive for editing and consulting it while working and merge changes after.

Due to the restrictions above, there is no simple way to do this, even if you want to save from a file to your local network :/ The browser just doesn't allow it.

I would be glad if you explain that a bit further. I have a NAS server and I think I can implement this scenario on it.

Thanks un advance

I actually don't have an easy solution for syncing a single file TW back to an URL-TW.

... So I think the easiest way for a portable solution would be a portable system :) like the one Rikomagic offers. linux http://www.rikomagic.co.uk/linux_mini_pcs.html or there android versions ... http://www.rikomagic.co.uk/

---------------

It is planned, that future TW5 versions will have the possibility to use the brwoser local storage, to save "offline edited tiddlers". It will be possible to sync the local storage back to a server TW. ... but ... you'll need to install your browser on the stick and run it from there, otherwise your local storage won't be on the stick. ... So I think there will be enough problems with this setting too. ...

-mario

[Danielo Rodríguez]

Sorry I thin I misunderstood something because this:

Editors, can have there own "local backup" with a TW5 installation, that is able to build a single file TW using a custom edition [1]
- TW5 editions basically are "directories" that contain some configuration info and many content tiddlers.
- The TW5 repo contains some scripts, (that can be optimized) to work with custom editions and build "single file TWs" [2]
- The build script could build a file TW and store it in the DP sync folder.

So what would be the advantage of such a configuration.

- Every "Editor" has a local backup on there computer. So if the server breaks, everything could be rebuilt using those local backups.

If every editor has a local backup, why I can't have the same? Could you write a simplified scheme of this implementation?

And where can I find that server version you're talking about?

[PMario]

As I read your post, you want to have a single file TW on your stick, that can be opened with any browser. You can edit and save a file back to your stick, even if you opened it on a 3rd party computer. right?

I was talking/writing about a nodejs driven server version. So you'd need to have the TW5 source ropo on your stick. Which isn't a problem. The problem is, if you want to run the nodejs server on a computer, that is not your own. -> I doubt, they let you install the required programms there :)

 

And where can I find that server version you're talking about?

TW5 contains a "serve.sh" command, that starts a local nodejs server, that can be accessed with any browser at http://localhost:8080
It serves the "editions/clientserver" directory. If you edit a tiddler, it is saved back as a single file. The downside of this possibility is, that you need to install the nodejs stack on every pc, that should be able to run it.

------

In my brainstorming I was thinking about a git based "deploy" system, that works with the source tiddlers.
File TWs are created on demand, if the source was changed.

eg:

the server hosts the main repo
the server is configured to allow ssh connection to a git daemon
every editor can do a git clone to get the source from this server
the editor can locally run nodejs server with "serve.sh" and edit the content
... git commit
... git push to the server

since git server has hooks, that can identify a push it can run scripts, that build a new file version
the new file version can be saved at the dropbox sync directory ....

--> This "brainstorming" is centered around a deployment system using a version control system.

-m

[Danielo Rodríguez]

So your idea is, mainly based in a git server/git clients and the node JS implementation of tiddlywiky. I suppose the node part is just for creating individual files for an easier git control.
I don't like using different browsers, that's why I carry node webkit and tiddlyDesktop for a consistent experience. Is node webkit able to run a node implementation of tiddlywiky?

Thank you for your detailed explanation.

[PMario]

On Friday, January 24, 2014 12:43:01 AM UTC+1, Danielo Rodríguez wrote:

So your idea is, mainly based in a git server/git clients and the node JS implementation of tiddlywiky. I suppose the node part is just for creating individual files for an easier git control.

 

kind of
 

I don't like using different browsers, that's why I carry node webkit and tiddlyDesktop for a consistent experience. Is node webkit able to run a node implementation of tiddlywiky?

If I did understand Jeremy right, he's working on it. http://www.youtube.com/watch?feature=player_detailpage&v=I7vzfWieF_4#t=3066
But there still may be the problem to sync stuff back to the web. ... The whole setup needs to be secure ...

-mario

[Jeremy Ruston]

I think our best chance of success for syncing an offline TiddlyWiki file back up to an online service is to import the wiki file into the online service. Imagine opening yourspace.tiddlyspace.com, and then just dragging into that window your offline html file from your pendrive. The import process could popup a list showing all the tiddlers found in the file, with checkmarks against those that have a later modified date than an existing tiddler.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

--
Jeremy Ruston
mailto:jeremy...@gmail.com

[Danielo Rodríguez]

Sync back to the web will be difficult because limitations or security reasons? As far as I know, node webkit has the same privileges as a native application.

Jeremy, that approach sounds good, but I would prefer an automatic version. Is there any way to detect new Tiddlers within a single file implementation? Maybe I could try send new Tiddlers to my own server and merge then.

[Jeremy Ruston]

On Fri, Jan 24, 2014 at 5:54 PM, Danielo Rodríguez <rdan...@gmail.com> wrote:

Sync back to the web will be difficult because limitations or security reasons? As far as I know, node webkit has the same privileges as a native application.

The discussion has touched on both node-webkit and ordinary browsers. The restrictions apply to browsers, code in node-webkit can indeed freely talk back to a server.

Having said that, TiddlyDesktop currently runs TiddlyWiki files in a sandbox with much the same restrictions as a browser. This is done for the same reason that browsers do it: it would be undesirable for a buggy or malicious TiddlyWiki document to be let loose on your computer. Anyhow, we can arrange for privileged operations to be carried out via the same bridge that TiddlyFox uses.

 

Jeremy, that approach sounds good, but I would prefer an automatic version.

We'd all prefer something more convenient, the point is that browsers explicitly disallow the scenarios that would be most useful.

 

Is there any way to detect new Tiddlers within a single file implementation? Maybe I could try send new Tiddlers to my own server and merge then.

The issue is that browsers have shut down the ways that an HTML file loaded from a file:// URI can talk back to a server. That's why we have to workaround by initiating the sync from within the server, or use node-webkit.

Best wishes

Jeremy

 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

[Danielo Rodríguez]

Hi Jeremy,

El viernes, 24 de enero de 2014 18:59:47 UTC+1, Jeremy Ruston escribió:

The discussion has touched on both node-webkit and ordinary browsers. The restrictions apply to browsers, code in node-webkit can indeed freely talk back to a server.

Having said that, TiddlyDesktop currently runs TiddlyWiki files in a sandbox with much the same restrictions as a browser. This is done for the same reason that browsers do it: it would be undesirable for a buggy or malicious TiddlyWiki document to be let loose on your computer. Anyhow, we can arrange for privileged operations to be carried out via the same bridge that TiddlyFox uses.

 

 I never mentioned the ordinary browsers because I don't like the methods for saving using them.

Looks the only easy way is using Node webkit.  About the restrictions of TiddlyDesktop, I suppose you can control them right? So no big issue here.

We'd all prefer something more convenient, the point is that browsers explicitly disallow the scenarios that would be most useful.

 

Is there any way to detect new Tiddlers within a single file implementation? Maybe I could try send new Tiddlers to my own server and merge then.

The issue is that browsers have shut down the ways that an HTML file loaded from a file:// URI can talk back to a server. That's why we have to workaround by initiating the sync from within the server, or use node-webkit.

Best wishes

Jeremy

That's the point. Do you have any pre-implementation or workaround using node webkit? I will be glad to try it out. 

A small offtopic : is there any easy way I can try the new feature from 5.0.7 for syntax highlighting in the current one? I'm not very familiar with Git.

[Jeremy Ruston]

Hi Danielo

That's the point. Do you have any pre-implementation or workaround using node webkit? I will be glad to try it out. 

These instructions on tiddlywiki.com tell you how to run a TiddlyWiki standalone inside node-webkit, without any of the TiddlyDesktop components:

http://tiddlywiki.com/static/TiddlyWiki%2520on%2520node-webkit.html

There are no restrictions in that configuration, and so you should be able to sync up to a server.

 

A small offtopic : is there any easy way I can try the new feature from 5.0.7 for syntax highlighting in the current one? I'm not very familiar with Git.

We don't currently have nightly builds of TW, so you'd have to follow the instructions for downloading and building the latest source from the repo:

http://tiddlywiki.com/static/Working%2520with%2520the%2520TiddlyWiki5%2520repository.html 

http://tiddlywiki.com/static/Scripts%2520for%2520TiddlyWiki%2520on%2520Node.js.html

Best wishes

Jeremy