[Webdav] Backups ?

[FrD]

Hi,

I've been using webdav for a few days (weeks ?) now. The main point for me is to be able to save easily my TWs as files. It works very well locally and for files on dropbox (I've tested with OneDrive : works fine too). I'm using Caddy on my PC (windows 10) and Serveur WebDAV on my android phone.

IMO there is one missing feature in the webdav saver : the ability to make backups. It could be for instance the way save-tiddlers (PMario's plugin) works or the way TW Classic manages this question.

I have no idea how to do that and where ; a modification in the putsaver ? a dedicated plugin ?.

If someone is interested and could take from here ...

I could at least do the testings :-)

Thanks

Regards

FrD

[PMario]

On Thursday, May 31, 2018 at 8:52:53 AM UTC+2, FrD wrote:

...

 

IMO there is one missing feature in the webdav saver : the ability to make backups.

IMO the simplest form of backups, with this setting would be a button, like "download-full", which creates a local "backup".

The mechanism is manual and a bit hacky, but it should be possible with 1 additional tiddler. ... We need to think about it ...

That's definitely the fastest way. But imo it's a work-around.

 

It could be for instance the way save-tiddlers (PMario's plugin) works or the way TW Classic manages this question.

My browser addOn is named: file-backups ;) ... It creates "automatic" backups, for a locally saved TiddlyWiki files.

For a client-server based version using WebDav we would need to specify some more requirements.

eg: Should the backups be pushed to the server or should they be local ?

 

I have no idea how to do that and where ; a modification in the putsaver ?

IMO there are 2 possibilities

 - A modified version, as a plugin, that replaces the PUT saver

 - A second plugin, that works besides the existing PUT saver ... BUT ... we would need to make "core-savers" aware of additional plugins, that also want to save !!!

 

a dedicated plugin ?.

Is possible, but more work needed ... also in the core.

 

If someone is interested and could take from here ...

There is an issue at github, which requests a new feature for file-backups in combination with WebDav.

I did think about it, but we will also need to change the core, to make functions like this possible.

So it won't be "short term" :/

-m

[PMario]

On Thursday, May 31, 2018 at 9:29:11 AM UTC+2, PMario wrote:

For a client-server based version using WebDav we would need to specify some more requirements.

eg: Should the backups be pushed to the server or should they be local ?

I personally would be in favor of a local backup, using the same mechanism as file-backups.

Advantages:

 - Backups "disk" usage is finite

    - Number of backups is deterministic (max 26)

 - We would have backups on 2 or more physically different locations.

 - The "backups" would be available, even if you are offline

 - Every device has it's own backups

 - The mechanism is: "low complexity" - backup syncing problems would be avoided ...

Disadvantages:

 - backups would be "per device" only.

 - The backup needed may be at a different device :/

 - User has to rely on "sync-platform" backups, if available.

   - may be more complex.

just some thoughts.

have fun!

mario

[PMario]

On Thursday, May 31, 2018 at 8:52:53 AM UTC+2, FrD wrote:

I've been using webdav for a few days (weeks ?) now. The main point for me is to be able to save easily my TWs as files. It works very well locally and for files on dropbox (I've tested with OneDrive : works fine too). I'm using Caddy on my PC (windows 10) and Serveur WebDAV on my android phone.

You know, that Windows10 has a build-in possibility to restore "older file versioins"?

-m

[FrD]

Hi PMario,

Sorry for the name of your plugin ! I should have known better as I'm still using it for two TWs !   :-) I like the way it manages backups files (I've chosen to have 4 backups). And I've made a directory junction so my TWs can live outside the download folder.

Le jeudi 31 mai 2018 09:29:11 UTC+2, PMario a écrit :

IMO the simplest form of backups, with this setting would be a button, like "download-full", which creates a local "backup".

The mechanism is manual and a bit hacky, but it should be possible with 1 additional tiddler. ... We need to think about it ...

That's definitely the fastest way. But imo it's a work-around.

As a workaround I think it's a great idea and I could use this kind of backup (until a more definite solution ?)

Regards

FrD

[FrD]

Hi,

I didn't know that. I'll have a look. Thanks

Regards

FrD

[PMario]

On Thursday, May 31, 2018 at 9:53:30 AM UTC+2, FrD wrote:

I didn't know that. I'll have a look. Thanks

I had a fast look. .. The most convenient way is, if you have a  "spare harddrive". Every odther possibility is much more complex to setup.

-m

[PMario]

On Thursday, May 31, 2018 at 9:47:30 AM UTC+2, FrD wrote:

 ....

As a workaround I think it's a great idea and I could use this kind of backup (until a more definite solution ?)

I'll post a "sidebar-button" here soon!

-m

[PMario]

Hi,

BACKUP FIRST!!!!!

 - Please download the attached file.

 - Drag & Drop import this file to your TW.

 - It will directly work after import.

 - You can activate, deactivate it as any other sidebar button.

 - It uses index.html as a filename.

It uses the "$:/core/save-all" filter, which looks like this:

\define saveTiddlerFilter()
[is[tiddler]] -[prefix[$:/state/popup/]] -[[$:/HistoryList]] -[[$:/boot/boot.css]] -[type[application/javascript]library[yes]] -[[$:/boot/boot.js]] -[[$:/boot/bootprefix.js]] +[sort[title]] $(publishFilter)$
\end
{{$:/core/templates/tiddlywiki5.html}}

Please test and give feedback.

have fun!

mario

[FrD]

Hi PMario,

It works fine, thanks a lot !

It creates index.html in the download folder, then index (1).html, index (2).html, as expected.

Regards

FrD

[PMario]

On Thursday, May 31, 2018 at 11:00:03 AM UTC+2, FrD wrote:

It works fine, thanks a lot !

It creates index.html in the download folder, then index (1).html, index (2).html, as expected.

(1), (2) .. is standard browser behaviour. Which in this case is "wanted behaviour" :)

-m

[PMario]

I was thinking about a possibility to make the name configurable. ...

But that would need several tiddlers and should be packed into a plugin. ... no promise, no eta ;)

have fun!

mario

[FrD]

Hi,

Is it possible to download to a specific folder inside the Download folder ?

I could create a folder for TWs backups like "TWBackups" inside the Download folder and have my backups of TWs files come at this location. And eventually make a junction to another folder near the location of the TW.

Another idea would be to get the file name of the tw and add a time stamp before downloading (like in TW classic).

Maybe this subject deserves another thread as it is not directly related to the original subject ...

Regards

FrD

[Lost Admin]

Hi all,

This has been an interesting thread to read. Like @FrD, I use webdav for most of my TiddlyWikis. What makes this conversation interesting is how you are trying to solve the problem. In my opinion, the TiddlyWiki code shouldn't include backup capabilities. The backup should be handled on the server side (or desktop side in the case of local savers). Some WebDAV server implementations include versioning (i.e. http://sabre.io/dav/versioning/). Most operating systems support a wide variety of backup solutions. Even the PHP saver has built-in versioning/backup (it saves the last 10 versions by default in a backup folder).

Dropbox also has built-in versioning, so the service is doing backups for your. (https://www.dropbox.com/help/security/version-history-overview)

So does Google Drive (https://support.google.com/drive/answer/2409045?co=GENIE.Platform%3DDesktop&hl=en)

CouchDB (used by Note Self) also has built in versioning (sorry no clean link lots of how it works links like http://guide.couchdb.org/draft/documents.html)

[@TiddlyTweeter]

Lost Admin

I agree. External backup is standard practice. It always seemed silly to me to burden TW with that. What I need TW to do is SAVE. Once its saved its a "doddle" to back it up. 

Lost Admin wrote:

... In my opinion, the TiddlyWiki code shouldn't include backup capabilities.

[Tristan Kohl]

I am happy to see that Caddy got some friends here :)

My backup strategy is not TW specific but since I run my Raspberry 24/7 I need a way to backup some directories from it. Therefore I use a shell script on my workstation which runs a rsync pull against my Raspberry before performing the local backup with borg. So I have always a copy from my last backup run (usually once a day) on my local hard drive and versioned backups in my borg repository. Since I need 5 or 6 locations to backup this way (no direct backup from my Pi), I just dropped the TW folder into the mix.

As I said this is not TW specific but it saves me from forgetting about backing up my wikis because rsync checks for changes.

Cheers,

Tristan

On Thursday, May 31, 2018 at 8:52:53 AM UTC+2, FrD wrote:

[PMario]

On Thursday, May 31, 2018 at 1:23:05 PM UTC+2, FrD wrote:

Is it possible to download to a specific folder inside the Download folder ?

No. I only uses standard browser download behaviour. .. So if you would change your browser settings: "Always ask before save", the OS "file save as" dialog will open. ...

The problem is: This setting is annoying, if you use the file-backups plugins. You'll always get the dialog several times. eg: when it wants to save the backups :/

 

I could create a folder for TWs backups like "TWBackups" inside the Download folder and have my backups of TWs files come at this location. And eventually make a junction to another folder near the location of the TW.

As I wrote, that may be an extension to the file-backups plugin.

 

Another idea would be to get the file name of the tw and add a time stamp before downloading (like in TW classic).

I think, a simple dynamic filename should be possible. 

-m

[PMario]

On Thursday, May 31, 2018 at 3:49:03 PM UTC+2, Lost Admin wrote:

This has been an interesting thread to read. Like @FrD, I use webdav for most of my TiddlyWikis. What makes this conversation interesting is how you are trying to solve the problem. In my opinion, the TiddlyWiki code shouldn't include backup capabilities.

Why not. It's a plugin and a convenience function, if I don't want or can't use any other 3rd party app. .. In my experience they go away after some time, or the experience, in my area, is broken.

 

The backup should be handled on the server side (or desktop side in the case of local savers). Some WebDAV server implementations include versioning (i.e. http://sabre.io/dav/versioning/).

That's nice, when I use IIS, apache or nginx setup :/

 

Most operating systems support a wide variety of backup solutions.

I use windows 10. Built in file versioning is nice and simple, if I have a spare harddisk built in my computer. It's incredibly complicated and error prone if I haven't.

Guess what: I haven't ;)

Deja Dup for ubuntu is a complete mess, if you update from an older OS version to a new one and want to use old backups from an external HD. Even worse if username and directory structure changed.

Even the PHP saver has built-in versioning/backup (it saves the last 10 versions by default in a backup folder).

That's nice. I don't intend to use a local php-server.

 

Dropbox also has built-in versioning, so the service is doing backups for your. (https://www.dropbox.com/help/security/version-history-overview)

Cool. I would like to use it and share holiday fotos with my peers. ... But it's way faster and more convenient to store them on a memory stick and drive to their place by car. It's probably 100 times faster, than uploading my images to the cloud.

 

So does Google Drive (https://support.google.com/drive/answer/2409045?co=GENIE.Platform%3DDesktop&hl=en)

I'll never ever upload personal files to google. And my car wins here too, because of upload speed :/

 

CouchDB (used by Note Self) also has built in versioning (sorry no clean link lots of how it works links like http://guide.couchdb.org/draft/documents.html)

 

NoteSelf isn't file based. IMO we can't compare it.

-m

[FrD]

Hi,

Le vendredi 1 juin 2018 12:34:03 UTC+2, PMario a écrit :

On Thursday, May 31, 2018 at 3:49:03 PM UTC+2, Lost Admin wrote:

This has been an interesting thread to read. Like @FrD, I use webdav for most of my TiddlyWikis. What makes this conversation interesting is how you are trying to solve the problem. In my opinion, the TiddlyWiki code shouldn't include backup capabilities.

Why not. It's a plugin and a convenience function, if I don't want or can't use any other 3rd party app. .. In my experience they go away after some time, or the experience, in my area, is broken.

 

The backup should be handled on the server side (or desktop side in the case of local savers). Some WebDAV server implementations include versioning (i.e. http://sabre.io/dav/versioning/).

That's nice, when I use IIS, apache or nginx setup :/

 

...

 

-m

 

Remember that I don't know much about http protocol and XMLHttpRequest() ...-:)

And I'm only concerned about serving and saving local files via webdav (or on dropbox, but it's quite identical).

Going back to the put.js saver, what about doing a second "put" request just after the first one and giving it an URL pointing to a file (to be created) inside a folder (like "TWBackup") which could be on the same level as the original file. Same structure as in TW Classic.

Would it be a way to go ?

Regards

FrD

[PMario]

On Friday, June 1, 2018 at 1:11:32 PM UTC+2, FrD wrote:

....

Going back to the put.js saver, what about doing a second "put" request just after the first one and giving it an URL pointing to a file (to be created) inside a folder (like "TWBackup") which could be on the same level as the original file. Same structure as in TW Classic.

Would it be a way to go ?

Not really. ... If the TWBackup directory doesn't exist on the server, the client / browser would need to create it first.

While the "save action" seems to be simple, there is a littlebit going on in the background.

WebDav [1] is an extension to the HTTP(S) request methods [2]. At the moment we only use 3 HTTP functions:

- OPTIONS is used to get info about the server. eg: If PUT is allowed.

- PUT method to write a file back to the server. 

- HEAD is used to get the ETags

This PUT mechanism is "relatively" simple, if there wouldn't be the ETag handling.

It caused quite some headaches to get it right for different servers. (apache, nginx, IIS, others).

ETags are used to identify "overwrite collisions".

Eg:

 - User A and user B open the same file at the same time.

 - User A modifies the file and saves it back to the server.

 - Later user B also modifies the file and wants to save it. -> problem

The problem is, that user B would overwrite the changes from user A. Etags [3] help us to detect this problem. ...

Similar protections would be needed for a WebDav, server-side backup handling. ... As others point out: Serverside backups should be done server-side.

That's why I personally prefer "local backups". They are much simpler to handle.

have fun!

mario

[1] https://en.wikipedia.org/wiki/WebDAV#Implementation

[2] https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods

[3] https://en.wikipedia.org/wiki/HTTP_ETag

[FrD]

Hi PMario,

Thanks for the detailed explanation.

I agree that the general case is quite complicated (collision handling as you say for instance).

But I was talking about a more simple use case : serving and saving local files (on the local filesystem or on dropbox or onedrive as you have local copies of the files). There should (?) be no access conflict on the files (of course it may happen, but you should master your own files ...).

The backup folder could bee created before using the saver or the webdav server could maybe do it (after all it's on your filesystem).

So I'm pretty much in the case you refer as "local backups".

Maybe something like a "putlocal.js" file with backup ability could be created from the put.js. That was more my point.

Regards

FrD

Le vendredi 1 juin 2018 13:51:54 UTC+2, PMario a écrit :

On Friday, June 1, 2018 at 1:11:32 PM UTC+2, FrD wrote:

....

Going back to the put.js saver, what about doing a second "put" request just after the first one and giving it an URL pointing to a file (to be created) inside a folder (like "TWBackup") which could be on the same level as the original file. Same structure as in TW Classic.

Would it be a way to go ?

Not really. ... If the TWBackup directory doesn't exist on the server, the client / browser would need to create it first.

While the "save action" seems to be simple, there is a littlebit going on in the background.

WebDav is an extension [1] to the HTTP(S) request methods [2]. At the moment we only use 3 HTTP functions:

[PMario]

On Friday, June 1, 2018 at 2:05:52 PM UTC+2, FrD wrote:

...

But I was talking about a more simple use case : serving and saving local files (on the local filesystem or on dropbox or onedrive as you have local copies of the files). There should (?) be no access conflict on the files (of course it may happen, but you should master your own files ...).

The backup folder could bee created before using the saver or the webdav server could maybe do it (after all it's on your filesystem).

"I can hear you" :) ... But as a software author, you can't rely on assumptions. Especially, that the setup is done right. Murphy's law will bite you ;) There's no way around it.

 

So I'm pretty much in the case you refer as "local backups".

It may look like this, but it isn't. There is a server software in the middle. ... Trust me: In the long run, we will be happyer if we do it right.

 

Maybe something like a "putlocal.js" file with backup ability could be created from the put.js. That was more my point.

The problem is, that browsers prevent us, from easily accessing the local file-system.

That's why we are forced to use addOns as file-backups, or put a WebDav server in between our local filesystem and the browser, just to write to our own local filesystem. ... It's insane!

-mario

[FrD]

Hi PMario,

Le vendredi 1 juin 2018 14:39:28 UTC+2, PMario a écrit :

On Friday, June 1, 2018 at 2:05:52 PM UTC+2, FrD wrote:

...

But I was talking about a more simple use case : serving and saving local files (on the local filesystem or on dropbox or onedrive as you have local copies of the files). There should (?) be no access conflict on the files (of course it may happen, but you should master your own files ...).

The backup folder could bee created before using the saver or the webdav server could maybe do it (after all it's on your filesystem).

"I can hear you" :) ... But as a software author, you can't rely on assumptions. Especially, that the setup is done right. Murphy's law will bite you ;) There's no way around it.

-mario

Maybe by testing if the document URL begins with "localhost" or "127.0.0.1", one can see if it's a local file system save case and then fire the saver ...

If I can wrap my head around the http protocol, XMLHttpRequest, I'll give it a try, writing something like a local put saver with "backup" ability (MKCOL request I've read).

Oups !!! I've said "backup"  -:)

Regards

FrD