[BETA] - TiddlySpace Backup Scripts

[PMario]

Update - Hi Folks,

Here are the 2 public beta releases:

Linux, Mac: https://github.com/pmario/ts-backup-bash/releases
Windows: https://github.com/pmario/ts-backup-powershell/releases

There are 2 short videos, which show, how it should look like :) Links can be found in the readme's.

have fun!
mario
PS: If these scripts helped you, to save valuable time, you can help me spend more time creating useful things. Thanks!

[PMario]

--- 28. Nov. 2016

If Jeremy is right, and TS will shut down at 9th of December there's only 10 days left to save your stuff.
I just want to bump the topic and also link to my [video] How to backup TiddlySpace spaces using internet archive.org thread.

--- 28. Nov. 2016

Some more info: https://groups.google.com/d/msg/tiddlywiki/k6ht7MM4Tos/J7E7wI7DBAAJ

--- 11, dec. 2016

see: https://groups.google.com/d/msg/tiddlyspace/1ZzYw953MXo/ELV6M-nzDQAJ

We have no control over when Bytemark will actually shut down the server hosting the service. Technically, the contract expired at midnight this past Friday, December 9, but we have been advised that termination is almost never immediate. I would expect the shut down to happen this week sometime.- osmosoft

---

[HansWobbe]

What a great offer !!  As a beneficiary of quite a few of your prior contributions to the tiddleyVerse, I will happily re-invest more time ( sort of like a software equivalent of a Dividend Re-Investment Program ... DRIP for financial investors )

Thanks for all of your contributions,
Hans

[Steven Schneider]

Thank, Mario.

I ran the script, and it appears to have collected all wikis my account owns. Worked flawlessly so far.

Any possibility of grabbing all tiddlyspaces that match a particular express (in my case, I'd be interested in having a copy of all spaces whose names began with "sunyit"

//steve.

On Thursday, November 10, 2016 at 4:47:02 PM UTC-5, PMario wrote:

---

[PMario]

On Friday, November 11, 2016 at 8:24:01 PM UTC+1, Steven Schneider wrote:

Thank, Mario.

I ran the script, and it appears to have collected all wikis my account owns. Worked flawlessly so far.

 

Nice to read this :)
 

Any possibility of grabbing all tiddlyspaces that match a particular express (in my case, I'd be interested in having a copy of all spaces whose names began with "sunyit"

I'll need to have a look, if we get a corresponding list. ... If there is a filter expression, it should be possible. ... BUT _publilc only. To get the _private bags, we need the PWs.

eg: http://pmario.tiddlyspace.com/recipes?select=rbag:neui-em_public  returns all spaces, that include the neui-em theme. it returns 404 names. A big part of them are sunyit-something. ...
It seems to be related to the TS lookup plugin

@cdent is there a better search option?

have fun!
mario

[Jeremy Ruston]

Hi Mario

I'll need to have a look, if we get a corresponding list. ... If there is a filter expression, it should be possible. ... BUT _publilc only. To get the _private bags, we need the PWs.

eg: http://pmario.tiddlyspace.com/recipes?select=rbag:neui-em_public  returns all spaces, that include the neui-em theme. it returns 404 names. A big part of them are sunyit-something. ...

One useful tip is that http://tiddlyspace.com/recipes.json gives you an array of all the recipes corresponding to public spaces (the data I retrieved just now is attached for reference).

Best wishes

Jeremy

[PMario]

Thx Jeremy. That's a possibility too. We can just filter them on the client side. I'll have a look at it.
-m

[Chris Dent]

@cdent is there a better search option?

It's possible to filter the list of recipes with something like:

    http://tiddlyspace.com/recipes?select=name:%3Esunyit&select=name:%3Csunyiu

Basically that's saying get me all the recipes with names between sunyit and sunyiu. When someone is logged in it will also return those private recipes which the current user is able to see.

There are other filtering incantations that can be done on the recipes list as well. There's more information on filters in http://tweb-filters.tiddlyspace.com/

Note that I have moving information like that to tank on my do-really-soon to do list, so that it doesn't get completely lost.

Your script makes a lot of sense Mario. Not overly complicated but does the right stuff. Getting both the fat JSON and the HTML of the tiddlywiki is a great way to do it. One thing you might consider is instead of pulling the bag tiddlers, get the recipe tiddlers. For people with multiple spaces this would result in a great deal of duplicated data but might make reconstructing things later a bit easier. Not sure if that's a good idea or not, but I though it was worth mentioning.

Thank you for cc'ing me on this and if other things come up where you think I can help please cc me again. I don't actively read these or the other tiddly groups anymore (due to lack of time, not lack of interest) so I don't become aware of stuff unless people let me know directly.

[Tobias Beer]

Hi Mario,

Great stuff. Any chance / idea for also grabbing binary files once put up via:

<<binaryUploadPublic title:foo.jpg>>

?

Thanks a lot.

Tobias.

[PMario]

On Sunday, November 13, 2016 at 4:04:52 PM UTC+1, Chris Dent wrote:

@cdent is there a better search option?

It's possible to filter the list of recipes with something like:

    http://tiddlyspace.com/recipes?select=name:%3Esunyit&select=name:%3Csunyiu

thx :)
 

Basically that's saying get me all the recipes with names between sunyit and sunyiu. When someone is logged in it will also return those private recipes which the current user is able to see.

 

There are other filtering incantations that can be done on the recipes list as well. There's more information on filters in http://tweb-filters.tiddlyspace.com/

I did read a bit there, but couldn't find the right stuff. Your suggestion seem to be good.

I also checked the  whole list that we get from: tiddlyspace . com / recipes.txt
It's a perfect fit for bash scripts. .. And some suny-users used sunyit in the middle of the name, which we will get processing the whole list with grep too .

Note that I have moving information like that to tank on my do-really-soon to do list, so that it doesn't get completely lost.

Looking forward to get more info!
 

Your script makes a lot of sense Mario. Not overly complicated but does the right stuff. Getting both the fat JSON and the HTML of the tiddlywiki is a great way to do it.

The very first script imo was from FND, but I did add some more UI and removed a 3rd party dependency, that IMO doesn't exist in standard installations.
 

One thing you might consider is instead of pulling the bag tiddlers, get the recipe tiddlers. For people with multiple spaces this would result in a great deal of duplicated data but might make reconstructing things later a bit easier. Not sure if that's a good idea or not, but I though it was worth mentioning.

Yea. Thought about that, but we have the xx.html files, which contains the whole recipe tiddlers. ... and I downloaded the recipe.txt to extract stuff from the html files in the future. So I think we should have enough info for a full reconstruction of the whole structure.
 

Thank you for cc'ing me on this and if other things come up where you think I can help please cc me again. I don't actively read these or the other tiddly groups anymore (due to lack of time, not lack of interest) so I don't become aware of stuff unless people let me know directly.

Thaks for your work with TS and your feedback !

have fun!
mario

[PMario]

On Sunday, November 13, 2016 at 10:41:10 PM UTC+1, Tobias Beer wrote:

Great stuff. Any chance / idea for also grabbing binary files once put up via:

<<binaryUploadPublic title:foo.jpg>>

I didn't think about them yet.

Downloading the stuff should be possible, but how can we "automagically" find them?
Is there anything we can use to identify them.

@cdent?

-m

[Chris Dent]

 Each tidlder has an attribute named 'type' that, if it has a value is the type of tiddler that it is, for example 'image/jpeg'. The fat json that your script downloads can be inspected for that. The 'text' attribute contains the base64 encoded data that represents the binary. That can be decoded. Thus is it possible to go through a fat json tiddlers collection and extract all the images to separate file.

I've made a quick python (works for both 2 and 3) script that will do this for images: https://gist.github.com/cdent/1254a9c98d8ba316e6c0223f9dd03925

If you run it in the directory where the fat json files are as `python extractbin.py *.json` all the image tiddlers will be dumped into a directory named 'binaries'.

I hope this is enough to keep people some ideas on how to process the data. There are also ways to search tiddlyspace for tiddlers of particular types, but since the fat json already includes the data locally processing that seems easier.

[PMario]

On Friday, November 11, 2016 at 8:24:01 PM UTC+1, Steven Schneider wrote:

Any possibility of grabbing all tiddlyspaces that match a particular express (in my case, I'd be interested in having a copy of all spaces whose names began with "sunyit"

Hi Steve,
A new bash version is online. No such windows version is planned atm.

download: https://github.com/pmario/ts-backup-bash/releases
readme: https://github.com/pmario/ts-backup-bash

Your command should be

./ts-backup backupDirName -s sunyit

or

./ts-backup -h

for more info see the readme: https://github.com/pmario/ts-backup-bash

have fun!

[PMario]

On Tuesday, November 15, 2016 at 4:00:46 PM UTC+1, Chris Dent wrote:

I've made a quick python (works for both 2 and 3) script that will do this for images: https://gist.github.com/cdent/1254a9c98d8ba316e6c0223f9dd03925

If you run it in the directory where the fat json files are as `python extractbin.py *.json` all the image tiddlers will be dumped into a directory named 'binaries'.

I hope this is enough to keep people some ideas on how to process the data. There are also ways to search tiddlyspace for tiddlers of particular types, but since the fat json already includes the data locally processing that seems easier.

Thx. Will have a look.

-m

[Handoko Suwono]

Steps on making static backup of your TS

I am in the process of making manual backup of my TS. My method is done by making an offline copy of my TS and to convert it to recent version of TW5 thus providing a static backup. At least this is something that I can do before trying Mario's script.

I made a download copy of my TS using the export function from the backstage and it turns out to be a classic version 2.8.1.

I use a recent TW5 empty.html ver 5.1.13 and imported all the tiddlers from here and importing works as a charm. It showed incompatibility issue because of its classic version which can easily be rectified using the upgrade button to each.

"This tiddler is written in TiddlyWiki Classic wiki text format, which is not fully compatible with TiddlyWiki version 5. See http://tiddlywiki.com/static/Upgrading.html for more details."

Tiddlers with images (say imagex.jpg) and TS related functions can not be upgraded and need to be discarded.

Then I renamed all the title related in the control panel to identify it as a new TW under TW5.

Let me know if you are eager to preserve your TS using this method.

handoko -

[PMario]

Hi Folks,

If Jeremy is right, and TS will shut down at 9th of December there's only 10 days left to save your stuff.
I just want to bump the topic and also link to my [video] How to backup TiddlySpace spaces using internet archive.org thread.

have fun!
-m

[PMario]

Just bump it one more time.
There's not much time left, to create backups.

have fun!
mario

[Chris Dent]

It's really a shame how this shutdown has been handled. I can see from the frontpage and tapas that people are still creating new content and new accounts on tiddlyspace. I suspect this is because the frontpage was never updated to announce the shutdown. It's only once you've created an account that you ever hear about the shutdown.

Thank you mario for making sure people have a good way to save their stuff.

[PMario]

On Monday, December 5, 2016 at 3:00:22 PM UTC+1, Chris Dent wrote:

It's really a shame how this shutdown has been handled. I can see from the frontpage and tapas that people are still creating new content and new accounts on tiddlyspace. I suspect this is because the frontpage was never updated to announce the shutdown. It's only once you've created an account that you ever hear about the shutdown.

Yea, It will be interesting, to see, what happens after it is switched off. .. We'll see.
 

Thank you mario for making sure people have a good way to save their stuff.

Thanks for your kind words.

I also want to thank you, for the great work you did with TiddlyWeb and for TS.

have fun!
Mario

[Mat]

Time to unstick this thread?

...and APPLAUD MARIO for his help in this! :-D

<:-)

[PMario]

On Thursday, January 12, 2017 at 3:06:37 PM UTC+1, Mat wrote:

Time to unstick this thread?

Yea.

 

...and APPLAUD MARIO for his help in this! :-D

thx :)

-mario