TW5 - Suggestions for truly long term use?

[RickL]

I love TiddlyWiki and have used it daily for almost a year.  I began in January to make my journal and notebook digital instead of a paper notebook.  I love the flexibility and the linking of topics.   The closest thing I have found that works.  

My concern is with true longevity. Though I have several TiddlyWikis, my main notebook (Common-place book, Zettelkasten, call it what you will - that's how I use it) is 2.7 GB and I have started noticing a lag in saving and loading, particularly on my phone (I use AndTidWiki and Dropbox with my Android phone).  At this rate of growth I worry about how useful it will be in 3 years.  I use it with Chrome and Arlen's TiddlyChrome.

There is the option to split the TiddlyWiki into two or more, but that takes away a lot of the functionality with no way to link between files.

Every release amazes me with the new tools available. I know this product is just out of beta and under constant development so I hope there can be a way to address this.  Maybe the solution is already out there and I don't know about it.

Keep up the great work!

Thanks

Rick

[Mark S.]

Are you sure that it's 2.7 GB? Not maybe 2.7 Mb?  that's an astounding amount of information. It's hard to imagine that you're experiencing only a little lag saving/loading. I can't think how you could have acquired that much data -- unless you're loading it up with images. If so, then the solution is to make the images external to the actual TW.

There's no relational database behind TW. Indexed or relational databases are usually necessary for dealing with large amounts of data. That means that TW will always hit an impasse based primarily on the power of the hardware that supports it.

Mark

[RickL]

Yes, of course it is 2.7 MB - sorry for that...don't know what I was thinking.

I have read a bit more on the forums and see there are others with much larger TW than that which seem to function OK...

[Jed Carty]

Unless we can make some sort of tiddler based database inside an html file you may have to have separate archive wikis you move old things into to keep the size of your main wiki down. Or maybe we could make a plugin that can archive tiddlers by bundling them into a disabled plugin. That way you could enable the plugin to search through old content but it shouldn't affect your normal use.

Does anyone know of a way we could make a database part of tiddlywiki? I don't know what the limits are for javascript and html.

[Mark S.]

It's supposed to be possible to use HTML5 to link with sqlite. I imagine it would take a bit of a re-write to get tw to use an entirely different approach to storing tiddlers.

With the old TW someone had written a plugin that would allow you to bring a separate TW online into the current TW with a click. (the tiddlers brought online were read-only). This meant that you could, for instance, toggle records for different years. So usually you would just look for the current year, but you could extend your search and add other years.

I'm not sure that putting things into a disabled plugin would help, since the plugin still has to be loaded into memory.

Another idea is to create a master index TW that contains all the key terms in a set of TW's. Then you can click through to what you want in a couple steps without having to load one monster TW.
 
Mark

[iain]

Rick,

My largest TW is 14.5 kb and was started in 2008 based on a theme by Morris Gray's who modified No Brainer notes.

I think the greatest problem is not size but  technical obsolescence my version is TW classic and I haven't been successfully been able to update past 2.7.1 because all the next version wrecked all the formatting and macros and I had no idea how to fix them and they didn't seem to be there in the post classic version TW 5.

Iain

[Jeremy Ruston]

Hi Mark

On Sun, Aug 2, 2015 at 3:19 AM, 'Mark S.' via TiddlyWiki <tiddl...@googlegroups.com> wrote:

It's supposed to be possible to use HTML5 to link with sqlite. I imagine it would take a bit of a re-write to get tw to use an entirely different approach to storing tiddlers.

It would be possible to write a sync adaptor that synced changes to HTML5 WebSQL or IndexedDB - there's an example of the latter linked from tw.com:

http://tiddlywiki.com/#IndexedDB%20Plugin%20by%20Andreas%20Abeck

However, personally, I'm not much of a fan of storing content within the browser. The trouble is that (a) the browser places pretty low storage limits - often just a handful of megabytes and (b) the data is rather inaccessible to the end user. You can't easily verify that the data is intact, nor back it up, nor move it to a different browser or different computer.

 

With the old TW someone had written a plugin that would allow you to bring a separate TW online into the current TW with a click. (the tiddlers brought online were read-only). This meant that you could, for instance, toggle records for different years. So usually you would just look for the current year, but you could extend your search and add other years.

The same thing is possible with TW5 - it's actually how the plugin library works; the plan is to adapt it to retrieve tiddlers from other tiddlywiki files.

 

Best wishes

Jeremy.

Mark

On Saturday, August 1, 2015 at 4:48:53 PM UTC-7, Jed Carty wrote:

Unless we can make some sort of tiddler based database inside an html file you may have to have separate archive wikis you move old things into to keep the size of your main wiki down. Or maybe we could make a plugin that can archive tiddlers by bundling them into a disabled plugin. That way you could enable the plugin to search through old content but it shouldn't affect your normal use.

Does anyone know of a way we could make a database part of tiddlywiki? I don't know what the limits are for javascript and html.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/581f5ede-d10c-4a60-9cae-fae0c5097312%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jeremy Ruston
mailto:jeremy...@gmail.com

[G.J.Robert]

Thank God it's 2.7MB, not 2.7GB. I just thought my 22MB TiddlySpace space was so easily out-sized.
Big TWC files do function okay, but save and load slowly. Hopefully someday TW5 can save them.

RickL於 2015年8月2日星期日 UTC+8上午6時55分25秒寫道：

[Mal]

I thought I would add a couple of points to this discussion:

1. I understand that Dropbox breaks files up into blocks of about 4MB and only synchronises blocks that have changed.  This should mean that the sync time will not get much worse after the 4MB file size is reached.

2. I set up a note-taking tiddlywiki for a medical student and it grew to about 35MB.  This was synchronised with Dropbox across a laptop, an iPad and a phone and, surprisingly, worked quite well on all devices, although most editing was carried out on the laptop.  This gives me some comfort that my own TW journal will be usable for some time to come.

Regards,

Mal

[Danielo Rodríguez]

I know that I mentioned it several times, but, anyway: I am working on a plugin for TW that uses pouchdb as a store for tiddlers. This means an in-browser database with the possibility to synchronize to an external couch database. The plugin will also provide a way to save the DB as regular tiddlers and also as a bunch of tiddlers in form of BIG JSON tiddler.

Sadly, I don't have much time this month to work on this (I have some Linux exams). Expect a first release after 31st of august.

Regards

[Atul Grover]

Dear All,

Time and again TW5 + xAPI for mobile learning comes up as a long term use of TW5.

Here is how EPUB3 and xAPI have come together for Mobile Learning

https://www.youtube.com/watch?v=IZ7KEz3-N5s

A pinterest type story-river is another idea that recurs for elearning on TW5.

https://archive.org/details/image

Regards
Atul

[Danielo Rodríguez]

El jueves, 20 de agosto de 2015, 14:07:00 (UTC+2), Atul Grover escribió:

Dear All,

Time and again TW5 + xAPI for mobile learning comes up as a long term use of TW5.

Here is how EPUB3 and xAPI have come together for Mobile Learning

https://www.youtube.com/watch?v=IZ7KEz3-N5s

A pinterest type story-river is another idea that recurs for elearning on TW5.

https://archive.org/details/image

Regards
Atul

I really want to understand what the XApi is or can do, but I fail every time I try. Could you simplify it to absurd? 

[Arlen Beiler]

Nobody mentioned it yet, I don't think, but for TiddlyWiki5 there is a NodeJS edition that syncs changes to the file system live. It has a little webserver which serves the save all template to the browser, and then the browser syncs its changes with the server. It's not quite realtime, but it's very close, and it could be with Socket.IO, I think.

Also, you can run that in Node Webkit with some very slight modification and it will actually save changes directly to the file system. I think mainly all I had to do was make a custom tiddlywiki.js file. 

Not sure on the performance improvements over the browser, except for saving time, of course, because each tiddler is in its own file and only what needs to be saved gets written.

-Arlen

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/b975c277-fd54-4163-99e5-100ba0e92639%40googlegroups.com.

[RichardWilliamSmith]

I really want to understand what the XApi is or can do, but I fail every time I try. Could you simplify it to absurd? 

xAPI is a really simple api for recording student progress on an arbitrary curriculum - all we basically do is send messages that say "S/he did this" - ie; "Danielo passed his Linux exams".

The message is split into 3 parts - an identifier for the individual: usually just an email address, a verb from the approved vocabulary ("passed") and a specifier for the task ("Linux exams").

The identifier is supposed to be universal, the verbs and specifiers come from approved lists where they already exist but can also be created to meet your needs.

Because it's so simple in principle, there are lots of decisions to make in implementing the standard - which events will trigger a message, which verbs will we use, which set of specifiers will we use. 

For example, if I have built an interactive ebook for a cohort of 100 students and each copy will report back to my 'teacher server', what will they send and when? "John read for 20 minutes" or "John read 5 pages" or "John read page 11"/"John read page 12" etc. Do we want to record the fact that John *tried* to answer some questions but got them wrong? If John has a question about his work, do we send that to the xAPI server, or somewhere else?

So, you see, the implementation is quite simple - a client/server connection for exchanging and validating simple, 3-part statements - but to make a useful implementation requires us to agree on the allowed verbs and how the tasks will be broken up and reported: so quite a lot of design is necessary to make a useful system.

Regards,
Richard

[Atul Grover]

Hi Danielo,

Check out this video...

https://vimeo.com/102949934

the demos are available at 

http://z2.vc/r/

The LRS is not working but yo'll get the idea.

Regards

Atul

On Sunday, August 2, 2015 at 3:00:18 AM UTC+5:30, RickL wrote:

[RickL]

In the past few weeks, my TW5 file has grown from 2.4MB to just over 3MB.

This has resulted in a noticeable slow down of the file.  Making changes to any tiddler now takes between 3 and 4 seconds - every time!  Also, saving the file a full 7 seconds.  That seems  like an eternity in computer time.  I am currenty using the TiddlyChrome plugin 

I am running Windows 8.1 on an Intel i5 ultrabook with a TWC broadband connection.

Is this the performance I should expect?  

Thanks

[Evolena]

See Performance

If your current sidebar tab contains a long list (e.g. the Recent tab), it slows down TW. As if you've opened tiddlers with a lot of lists.
As an example, my 4,5 Mb TW saves in ~4s if I'm on the Recent tab, and less than 2s if I'm on "Open" with only a couple of opened tiddlers.

[RickL]

Thanks ....I tried that and it does help a little.  Still much slower than I prefer.  

[Mat]

RickL wrote:

 from 2.4MB to just over 3MB.

This has resulted in a noticeable slow down

The relatively small size change here can hardly be the reason for this slow down.

Could it be that you've added tiddlers containing code that are permanently taking system power? For instance some code that is in the viewtemplates or pagetemplate and that has a lot of list filters, particularly multi-layered list-filters?

<:-)

[Mark S.]

TW5 doesn't use any special indexing like real databases. So the performance is going to be tied to your hardware. You don't mention how many Ghz your ultrabook runs at. My desktop I3 3.6Ghz desktop machine can save a tiddler and then save a 3.5mb TW in under one second.

I guess the main thing to take from this is to try to arrange things so that you are capturing and saving information on the fastest machine available.

In my case, the saved TW is then transferred to a tablet, which is used to read the information previously captured on the desktop.

Mark

[David Myers]

Just want to say a few words about nremote saving.

I use a number of TW files locally, and have them all running into an external repo for sharig between devices, and on the move editing.

I have noticed that pushing the modifications to GitLab (my personal private space) is much quicker that pushing pushing to ourteams JIRA / BitBucket space. The differnce is noticable, almost time for a coffee notible.

Don't know why though, but it shows that somje services are certainly not doing the same thing underneath.

David