slowdown of tiddly wiki...debug protocol? Logs?

[Rich]

Hi, I recently had a slowdown of tiddlywiki.

conditions to replication:

* start with 120 meg tiddlywiki that has worked fine for many years.  running up-to-date manjaro with TiddlyWiki version: 5.1.17 on node.js.  running on chrome, most recent version.  Something around 985 tiddlers.  

* add text reslicer plugin by dragging reslicer and sax plugins from here (under "plugins" tab) to my tw5. Everything works fine, except  the styling, which I can figure out later I hope.

* add plugins for tiddler tools by dragging to my tw5.  

result: Adding tiddler tools causes tw5 to hang while loading and then crashing.  

workaround:

* I tried to disable the tiddler tools (I could just get to the config for plugins before the browser window crashed after five or six tries)  but that does not fix it.

* delete the $_plugin files for tiddler tools (inmysocks) from the tw5 tiddlers directory. 

* kill and restart nodejs tiddlwiki server 

* chrome crashed but I was able to load in firefox for some reason.  saved in firefox.

restart server again.  After that I could usually load the wiki without crashing, but it has become very slow.  I

 tried watching the console in the browser and launching from the terminal for tiddlywiki nodejs server, but nothing stood out as a problem. The messages just responded slow like the browser.

Are there any other ways of debugging how TW5 is working?

I would like to explore them if they exist before starting fresh, as I have a fair number of plugins and custom stuff that I depend on an and restarting from scratch is not trivial.  

Any guidance appreciated.

~Rich

[Mark S.]

I'm stunned with that 120meg number. That's 12x larger than I would call top-normal. But maybe results on Linux are better (haven't tested there yet). How much system memory do you have?

After your crash, did you check your memory? Did you reboot your system? Or close out your browser? For me FF steals memory growing larger over time. I believe this happens both in Windows and Linux (though to a smaller extent). It has to be periodically reset, and it does appear that TW exercises it more.

In your notes you say you added text slicer, but it sounds like you may have also run it? Which would presumably increase the number of tiddlers.

I don't know about TiddlerTools, but I have encountered other plugins that generate system tiddlers behind the scenes. Does your tiddler count include system tiddlers? You can use a filter like [all[tiddlers]count[]] in the advanced search to get a full count.

Good luck!

-- Mark

[Jed Carty]

I am not surprised that the tiddler tool thing I made doesn't do well with that many tiddlers, there are probably a lot of recursion  errors. Check if there are any edited tiddlers, they would start with $:/plugin/inmysocks/

[Rich]

On Sunday, October 21, 2018 at 3:49:30 AM UTC-10, Jed Carty wrote:

I am not surprised that the tiddler tool thing I made doesn't do well with that many tiddlers, there are probably a lot of recursion  errors. Check if there are any edited tiddlers, they would start with $:/plugin/inmysocks/

Hi Thanks Jed, 

I deleted all the $_plugins/inmysocks/ tiddlers from the tiddlers folder, and there are not any $:/plugin... files at all (probably different characters allowed on nodejs/linux)

On Sunday, October 21, 2018 at 3:49:30 AM UTC-10, Jed Carty wrote:

I am not surprised that the tiddler tool thing I made doesn't do well with that many tiddlers, there are probably a lot of recursion  errors. Check if there are any edited tiddlers, they would start with $:/plugin/inmysocks/

~~~

Mark S wrote:

I'm stunned with that 120meg number. That's 12x larger than I would call top-normal. But maybe results on Linux are better (haven't tested there yet). How much system memory do you have?

Thanks Mark, 

I have been keeping notes and writing in TW since 2006.  Some of the megs are from images, but there is a lot of text too.  Eventually, three major publications should come out of it: two books and the Great American Hypertext Novel :) along with a bunch of subgroups on sound and hearing and technology and mediation.

Memory: 16 gigs, but I think one advantage of nodejs is that the tiddlers are not all one file, so it does not load the whole thing into memory at once.

After your crash, did you check your memory? Did you reboot your system? Or close out your browser? For me FF steals memory growing larger over time. I believe this happens both in Windows and Linux (though to a smaller extent). It has to be periodically reset, and it does appear that TW exercises it more.

Did not check memory immediately after the crash, but when linux moves to swap, it slows everything down (even with SSD swap space), this is only slowing down TW5.  Everything else is running fine.  Don't use FF often for TW5, but checking in FF on a fresh start, it is slowed down there too, with the side effect of eating more memory than Chrome in this case.  

Didn't think to reboot and I may have not restarted Chrome either.  will try and see if it fixes things.  MS-DOS advice to the rescue!  My bet is this fixes it :)  I still always forget to try it.

In your notes you say you added text slicer, but it sounds like you may have also run it? Which would presumably increase the number of tiddlers.  

I have run the text slicer on one file and it increased the file count to 1185, so I am a bit concerned about that in the long run.  the 985 count was pre-slice.

I don't know about TiddlerTools, but I have encountered other plugins that generate system tiddlers behind the scenes. Does your tiddler count include system tiddlers? You can use a filter like [all[tiddlers]count[]] in the advanced search to get a full count.

Felix Hayashi's TiddlyMap is generating some files, but I have been using that without issue for a long time. Did just update it though so I will check with him about removing these files.  The filter you gave counts 1129 tiddlers, 56 less than in the file folder count.  I found 42 conflict resolution files from DropBox that are a byproduct of using the wiki on different computers.  They are not counted by TW5 and I removed them to a backup folder, but they have also been a part of the workflow for years without issue.  the others are probably retired media files or something.  

Good luck!

Thanks! will update after reboot

~Rich

[TonyM]

Rich,

Remember the browsers have their own internal memory setting that stop them consuming too much of the host computers memory resources. Chrome and FireFox have settings that allow this to be increased, especially appropriate to TiddlyWiki users who's key application is the browser and TiddlyWikis within that. These browser limits met I was just not utilising the 16GB on my laptop, upping these has helped me, but is likely to help you more if your computer has the RAM.

I would be interested to know where you use the most memory in your tiddlywikis, because the size you have could be a thousand encyclopedias, do you include media and images? Few people could write one encyclopedia in a life time, 

Regards

Tony

[Mark S.]

Where do you tweak these settings? Because on my Windows machine FF will keep gobbling up memory until 90% of memory is consumed. Not sure if the same thing happens with Linux yet.

Thanks,

-- Mark

[Rich]

Hi Tony and Mark, thanks for helping.  

I think it was reboot + cruft removal for the win. At least so far the responsiveness is back to normal.  Cruft removal brings the folder size down to 35 Mb instead of 120 Mb.

I was curious about the images to text/media ratio.  I have a combined total of about 21 megs in image files, mp3 files, and config tiddlers.  The other 14 M are all basically text tiddlers, and 14 Mb is a lot less than the 120 of yesterday but still pretty substantial.  The largest two text tiddlers are 188kb and 51kb.  The larger one is a long string of notes I took in Zotero, copypastaed into TW classic about 2009, and imported into TW5 when I upgraded from classic.  It is all text with some formatting as far as I can see. The next one is all notes on a particular book.  I do a lot of quotes with comments from books and articles so I don't have to dig out the original every time I write "real" stuff from my notes, so it is not like I type everything.  It is probably a couple of volumes worth of linear text but not an encyclopedia.  I write for a living and have used TW since 2006 for my notes and ideas, so there really is a lot of text.    

I'd be curious to know where the max memory settings are for Chrome, I think FF has a setting in the regular settings menu to only load pages on focus, which saves a lot of memory.  On Chrome, I use an extension called "Tiny Suspender" which will suspend the tab after x minutes, replacing it in memory with a tiny URI file that brings back the page from the web on focus.  It can be turned off per page, which is a good idea on TW5 because it is writable and gives a warning everytime Tiny Suspender goes into action, which is a pain.  I seldom run out of memory and know immediately when I have.  After a reboot, loading all my Chrome pages w/ tiny suspender active on everything but gmail and TW5, I only use up 4.5 gigs.  Even if the whole 120 megs is loading, it should be fine once it is in, but as I said, I don't think everything has to load at once on Node.js, just the core and whatever tiddlers are open, which is one of the reasons I use it.

[Mark S.]

Hello Rich,

You're describing "lazy-loading". I'm pretty sure you need to specify this in your settings or in the way you launch node. Otherwise everything gets loaded just as if it was a single TW file.

The images impair performance as much as text, and could be put into external files on relative paths, if using tiddlyserver, BOB, or tiddlydesktop.

It would be sensible to think that if your TW file is only 10 megs, then it shouldn't take up more than maybe 15 megs of memory. But that isn't what seems to happen, at least on my systems. For some reason, maybe because of the javascript, the browser needs many times more memory than the size of the file you're loading.

While I'm on the topic, it appears that TiddlyDesktop also takes up a lot more memory than one would expect, at least in my Windows setup.

Good luck!

-- Mark

[TonyM]

Rich,

I am all for being able to maintain a core tiddlywiki, thus a large one for integrated knowledge. However I do have a practice of building separate wikis for defined functions. Writing a book may be one of them. Developing and building plugins and macro suites another. I have now got a high degree of loosely coupled TiddlyWikis using curated json files, which have a wiki owner. 

Want to share the subject of you book? I have an idea for one, I have never written a book before so it may be a pipe dream. Mine would be on knowledge and Information Management and the application of Occam's Razor, clearly a document that could accompany tiddlywiki.

Mark,

There are two lots of settings that can be increased to give a browser more resources, one in the cache size, instructions are available for most browsers.

To save putting the argument myself see https://www.howtogeek.com/334594/stop-complaining-that-your-browser-uses-lots-of-ram-its-a-good-thing/

However I cant find my notes (odd) or again in Google.

I will post if I find it.

Tony

[Rich]

Mark S...trying out lazy loading of images.  Lazy loading everything crashes my templates. Thanks for the tip. Will also look at offloading images outside of tiddlers folder.

On Sunday, October 21, 2018 at 8:05:30 PM UTC-10, TonyM wrote:

I am all for being able to maintain a core tiddlywiki, thus a large one for integrated knowledge. However I do have a practice of building separate wikis for defined functions. Writing a book may be one of them. Developing and building plugins and macro suites another. I have now got a high degree of loosely coupled TiddlyWikis using curated json files, which have a wiki owner. 

I like the occasional insights I get from having the block of reading and research and first draft writing together.  Using Felix's TiddlyMaps, I sometimes find connections I would miss if I siloed things in separate wikis.  LOL, eventually I'll probably pay with crashes, but still working after 12 years is a pretty good track record.  I'll occasionally start other wikis, but if I get anywhere with them, they get incorporated into the big wiki eventually.   

Want to share the subject of you book?

The one I am desperately trying to finish right now is a History of Hearing from the Big Bang to the Present. down to two chapters left.  I got through the first 13.5 billion years pretty quickly, but that last two hundred is killing me ;)

 

I have an idea for one, I have never written a book before so it may be a pipe dream. Mine would be on knowledge and Information Management and the application of Occam's Razor, clearly a document that could accompany tiddlywiki.

sounds interesting! 

[TonyM]

Rich,

Yours sounds interesting too. When its finished make sure you post here, because hay, it was written in tiddlywiki.

Tony

[Mark S.]

Are your templates system (starts with $) tiddlers or regular tiddlers? If they're system tiddlers, then I think TW will load them when it starts up. But if they're regular tiddlers, they'll be loaded skinny (empty) causing problems when you go to use them.

Good luck!

-- Mark