There's only so much optimization TW5 and the browsers which run it can achieve. We're hitting the limits of single-threaded performance, and I am not convinced there are many huge gains left for javascript in the browser on high-end x86_64 CPUs (I desperately hope I'm wrong). Performance for large TWs is looking a bit grim. I'm trying to look down the road 10 years from now and ask myself if my Tiddlywiki is going to be the right tool for the job. It's an amazing rapid-prototyping tool, but maybe it can't be made performant enough. What do you think?
Even though I'm still a noob at it, I use TW a lot; I'm an addict. In 3 years, I've amassed 10k tiddlers of almost pure text in a 30MB TW (that's ~60 novels in length), and I don't see myself slowing down. I don't do anything fancy, but this tool is heavily integrated into my life. I'm a unificationist too: part of the strength of this tool is that I don't have to separate it into unconnected documents. I want to search, navigate, hyperlink, and construct the whole. Unfortunately, the tool is getting slower and slower for me. I'm pretty worried I need to move away from Tiddlywiki.
The standout property of Tiddlywiki is that I get to serve a self-modifying IDE+Product as a single html file (though I no longer develop it without Bob). The client's browser does most of the calculation, and I don't have to rely upon having a server which does anything more than dishing out static files; I'm not beholden to centralized webservers (though I still use github for now). I adore how it is censorship resistant, cryptographically signable, easy to distribute, and it runs on almost any device (though, at this point, my wiki barely runs on a phone). It's perfect for P2P-serverlessness. There is nothing else like it in this respect unless I'm handing someone a complete VM or container, but that doesn't work nicely in a browser. There is no replacement as far as I can tell.
I've probably made plenty of mistakes in attempting to optimize, but I'm trying. I try to stick to hardlinks, and I do my best not to generate anything dynamically when I can. There's only so much optimizing one can do for complex filter expressions. I even bend over backward to do what I call "Firmcoding" in which dynamically generated lists are pre-computed into static indexes (basically, this enables link references to function, limits clientside computation, and if I have to move out of Tiddlywiki, all of my linking structures are still hardcoded). What kinds of precomputing can I do here to help the client side? I've only started really using tags this past year, and I just ripped out all of the tags to find almost no performance gains either. Do I have too many tiddlers? Do I need to start finding ways to molecularize (rather than atomize) content into larger tiddler bodies and then build specialized parsing for those large tiddlers? I'm growing desperate.
I feel like I'm going to puke. I've seen this coming for a while, and I've been burying my head in the sand hoping it wasn't true. This is my evolving horcrux-pensieve, and now I feel like a hermit crab who might have to find another shell. I'm heartbroken at the thought. This is the best damn tool I have ever used in my entire life; I can't bear to lose it. NO! NO! NOO!!!!!!!!! /hissy-fit.
Is there no hope for me here? Will the next TW-X be built with WASM? What other toolset do you recommend for me? Where should this pilgrim go, and what should he do?
--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/ef383bab-eed0-4549-b407-b16dba96dae9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Anyhow, probably the best we can do at the moment is to continue to improve the performance monitoring tools so that anyone can make observations of their own wikis, and we can figure out how best to act on the resulting telemetry (i.e. without everybody having to send a copy of their private wikis to me!)
Anyhow, probably the best we can do at the moment is to continue to improve the performance monitoring tools so that anyone can make observations of their own wikis, and we can figure out how best to act on the resulting telemetry (i.e. without everybody having to send a copy of their private wikis to me!)The advantage of h0p3's wiki is, it's public ;)
-m
--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/83022264-780c-4b18-ba65-a895a33cf647%40googlegroups.com.
Have you tried the 5.1.20 prerelease ? It's supposed to have indexing optimizations that will allow larger TW files. If so, it will be a real game changer.Good luck
What are the performance issues you are encountering?
I'm a touch typist, but I could only get about two words ahead when typing. But according to your blog, if it's not fiction, you use typewriter for creating content, so typing shouldn't be a big deal.
It does take about 7 seconds to save with the default saver, but that's not too bad. I would turn off auto save if I was were going to be making a lot of tiddlers.
h03p,
Sounds like this is a case where TiddlyWiki's success means your demand on it increased. As they say, "our needs expand to fill the resources available".
You are aware of performance improvements with indexing in the 5.1.20 pre-release?
You can move content to skinny tiddlers and external media files canonicalURL
You can use html object tags to include external content at "run time"
You can use node to access static versions of tiddlers, without loading the wiki.
Are you aware of Jeremys Amazon Web Services deployment of a large tiddlywiki?
There are plenty of ways to integrate independant wikis so it is always possible to divide and conquer
You could also insert subroutines that are processed elsewhere with some cunning design.
Custom indexing, compression another tricks could be used in extreme circumstances.
I use tiddlywiki a lot, it is one of my main applications suits, mostly self build wikis. This means I load a lot more in my browsers than most. I have a 16GB Ram laptop. As a result I lifted the maximum ram available to Chrome and FireFox (in their settings) so they can use a greater share of my laptops ram and they perform even better as a result. If we move from executables to browsers we should give browsers the resources they need. People forget this because they are so often working to reduce the wikis use of browser resources they forget they can give the browser more resources.
Sometimes the best method is loosely coupled TiddlyWikis. I can drag and drop tiddlers and tiddler bundles between wikis using an iframe and the multi-user and multi-access methods that continue to be developed will only make this easier.
I do not think we need more threads I think it can adapt a great deal, not to mention new developments to respond to future issues. ...Try my suggestions above if this persists, give us the opportunity to address it. I do not think you will need to move away, just tackle it differently.
I don't try and get my primary wiki working on mobile, I design independent solutions and integration as needed.
Do not get desperate until you have followed all leads. Recent discussions suggest in computation intensive tiddlers, you can take a snapshot (to a static Tiddler) and show that rather than the active tiddler) so no widget tree update will be necessary when other things change. In an application I am building, I am creating a mechanism to identify when such a refresh may be needed and indicate as such. ...
I believe your anxiety is unwarranted. ...
If TWX makes this simpler for you I do not know, but I seriously believe we have the technology to avoid what ever you may throw at it, just sometimes you may need to re-engineer things. TiddlyWiki is extensible in so many directions out of the box, and easy to modify and extend.
Regards Tony
One more thing,
Do not assume your current methods are the optimal methods. Perhaps you just need to redesign some components. Are your filters efficient? Do they eliminate the largest number of tiddlers in the first filter operator?
, should you be giving yourself read only views, rather than always having inline editing....
Regards Tony
If you already use Bob than you can use it to manage multiple wikis. I suspect that most of the time you don't have any reason to have full access to 30MB worth of text. So splitting things out into separate wikis by category would be my first suggestion. Since you are already using statics links you could have permalinks in other wikis instead of links pointing to a tiddler in the current wiki.
My advice comes down to 'it is built to be modular, so let it be modular'
Hi h0p3
Great post, thank you for sharing your hopes and fears!
With regard to performance improvements and optimisations in general, I’ve found that whenever I’ve gone in pursuit of them I’ve been able to squeeze better and better performance out of the core itself. As you know, I’m working with very large wikis too, and as I’ve encountered serious issues I’ve tried to fix them, and largely succeeded.
I’ve found that to be true at all levels of the design: if I can measure the performance of something then I can almost certainly improve it. The vital component is to be working with real-world data.
With regard to browsers, I’m much more bullish that we’ll continue to see significant improvements, particularly for the large processes that browsers require for these large wikis.
One interesting point about TiddlyWiki is that the refresh algorithm is inherently parallelisable: refresh processing of sibling nodes is completely independent of one another. So, to the extent that browsers will expose more multicore functionality in their quest for performance, we’re in a good position to benefit.
Looking further to the future, my thinking for some time around TWX is to write it in an LLVM-compatible language, and compile it to WebAssembly for the Web. (I hesitated to write that because we really shouldn’t de-rail this discussion, but it does seem relevant that I’m planning a long term future for TiddlyWiki).
Anyhow, probably the best we can do at the moment is to continue to improve the performance monitoring tools so that anyone can make observations of their own wikis, and we can figure out how best to act on the resulting telemetry (i.e. without everybody having to send a copy of their private wikis to me!)
Best wishes
Jeremy
The file at https://philosopher.life/ isn't as big as that, so I think h0p3 was referring to a private wiki.
Best wishes
Jereym
h0p3
for you site https://philosopher.life have you tried disabling plugins? From what I've heard, 10000+ mostly text based tiddlers shouldn't be a problem in TW. My (not very qualified) guess is that you're using some plugins or templates that cause a lot of repeated and nested iterations or recursions.
A quick look at your plugin list shows you have the Kin filter plugin. I imagine this is a pretty system taxing plugin IF used in some template that is called all the time.
Maybe you're using a ToC that is active all the time? These are, I assume, recursive cratures.
I note these two overwritten shadow tids. I'm guessing all things "historiy" iterates through all tiddlers contiuosly to be up to date. $:/plugins/wimmoermans/history/HistoryTab $:/plugins/wimmoermans/history/HistoryTab2
I did not look at your tagging structure but if there is extensive cross tagging then maybe this would also tax the system. I'm just guessing. (If anyone has facts on these things, please shout out.)
<:-)
h0p3,
First of all, a great respect for you and for the unique design of your wiki! I haven't read your writings yet, but what I've seen is like it.
I started to investigate what could be the problem and tried the following (after each steps, I checked whether the speed had improved by switching between the "Hub" and "Keys" sidebars).
I have disabled all plugins; the speed has improved a little, but not so much Disabled background; nothing changed I closed every tiddler; it became faster very, very minimally (ASCII art might have to be put in the form of an image)
If every tiddler is closed, what can slow down the system? Just because there are many tiddlers in it, it can't slow down yet. So I was looking for tamplets that are constantly present and can slow down the wiki ($:/tags/ViewTemplate, $:/tags/PageTemplate, $:/tags/SideBarSegment), but I could not find it. Then I noticed the button in the menu bar: firmcode-all button. I do not quite understand what it is doing, but after deleting this tiddler, the wiki has greatly accelerated.
TiddlyWiki is only slowed down mostly by what is displayed. For example, if you have a very complex and slow filter (which uses Kin for example), it will not affect the speed until you open the tiddler containing it. So when you look for a "general" speed error, you should first start with the currently visible and template tiddlers.
I hope I could help.
Close the sidebar or hide the sidebar tabs or select the contents tab since what is visible in your tab often needs to be refreshed. This simple strategy can improve performance in both tw5 and TWC.
As I understand it if you use skinny tiddlers or external images the tittle tags and fields are in memory and thus searchable. All that is not loaded is the content of the tiddler. In case of images you are unlikely to want to search their content and even with external text if you have curated your tiddlers finding without search means you can find them.
Tony
Actually you don't need to delete the tiddler, but to hide it (by moving to page toolbar dropdown for example in ControlPanel -> Apperance -> Toolbars -> Page Toolbar).