TiddlyWiki slows when using 10,000 or more word Tiddlers

[j3d1H]

When I use a Tiddler with 10,000 words in it (opening it, changing it, etc.), TiddlyWiki slows to the point of being unresponsive. I do have a very large Wiki file, is that the problem?

I use Firefox ESR, I've got a font embedded, I've tweaked the theme, and I have a few addons.

I'm happy to give more details if needed, and thank you for any answers!

[coda coder]

I have some huge tiddlers but I don't know how many words.  How are you counting them?

[Birthe C]

I do not have any tiddlers so large, but using Firefox ESR I have experienced breakdowns when TiddlyWiki slows to the point of being unresponsive. Quite a few times ending up with a ruined tiddlywiki (0 file).

Take care,

Birthe

[PMario]

On Monday, December 11, 2017 at 11:43:16 PM UTC+1, j3d1H wrote:

When I use a Tiddler with 10,000 words in it (opening it, changing it, etc.), TiddlyWiki slows to the point of being unresponsive. I do have a very large Wiki file, is that the problem?

I did a short test with FF57, but will test ESR also. I did use http://www.loremipsum.de/ to generate 1000 words and

compied them 10 times into on tiddler a tiddlywiki.com ... This works as expected. No slowdown at all.

So can you have a closer look at the plugins you use?

Did you make heavy changes to the View- or EditTemplates?

 

I use Firefox ESR, I've got a font embedded, I've tweaked the theme, and I have a few addons.

Which font did you embed and how? ....

How big are the font files?

Is the font embedded into the TW?

This would be the first thing I would have a closer look to. 

 - Create a backup

 - Throw out the font from the TW if it is embedded.

 - test again

 - Install the font on the OS ... Only makes sense, if you are the only one, who wants to see the real beauty of your TW ;)

 - test again.

just some thoughts.
-mario

[Jeremy Ruston]

I’d concur that the problem is unlikely to be a simple matter of the number of words in the wiki.

For comparison, I’m working on a project for a client to convert a 25MB Microsoft Word file into TiddlyWiki. It contains 14,800 pages, 6,300 paragraphs, 8,580,000 words and 46,000,000 characters (not counting spaces).

Converting it into a TiddlyWiki yields a 55MB HTML file. On my five year old Mac, performance is astonishingly good: it loads in less than 5 seconds, and I can search the entire file in less than 5 seconds.

In contrast, typing a single character into the Word document takes about 10 seconds to update the screen…

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/c223075f-ef26-46ed-9714-1d39d4d80f87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[@TiddlyTweeter]

I wish Google had a RETWEET button.

Jeremy Ruston wrote:

... problem is unlikely to be a simple matter of the number of words ...

... I’m ... convert(ing) a 25MB Microsoft Word file into TiddlyWiki. It contains 14,800 pages, 6,300 paragraphs, 8,580,000 words ...

... yields a 55MB HTML file. ... I can search the entire file in less than 5 seconds.

... Jeremy

[Mark S.]

Just for the record, what platform, OS, and hardware are we talking about? What is the actual size of your TW?

TW is not indexed, meaning that it scales with the hardware it is on.

-- Mark

[j3d1H]

Thank you all for your responses.

I do not have any tiddlers so large, but using Firefox ESR I have experienced breakdowns when TiddlyWiki slows to the point of being unresponsive. Quite a few times ending up with a ruined tiddlywiki (0 file).

 

I've had hiccups like that on several versions of Firefox before. It turns out, ESR is my problem.

So can you have a closer look at the plugins you use?

Did you make heavy changes to the View- or EditTemplates?

• Danielo's footNote add-on
• Highlight.js plugin
• A single filter in $:/core/macros/timeline
• No changes to View and EditTemplates

Which font did you embed and how? ....

How big are the font files?

Is the font embedded into the TW?

Input Font, embedded base64 encode stylesheet, 115KB

 

 - Create a backup

 - Throw out the font from the TW if it is embedded.

 - test again

 - Install the font on the OS ... Only makes sense, if you are the only one, who wants to see the real beauty of your TW ;)

 - test again.

I've tried these. It does not improve the performance.

I’d concur that the problem is unlikely to be a simple matter of the number of words in the wiki.

For comparison, I’m working on a project for a client to convert a 25MB Microsoft Word file into TiddlyWiki. It contains 14,800 pages, 6,300 paragraphs, 8,580,000 words and 46,000,000 characters (not counting spaces).

Converting it into a TiddlyWiki yields a 55MB HTML file. On my five year old Mac, performance is astonishingly good: it loads in less than 5 seconds, and I can search the entire file in less than 5 seconds.

In contrast, typing a single character into the Word document takes about 10 seconds to update the screen…

Load time is good. Search time is excellent. This is primarily an issue editing a large tiddler. Opening and saving the large tiddler has a bit of a pause, but typing in the editor has a ~3 second delay. That much delay takes me out of my writing flowstate.

Just for the record, what platform, OS, and hardware are we talking about? What is the actual size of your TW?

TW is not indexed, meaning that it scales with the hardware it is on.

For the record: E3 Xeon, 16GB RAM, 750Ti, Ubuntu 16.04, Firefox ESR, 10MB wiki. This Wiki has this same problem on multiple devices.

So, I grabbed Chromium, and I have almost no performance problems on it. I can feel the difference, but it's acceptable. It is not like typing inside a text editor on my computer.

The only reason I'm using ESR is because I love some of my extensions, including Tiddlyfox. I tried Firefox Quantum. It isn't as performant as Chromium, but it is better than ESR. I'm still getting tiny delays while typing inside the editor of Firefox Quantum.

[@TiddlyTweeter]

Small comment...

j3d1H wrote:

This is primarily an issue editing a large tiddler.

Edit huger tiddlers in a plain text editor outside TW (unless you need preview)?

Personally I'm still on FF ESR and regularly use the FF plugin "It's All Text" to seamlessly invoke a plain text editor.

Just a thought

Josiah

[Mark S.]

Oh! You didn't say that it was *editing* that was the problem. I've complained about slowness in typing since the very beginning, even with smallish TW's. But it's not too bad at the moment on my desktop FF ESR with my Big Book.

What add-ons/extensions do you have in your browser? If you disable them and reboot does your ESR work any better?

Good Luck,
Mark

[PMario]

On Tuesday, December 12, 2017 at 4:31:42 PM UTC+1, j3d1H wrote:

So can you have a closer look at the plugins you use?

Did you make heavy changes to the View- or EditTemplates?

• Danielo's footNote add-on
• Highlight.js plugin
• A single filter in $:/core/macros/timeline
• No changes to View and EditTemplates

Did you try to disable the plugins one by one and see, if it changes

-m

[j3d1H]

I've figured out it isn't the plugins or extensions that was the problem - I took those off, and I moved the problem Tiddler to an empty TiddlyWiki. It's still just as slow to edit.

I'm planning on trying It's All Text, since I was going to anyway.

[TonyM]

j3d1H,

I was recently - couple of months ago, experiencing sluggish performance in FF with 8 tabs and medium sized tiddlywikis. Apart from now accessing single file and nodeJS wikis via TiddlySever I have also realised that a large part of my work is done in the browser because of tiddlywiki, 

so I increased the RAM both FF and Chrome can use and the performance is much improved. I have 16GB RAM like you so typically room to spare,  in addition to SSD I not longer feel like falling asleep after lunch waiting for a screen refresh.

The thing is we are always tempted to reduce our use of Computer resources when in fact a system running near 100% is arguably the most efficient. Make use of your ram by letting your browser get more.

Regards

Tony

[h0p3]

Excellent. I am having this problem. Could you point us to what changes you made in particular? How did you increase the memory ceiling for Firefox? Was it just in about:config, or what tool did you use?

[TonyM]

I increased both RAM settings (Still looking for my notes) and Cache Size.

See Here For Cache http://techiesstuff.net/2013/09/how-to-increase-cache-memory-in-chrome-fire-fox-and-internet-explorer.html (newer may exist)

Google is your friend here, eg FireFox increase RAM

be aware that information is often the opposite, eg; increase Ram so more is available to System, not more is available to Firefox

Please let us know what you find when its fresh in your memory, which I failed to do

Regards

Tony

[h0p3]

Sadly, increasing cache size has zero effect for me. Even turning off disk cache and running unlimited memory cache has no effect. I'm considering trying tmpfs cache of the profile directory, but that is probably useless. 

If I leave Firefox ESR for my wiki, then I'm going to Chromium. It is noticeably more performant than Quantum in this case.

[Birthe C]

I am using an old netbook 32 bit and only 2gb RAM. But still it worked very well with Firefox up to about half a year ago. Now I am running Chromium on the same netbook and have no problems. Another plus, tiddlyclip is working.

Birthe

[TonyM]

FYI:

The cache, was only one possible bottle neck, 

You may know that a bottle neck remains until you widen the neck, if no improvement comes from widening a particular neck, it was most likely not the bottle neck.

The Biggest bottle neck on my computer was the RAM the browsers could use, Most browsers have a fixed upper limit on the RAM (Not RAM Cache) they will use so that people do not complain about there impact on the desktop in question (opening multiple instances may differ). Increasing the RAM available to Chrome and Fire Fox had the biggest effect, it may not be yours.

However since the latest FireFox each tab seems to have its own memory with plenty of overhead. I recommend the latest browsers with TiddlyServer or Using TiddlyDesktop or NodeJS (Raw form of two previous). 

If you do use node JS, accessing via a Browser address (not file open) the save issues are not present, and every tiddler is in a single file, so I believe you can have much larger TiddlyWikis without the performance hits.

Food for thought

Tony