Another datapoint on extremely large TiddlyWikis

[Jeremy Ruston]

Several of the projects I’m working on for Federatial clients involve large wikis, in the 10MB to 100MB range. I’ve posted before about the surprisingly good performance of such large wikis, and recently worked on improving performance further through the introduction of more sophisticated indexing strategies.

As an experiment, today I just tried combining the data from several large wikis to make a compound wiki that weighs in at 874.9MB (nearly a gigabyte!). To my astonishment, Chrome and Firefox will both run it with reasonable performance (Safari complains about resource usage).

The wiki actually only contains 60 tiddlers, of which 13 are plugins containing a total of 64,202 shadow tiddlers (this project uses plugins to package wiki content). There are just over 3,000 images, weighing in at about 197MB of base64 encoded text.

I don’t think such large wikis are practical for everyday use right now, but they certainly will be in the next few years. (None of this is actually to praise TiddlyWiki; it’s the hardworking browser engineers over the last decade that we have to thank).

Best wishes

Jeremy.

[@TiddlyTweeter]

Jeremy, thanks ... Useful test ...

 There are just over 3,000 images, weighing in at about 197MB of base64 encoded text. 

FYI, that is particularly interesting to me. Being able to keep images in a wiki, rather than external, has good upsides on maintenance and ease of portability. I'm gonna test on one project.

Best wishes

Josiah

[bimlas]

I just tried to import 150MB of image. Previously, I thought it would take at least 20 minutes to load the wiki, but it was loaded surprisingly quickly. O_O Though I don't think it's a good idea to create such a wiki, and to keep the files physically there too, but it's good to know that TiddlyWiki is extraordinary from this perspective as well. :)

[TonyM]

Thanks for sharing Jeremy.

It great to hear the possibilities.

Can you tell the community single file or server, locally or remotely etc...? Just to flesh it out a little.

I tested the prerelease and the indexing made things possible that were not before.

Thanks so much for your ongoing innovation.

Tony

[Jeremy Ruston]

Hi Tony

> Can you tell the community single file or server, locally or remotely etc...? Just to flesh it out a little.

I ran the wiki under Node.js to generate an index.html file that I then ran standalone in the browser.

> I tested the prerelease and the indexing made things possible that were not before.

That’s great to hear,

> Thanks so much for your ongoing innovation.

Much appreciated, thank you!

Best wishes

Jeremy

>
> Tony
>
> --
> You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
> To post to this group, send email to tiddl...@googlegroups.com.
> Visit this group at https://groups.google.com/group/tiddlywiki.
> To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/deffbe4b-8276-42ff-8b5d-b4c00ca5799f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Mark S.]

What are the physical characteristics of the machine that ran your tests? RAM? Ghz? Make? Type of HD?

Thanks!

[Mark S.]

Is the indexing something that happens automatically, or something that you activate?

Thanks!

[Jeremy Ruston]

Hi Mark

Is the indexing something that happens automatically, or something that you activate?

The improved indexing doesn’t need activating, it should be entirely invisible.

It is worth knowing what gets optimised though:

* [all[tiddlers]tag[x]...

* [all[shadows]tag[x]...

* [all[tiddlers+shadows]tag[x]...

* [all[shadows+tiddlers]tag[x]...

* [all[tiddlers]field:y[x]...

* [all[shadows]field:y[x]...

* [all[tiddlers+shadows]field:y[x]...

* [all[shadows+tiddlers]field:y[x]...

The field indexer currently defaults to indexing field values of less than 128 characters; longer values can still be searched for, but no index will be constructed.

Note that the “field” operator is also used when the operator name is a fieldname, so, for example,  [all[shadows+tiddlers]caption[x]… is optimised.

Best wishes

Jeremy

Thanks!

On Friday, May 31, 2019 at 5:55:37 AM UTC-7, TonyM wrote:

Thanks for sharing Jeremy.

It great to hear the possibilities.

Can you tell the community single file or server, locally or remotely etc...? Just to flesh it out a little.

I tested the prerelease and the indexing made things possible that were not before.

Thanks so much for your ongoing innovation.

Tony

-- 
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/3ea44a29-03b9-45a4-9e71-03cb5ec18513%40googlegroups.com.

[Jeremy Ruston]

Hi Mark

What are the physical characteristics of the machine that ran your tests? RAM? Ghz? Make? Type of HD?

I’ve recently got a modern Mac with 16GB RAM, 512MB SSD and a 3 GHz Intel Core i5. But I went back to my old 2013 MacBook Pro (also 16GB RAM and 512MB SSD) and tried the file there. It only runs 10-20% slower than on the big computer.

I suspect that having 16GB RAM has the biggest impact on performance.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/2b9b25b7-67d8-4b1d-877e-d94e73c856d1%40googlegroups.com.

[Diego Mesa]

Hey Jeremy,

Ive recently noticed my wiki (10MB) has been very slow, much more so on FF than on chrome. Im very interested in your results, and how I would go about debugging my wiki.

On Friday, May 31, 2019 at 11:13:50 AM UTC-4, Jeremy Ruston wrote:

Hi Mark

What are the physical characteristics of the machine that ran your tests? RAM? Ghz? Make? Type of HD?

I’ve recently got a modern Mac with 16GB RAM, 512MB SSD and a 3 GHz Intel Core i5. But I went back to my old 2013 MacBook Pro (also 16GB RAM and 512MB SSD) and tried the file there. It only runs 10-20% slower than on the big computer.

I suspect that having 16GB RAM has the biggest impact on performance.

Best wishes

Jeremy

Thanks!

On Friday, May 31, 2019 at 3:50:24 AM UTC-7, Jeremy Ruston wrote:

Several of the projects I’m working on for Federatial clients involve large wikis, in the 10MB to 100MB range. I’ve posted before about the surprisingly good performance of such large wikis, and recently worked on improving performance further through the introduction of more sophisticated indexing strategies.

As an experiment, today I just tried combining the data from several large wikis to make a compound wiki that weighs in at 874.9MB (nearly a gigabyte!). To my astonishment, Chrome and Firefox will both run it with reasonable performance (Safari complains about resource usage).

The wiki actually only contains 60 tiddlers, of which 13 are plugins containing a total of 64,202 shadow tiddlers (this project uses plugins to package wiki content). There are just over 3,000 images, weighing in at about 197MB of base64 encoded text.

I don’t think such large wikis are practical for everyday use right now, but they certainly will be in the next few years. (None of this is actually to praise TiddlyWiki; it’s the hardworking browser engineers over the last decade that we have to thank).

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[Mark S.]

What platform? What operating system? How much memory? How many Ghz?

I find I have to reboot FF periodically. In the process manager, I can see it slowly take over more and more memory. It's done this in every version, including the latest, greatest, "improved" version.

[TonyM]

Jeremy,

As you say this is a new threshold that tiddlywiki crosses and think it would be fair to say there is no practical limit to the size for most applications that are not also acting as a large repository of data or media. Then even somewhat large repositories have being demonstrated by you. I believe The encyclopedia Britannica consumes not more than tens of megabytes so when it comes to incedental, business and life data few could ever reach the upper limits with or without storing files or media.

Of course there are limits but I believe few will be reached before server and multiple wiki options are taken and these should lift the potential ceiling higher.

I look forward to 1.5.20

Tony

[Jeremy Ruston]

Hi Diego

Ive recently noticed my wiki (10MB) has been very slow, much more so on FF than on chrome. Im very interested in your results, and how I would go about debugging my wiki.

The improved indexer is live on the prerelease. You can read more about it here:

https://tiddlywiki.com/prerelease/#Performance%20Instrumentation

Roughly, the process would be:

* Use the upgrade tool to upgrade your wiki to v5.1.20-prerelease

* Ensure performance instrumentation is turned on (see control panel “Settings”; and restart after turning it on)

* Open it in a browser and perform some of the actions that you’re finding too slow

* Open the developer console and type $tw.perf.log()

You’ll see a table that shows the filters you’re using ranked in descending order of total execution time.

Best wishes

Jeremy

[A Gloom]

On Friday, May 31, 2019 at 9:08:28 AM UTC-4, Jeremy Ruston wrote:

I ran the wiki under Node.js to generate an index.html file that I then ran standalone in the browser.

> I tested the prerelease and the indexing made things possible that were not before.

 This all has got my curiosity --

I'm looking at large wiki's and interlinking them.  I replaced my home brewed frames content browser with TW to be able to annotate and manipulate the presentation of the indexing of large amounts of external source and work files on local drive as well as online all from one place with the added bonus of editing capability of text, hmll and svg.  Direct copy & pasting in browser makes preparing Google Apps a minimum effort.  I only have to leave my browser to use a graphics or 3D editor for complex work.  That's why when you were assisting me with the svg's with js, they were external-- haf my wikis' content is external.

I wonder what "made things possible that were not before" is..

[Mark S.]

Does field:y include field:text ?

Thanks!

[PMario]

On Monday, June 3, 2019 at 5:33:42 AM UTC+2, Mark S. wrote:

Does field:y include field:text ?

There is a limit of 128 characters. So text field most of the time will be excluded.

-m

[Sean Boyle]

In the spirit of large tiddlywikis, is there a simple way to get a listing of tiddlers by size, sorted largest first?  I would like to do some weeding.

On Friday, May 31, 2019 at 3:50:24 AM UTC-7, Jeremy Ruston wrote:

[Magnus]

Check out https://groups.google.com/d/topic/tiddlywiki/hr75FTeEL_g/discussion

[Mark S.]

Here's some code that runs in the prerelease 5.1.20 (it needs the new operators).

ABSOLUTELY make sure you have a copy of everything in your browser and in your desktop, because if you have a lot of tiddlers, this may crash your browser. You may get a lot of those "A script on this page is slowing down your web page messages.". If so, just keep clicking on "wait". It should finish eventually.

The reason this is so slow is because the nsort won't work on a tiddler with the title "552 My tiddler" as a number (552). So what I did instead was to first find all the counts and sort them, and then match those counts against each tiddler. Crazy inefficient.

For your safety, there is limit[10] in the output filter. Remove this from the filter when you're ready to try it for read.

Currently this will show you the count of all standard tiddlers, including image tiddlers. The image tiddlers are likely to be your biggest.

Have fun, and be careful. But not in that order.

\define numbered-tiddlers()
<$vars pre="[[[[" suf="]]]]" mid=".0 ">
<$list filter="[all[tiddlers]!is[system]]" variable="tid">
<$list filter="[<tid>get[text]length[]]" variable="count">
<$list filter="[<count>]"/>
</$list>
</$list>
</$vars>
\end
\define find-tiddlers-by-number(num)
$num$<br/>
<$vars pre="[[[[" suf="]]]]" mid=".0 ">
<$list filter="[all[tiddlers]!is[system]]" variable="tid">
<$list filter="[<tid>get[text]length[]regexp[^$num$$]sort[]]" variable="tid2">
$num$ == <$link to=<<tid>>><<tid>></$link><br/>
</$list>
</$list>
</$vars>
\end


<$wikify text=<<numbered-tiddlers>> name="sortus" mode="inline">
<$list filter="[enlist<sortus>nsort[title]limit[10]]" variable="cnt">
<$macrocall $name="find-tiddlers-by-number" num=<<cnt>>/>
</$list>
</$wikify>
<$wikify text=<<numbered-tiddlers>> name="sortus" mode="inline">
<$list filter="[enlist<sortus>nsort[title]]" variable="cnt">
<$macrocall $name="find-tiddlers-by-number" num=<<cnt>>/>
</$list>
</$wikify>