Strategies for dealing with large TWs

[Nick Gibbins]

Hi,

I'm currently having difficulties with a rather large TW that I'm constructing, containing roughly 25,000 tiddlers (it's a specialised encyclopedia). Total file size is less than 20Mb; the majority of the tiddlers are small, but contain structured data in custom fields. I'm using TW version 5.1.13 and Firefox 50.1.0 under OS X 10.12.2. The machine I'm using has an SSD.

If I treat my TW as a single file, I get save times in excess of 120s (autosave is therefore not an option).

If I use node.js (v6.9.2) as a backend, opening or closing each tiddler typically takes around 50s (the server console suggests that much of this time is spent on the save task for $:/StoryList). During this time, the browser is unresponsive.

I also fairly often see the following error, typically when opening the TW when the server has already been running for longer than a few minutes.

RangeError: Maximum call stack size exceeded
    at Logger.log ($:/core/modules/utils/logger.js:29:108)
    at Syncer.dispatchTask ($:/core/modules/syncer.js:495:15)
    at Syncer.processTaskQueue ($:/core/modules/syncer.js:417:9)
    at $:/core/modules/syncer.js:425:27
    at $:/core/modules/syncer.js:505:4
    at FileSystemAdaptor.loadTiddler ($:/plugins/tiddlywiki/filesystem/filesystemadaptor.js:201:2)
    at Syncer.dispatchTask ($:/core/modules/syncer.js:496:20)
    at Syncer.processTaskQueue ($:/core/modules/syncer.js:417:9)
    at $:/core/modules/syncer.js:425:27
    at $:/core/modules/syncer.js:505:4

Any and all suggestions would be welcome at this point, including "you fool, you're using TW for things that it wasn't designed for".

Thanks, Nick

[Penguinlay]

I start learning TW to use as personal wiki notebook. Potentially, I would have more than 250k tiddlers over time. So, I need to know the answer to this question. But, first, is it possible to open and save time to be that long as long as we are saving and opening a few tiddlers at a time? Also, Nick, are you opening and editing tiddlers in bulk or a few?

On Thursday, January 26, 2017 at 6:00:47 AM UTC-8, Nick Gibbins wrote:

Hi,

I'm currently having difficulties with a rather large TW that I'm constructing, containing roughly 25,000 tiddlers (it's a specialised encyclopedia). Total file size is less than 20Mb; the majority of the tiddlers are small, but contain structured data in custom fields. I'm using TW version 5.1.13 and Firefox 50.1.0 under OS X 10.12.2. The machine I'm using has an SSD.

If I treat my TW as a single file, I get save times in excess of 120s (autosave is therefore not an option).

If I use node.js as a backend, opening or closing each tiddler typically takes around 50s (the server console suggests that much of this time is spent on the save task for $:/StoryList). During this time, the browser is unresponsive.

[Nick Gibbins]

On Thursday, 26 January 2017 14:20:31 UTC, Penguinlay wrote:

I start learning TW to use as personal wiki notebook. Potentially, I would have more than 250k tiddlers over time. So, I need to know the answer to this question. But, first, is it possible to open and save time to be that long as long as we are saving and opening a few tiddlers at a time? Also, Nick, are you opening and editing tiddlers in bulk or a few?

 

I'm opening and editing tiddlers one at a time, although the tiddlers containing structured data have ViewTemplates applied that involve lookups in about ten dictionary tiddlers, based on the fields of the tiddler; I've used the dictionaries to hold boilerplate text in order to reduce the size of each tiddler and the overall file size.

Nick

[Nick Gibbins]

I should also add that I'm having no problems at all with a subset of my encyclopedia that contains around 2,500 tiddlers; at that scale, it's performing very well indeed.

Nick

[Penguinlay]

I see. I just start learning so that I don't get to dictionary tiddlers yet. Talking about boilerplate text, currently, I am just using macro for those repetitions, custom view templates to avoid fields as best as possible. I will watch out for what TW experts have to say about your question.

[Danielo Rodríguez]

This is one of the main problems of tiddlywiki: everything is synchronous, therefore blocking.

I have suggested in a couple of times to adopt a lore asynchronous paradigm ,which will not block the user interface at least.

But I think it is an almost impossible task because how deeply integrated the synchronous behavior is. Probably a total rewrite would be necessary

[Nick Gibbins]

Agreed - that explains the blocking.

However, it doesn't explain the ~50s that it takes to open a tiddler when using the node backend. If I have a TW containing ~3,000 tiddlers, opening a tiddler for the first time takes less than 0.25s. Build a TW ten times the size, and suddenly the same operation takes more than two orders of magnitude longer.

Nick

[Danielo Rodríguez]

Do you use view templates ? If the answer is yes then you probably use the list widget a lot,which iterates all the tiddlers synchronously several times. Add some hundreds of tiddlers and some dozens of lists and it will grow exponentially in time.

Regards

[Penguinlay]

Nick and Danielo, by using view templates, you guys mean adding custom texts and stuffs using viewTemplate system tag, right? (I just want to clarity if I am thinking of the same thing as I am planning to customize the view templates a lot.)

Also, is there any particular additions to view template that can increase the load time exponentially or at least, more than others?

[Nick Gibbins]

On Thursday, 26 January 2017 22:07:39 UTC, Danielo Rodríguez wrote:

Do you use view templates ? If the answer is yes then you probably use the list widget a lot,which iterates all the tiddlers synchronously several times. Add some hundreds of tiddlers and some dozens of lists and it will grow exponentially in time.

I think that you're right - it's related to the use of the list widget, though not directly in the context of view templates. Where I'm using a list widget to choose between different templates based on tag, I'm using [all[current]tag[Foo]] as the filter; if my understanding of core/modules/filters/all/current.js and filter evaluation is correct, that shouldn't be iterating over all the tiddlers in the store.

On the other hand, elsewhere I have a list widget with a filter expression that uses the listed operator, and that certainly will be iterating over all tiddlers (given that I can't see any evidence of an inverted index in findListingsOfTiddler()), so that's almost certainly causing my woes. It's a shame that such operators don't carry health warnings in the documentation.

Thanks for your help, and back to the drawing board.

Nick

[wimm]

Sometimes it helps to select a different tab in the sidebar before opening a new tiddler. Especially recent is cpu hungry tab.
You might even opt for an new empty tab Create tiddler +, title: Fast, tag with $:/tags/sidebar (choose from dropdown for correct capitals), contents some hint //fast mode//.
kr wimm

[Erwan]

Hi,

In case anyone finds this thread later, it might be worth mentioning that extensively using tags puts a strain on large wikis. Using fields instead (everywhere where possible/relevant) can significantly improve the overall performance. see a case study at:
http://erwanm.github.io/tw-community-search/#TagsOptimizations

Erwan

On 26/01/17 14:00, Nick Gibbins wrote:

Hi,

I'm currently having difficulties with a rather large TW that I'm constructing, containing roughly 25,000 tiddlers (it's a specialised encyclopedia). Total file size is less than 20Mb; the majority of the tiddlers are small, but contain structured data in custom fields. I'm using TW version 5.1.13 and Firefox 50.1.0 under OS X 10.12.2. The machine I'm using has an SSD.

If I treat my TW as a single file, I get save times in excess of 120s (autosave is therefore not an option).

If I use node.js as a backend, opening or closing each tiddler typically takes around 50s (the server console suggests that much of this time is spent on the save task for $:/StoryList). During this time, the browser is unresponsive.

I also fairly often see the following error, typically when opening the TW when the server has already been running for longer than a few minutes.

RangeError: Maximum call stack size exceeded
    at Logger.log ($:/core/modules/utils/logger.js:29:108)
    at Syncer.dispatchTask ($:/core/modules/syncer.js:495:15)
    at Syncer.processTaskQueue ($:/core/modules/syncer.js:417:9)
    at $:/core/modules/syncer.js:425:27
    at $:/core/modules/syncer.js:505:4
    at FileSystemAdaptor.loadTiddler ($:/plugins/tiddlywiki/filesystem/filesystemadaptor.js:201:2)
    at Syncer.dispatchTask ($:/core/modules/syncer.js:496:20)
    at Syncer.processTaskQueue ($:/core/modules/syncer.js:417:9)
    at $:/core/modules/syncer.js:425:27
    at $:/core/modules/syncer.js:505:4

Any and all suggestions would be welcome at this point, including "you fool, you're using TW for things that it wasn't designed for".

Thanks, Nick

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/ea3947a7-c49f-4102-9b2b-a24e05810f02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[@TiddlyTweeter]

Erwan

Excellent bit of research you did on TiddyWiki Performance -- I bookmarked it to study later.

FYI, I been playing with TiddlyWiki as an e-pub format. Once the file gets beyond a smaller size and complexity performance issues begin to matter. I'm still not sure if its really viable for enhanced e-pubs when they get larger. Structurally its ideal, allowing easy annotation, bookmarking etc. But possibly not in terms of performance.

Best wishes
Josiah

[Mark S.]

The entire  bible broken out by chapters only runs around 6 megs, and can work reasonably well on phones and tablets. I'm thinking that if people have texts larger than that then must be including images which quickly inflate the size. One could externalize the images to reduce the size of the HTML component. This is what epub, mobi, and MHT do. They have the text in a set of HTML files and the images separate -- but the whole thing is zipped up into an archive format so it seems to be just one file.

-- Mark

[@TiddlyTweeter]

Ciao Mark S

Mark S. wrote:

The entire bible broken out by chapters only runs around 6 megs, and can work reasonably well on phones and tablets.

Great news. Is that TW bible available online? I'd really like to look at it. Its seeing something of decent size that actually works adequately that would likely help me most.

FYI, for test purposes I work on the novel Great Expectations by Dickens. Its to serve a project I head up so the use case need is quite well defined. I'm chunking it to 1 para = 1 tiddler. This will permit the kind of fine-grained bookmarking & annotation needed.

Its created via iterative use of regular expressions to the input text. Tiddlers are created directly in tid format, rather than going via JSON, then injected into a final TW. Its about 40,000 tiddlers. About 4.5 Mb size in an "empty" TW5. Currently it has far more added fields than needed because I'm still trying to work out the best way to organise the tiddlers. Final size could probably be reduced to about 3 Mb.

Mark S wrote:

I'm thinking that if people have texts larger than that then must be including images which quickly inflate the size. One could externalize the images to reduce the size of the HTML component. This is what epub, mobi, and MHT do. They have the text in a set of HTML files and the images separate -- but the whole thing is zipped up into an archive format so it seems to be just one file.

There are currently no images. Ideally I'd like to incorporate illustrations. Since they are line drawings there is the option to look at their conversion to SVG. But this should wait till I get the rest working as fast as it can. They could also be external. ??I don't think that would be a major issue?? because its just the browser loading them and there is no need for special indexing of them?

The main issue seems to be how to optimally combine decentish performance with the paragraph level needs for annotation and bookmarking (i.e. very high Tiddler numbers that need dynamically organising)--as well as the tools to be able to utilise these effectively.

Erwan's Notes on Performance have proved useful in helping me understand some of the performance issues I need to look at. I'm still some way off producing a proper viable working copy.

Your comment about many e-pub formats actually being zip files I was aware of, though have not yet looked at how they work. I doubt I would have the competence to actualise that methodology, though its an interesting thought whether TW could be bundled that kind of way too for superior e-pubs.

Thankyou for your helpful thoughts.

Best wishes
Josiah

[Jeremy Ruston]

Hi Josiah

On 13 Mar 2017, at 12:35, @TiddlyTweeter <tiddly...@assays.tv> wrote:

FYI, for test purposes I work on the novel Great Expectations by Dickens. Its to serve a project I head up so the use case need is quite well defined. I'm chunking it to 1 para = 1 tiddler. This will permit the kind of fine-grained bookmarking & annotation needed. 

Its created via iterative use of regular expressions to the input text. Tiddlers are created directly in tid format, rather than going via JSON, then injected into a final TW. Its about 40,000 tiddlers. About 4.5 Mb size in an "empty" TW5. Currently it has far more added fields than needed because I'm still trying to work out the best way to organise the tiddlers. Final size could probably be reduced to about 3 Mb. 

You can use the text-slicer edition to do this:

1. Visit http://www.gutenberg.org/ebooks/1400 and download the linked HTML file by right clicking and selecting “Save link” or “Save as"

2. Visit http://tiddlywiki.com/editions/text-slicer/

3. Drag the HTML file into the TW window

4. Click the “Import” button

5. Open the newly imported tiddler

6. Click the scissors icon in the tiddler toolbar

7. You should see the novel laid out in the left hand column as a collapsible table of contents. Clicking on an individual paragraph or heading will open that tiddler in the main story river

Best wishes

Jeremy