Single TW5 with 12500 tiddlers. How to make it work faster?

[Siniy-Kit]

Hi! I make little online stores on Tiddlywiky. When I have not more than 2000 items in my store, it works fast, but customer wants to have 12500 items with many parameters...

so I made a demo of this shop https://heeg.ru/shop_test244.html it is 21.8 Mb (very big) and it works slow.  

How to make it work fast? 

[cpjobling]

Perhaps use the Node.js Tiddlywiki - tiddlers are then stored in individual files and (I assume) loaded only on demand.

[Pit.W.]

Hi Siniy-Kit,

I am running a local Node.js TW with 6.000+ tiddlers. No internal PDFs. Lazy-loading all.
Speed is ... so,so..., but ok. I did not hit a brick wall so far.  Its on a Laptop which is ...  quite powerfull...

But I also have many growing singe-file TWs on normal PCs.

I cant help you with your original question, but maybe you can help me on the same issue:

What is the hardware you are using?

What is your operating system?

Kind regards,

Pit.W

Am 2018-08-15 um 16:38 schrieb Siniy-Kit:

Hi! I make little online stores on Tiddlywiky. When I have not more than 2000 items in my store, it works fast, but customer wants to have 12500 items with many parameters...

so made a demo of this shop https://heeg.ru/shop_test244.html it is 21.8 Mb (very big) and it works slow.  

How to make it work fast? 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/1148d2b7-4d38-4e54-a9c5-bd24dbbd3f2f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

_________________________________________________________________
________________________________________________________
Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de

[TonyM]

Siniy-Kit,

Is there a way to divide the site by multiple factors such as product types? then ensure the site look the same so user do not even know they are moving to another tiddlywiki. In this case you may need to keep some information in each wiki to assist product search (perhaps we would call these skinny products).

Do you have cloud flare or another caching solution in place? it made a big difference to a site of mine when I did this.

Also performance not tested by me, but NoteSelf can store tiddlers in both a local pouchDB and remote counchdb perhaps this would enhance the performance. The default pouchDB performance could be very good once the site is loaded and this is supported by a "Loading" message.

I have recently had node.JS running on my Cpanel hosting platform (CloudLinix) the various security issues may be addressed in the next release of tiddlywiki.

Best wishes

Tony

[Arlen Beiler]

If you are going to divide it into multiple pages you should definitely figure out a way to load the core and plugins from one URL so it can be cached. This will decrease the file size by 1.5MB or more, depending on your plugins. They can be added to the preloadTiddlers array like normal. I would recommend using a script tag in head with a src attribute that is simply $tw.preloadTiddlers.push({... json tiddler ...}). You can check out how I did it at https://github.com/twcloud/dropbox-datafolder/. Check out the tiddlywiki directory and the src/dropbox-filefunctions.ts file

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/3a1b4dcf-85cd-4252-9924-41ab03eb1024%40googlegroups.com.

[Siniy-Kit]

I use my TW5 as site (online store). So people must have opportunity to open it on their old telephones and notebooks.  The problem is not in speed loading. The problem is slow work of big TW5. All navigating and menus become very slow. I think the problem is very very big DOM tree of my site and it is very difficult for browser to update this tree every time. I think Node.js will not solve this problem....  How can I convert  my single html to Node.js variant on my computer for test?

четверг, 16 августа 2018 г., 4:15:26 UTC+3 пользователь Arlen Beiler написал:

[PMario]

On Thursday, August 16, 2018 at 11:49:38 AM UTC+2, Siniy-Kit wrote:

.. The problem is slow work of big TW5. All navigating and menus become very slow.

I think the problem is very very big DOM tree of my site and it is very difficult for browser to update this tree every time.

No the DOM is not the problem. Most of the time the problem are "slow filters". ... That means filters that work with a huge number of tiddlers every time they do some calculation.

So the general rule for filters should be: .. The first rule should minimize the number then the more complicated rules should follow.

eg: use [tag[]prefix[]] and not [prefix[]tag[]] ...

 

I think Node.js will not solve this problem.... 

That's right. The node server dynamically creates a "single page" file and serves it to the client. So the end result is the same.

"Lazy loading" doesn't help here, because the number of tiddlers isn't reduced, just the tiddler body is removed.

-m

[@TiddlyTweeter]

Ciao Siniy-Kit

As far as I understand it the problem is not the DOM per se, which TiddlyWiki does not handle in the standard way, its the REFRESH mechanism. Everything has to be rebuilt when a change is made. The problem with this is that the more complex the TW gets the worse performance (generally) gets.

Its certainly worth comparing if a Node.js might work better. But its likely there is a limit. If it works better it won't be because of "lazy loading" since, as PMario says, in your usage as the issue centres on Tiddlers invoked in lists/filters. "Lazy loading" relates to simply that the Title only is permanently in memory--and the Tiddler content loads on click. It won't speed up lists or filters.

In brief, at the moment, TW does not scale well.

Since I am a bit of an amateur (I'm not a programmer) others may say otherwise... but I wrote this to hopefully elicit clearer answers.

Best wishes
Josiah

[Mark S.]

Indexing is the key to scalability.

I can imagine a semi-indexed approach that might work:

Pack all the product tiddlers into a single tiddler (perhaps a JSON data tiddler, or perhaps something new). Let's call it tiddlers-tiddler. There's a zip plugin, so possibly the data could even be compressed.

Reference the product tiddlers using one or more data dictionaries, that are organized by product id, description, etc.

Change search look ups to use the data dictionaries.

When a product tiddler is needed, unpack it from the tiddlers-tiddler and then proceed as usual. Unless a customer obstinately insists on seeing ALL products at once, the size of the TW and (more importantly) the total number of deployed tiddlers would be greatly reduced.

Something to think about.

-- Mark

[Jeremy Ruston]

Just to clarify some of the points made by Josiah:

As far as I understand it the problem is not the DOM per se, which TiddlyWiki does not handle in the standard way, its the REFRESH mechanism. Everything has to be rebuilt when a change is made. The problem with this is that the more complex the TW gets the worse performance (generally) gets. 

It’s not really correct to say that TiddlyWiki doesn’t handle the DOM in the standard way. It’s more that it doesn’t use the DOM in the old fashioned way. All modern React apps, for instance, work the same way that TiddlyWiki works.

And it’s not quite right to say that everything needs to be rebuilt when a change is made. The refresh mechanism endeavours to make the minimal, selective updates to the DOM to reflect changes to the store as they occur.

But the conclusion isn’t far off: to be more precise, performance depends on the complexity of the page — ie the results of rendering the $:/PageTemplate tiddler.

Its certainly worth comparing if a Node.js might work better. But its likely there is a limit.

It is *not* worth comparing with performance under Node.js. As other posters have pointed out, loading the wiki from a Node.js server doesn’t affect the performance of what happens in the browser.

If it works better it won't be because of "lazy loading" since, as PMario says, in your usage as the issue centres on Tiddlers could be invoked in lists/filters. "Lazy loading" relates to simply that the Title only is permanently in memory--and the Tiddler content loads on click. It won't speed up lists or filters.

Indeed, execution of filters is a substantial proportion of overall rendering time.

In brief, at the moment, TW does not scale well.

Again, I don’t think that’s hugely helpful as a blanket statement, For example, I’m working with a 65MB TW5 at the moment, and it works fine on my 2013 laptop.

Perhaps better to say that as a TiddlyWiki gets more complex one has to pay more attention to performance.

Since I am a bit of an amateur (I'm not a programmer) others may say otherwise... but I wrote this to hopefully elicit clearer answers.

Indeed, your comments are helpful,

Best wishes

Jeremy.

[@TiddlyTweeter]

Thanks Jeremy. I feel like an utter twat :-) But I don't mind as I think your answers are really useful to see.

[@TiddlyTweeter]

Jermolene: Perhaps better to say that as a TiddlyWiki gets more complex one has to pay more attention to performance.

In the context of this thread what would you advise Siniy-Kit to look towards as a step towards solving his problem?

Best wishes
Josiah

 

[Jeremy Ruston]

Jermolene: Perhaps better to say that as a TiddlyWiki gets more complex one has to pay more attention to performance.

In the context of this thread what would you advise Siniy-Kit to look towards as a step towards solving to his problem?

First, turn on performance instrumentation in control panel settings. Look in the browser console to see the results: the "main refresh” time is the one to watch while you switch between tabs, or perform other activities that trigger a refresh.

Second, use a divide and conquer strategy to find the performance sensitive parts: disable or simplify the page and see what effect it has on the refresh time.

As others have noted, the complexity of filters has an impact on overall performance. Particularly watch out for filters that must search through all tiddlers, and as Mario says, try to order filter operators so that the expensive operations come as late as possible.

Best wishes

Jeremy.

Best wishes
Josiah

 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/84200d45-7996-4b05-b515-55e45addf9fc%40googlegroups.com.

[Siniy-Kit]

The problem is not in filters I use (I think, but not sure).  When I have problems with speed in 2000 items store, BJ from this  forum gives me a very nice plugin $:/plugins/bj/listsRefreshOFF 
 When this plugin enabled, it stops refreshing all $list filter="...  except filters with minlength[1]   this plugin makes TW5 very fast, but practically impossible to edit. So i enable it in final variant.

But when I have 12500 items  enabling of this plugin is not enough.  

I make a little experiment and export  my 13000 tiddlers to JSON  and import them to https://tiddlywiki.com/  it's strange, but tiddlywiki.com works very fast with all this tiddlers  without BJ's plugin.....

четверг, 16 августа 2018 г., 21:57:06 UTC+3 пользователь Jeremy Ruston написал:

[Erwan]

Hi Siniy-Kit,

In case the wiki contains many different tags, you might want to try to use fields instead if/when possible. I found this to significantly improve performance, I did a little study about this a long time ago, in case it helps: http://erwanm.github.io/tw-community-search/#TagsOptimizations.

Regards
Erwan

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/cbc59f4e-d49a-417c-8321-0a93750362c4%40googlegroups.com.

[Arlen Beiler]

So what happens if you import them into the latest empty? Each version of TiddlyWiki has performance improvements, so upgrading to a new version can make a big difference. 

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/cbc59f4e-d49a-417c-8321-0a93750362c4%40googlegroups.com.