TiddlyWiki starting time is sooooo slow

Florian Cauvin

unread,

Dec 6, 2019, 6:33:21 AM12/6/19

to TiddlyWiki

On my RaspberryPi 3 with Raspian and NodeJs:

$ time tiddlywiki --version
5.1.21

real 1m2,309s
user 1m2,727s
sys 0m0,511s

Once TiddlyWiki is "loaded" it works well and is responsive.... but >1 minute to start up? Is that normal???

Jeremy Ruston

unread,

Dec 6, 2019, 6:50:54 AM12/6/19

to TiddlyWiki

Hi Florian

Do you have a large number of tiddlers in your wiki?

The startup time under Node.js hasn’t had much optimisation attention, and I think there’s quite a lot of scope to improve things.

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/34128edc-f5b0-4154-9077-2d38e79d1eee%40googlegroups.com.

Florian Cauvin

unread,

Dec 6, 2019, 8:32:02 AM12/6/19

to TiddlyWiki

Hi Jeremy,

I have a few hundreds tiddlers, but I don't think that's a huge amount of content.

Starting time is not a show stopper (as once it's up and running, TiddlyWiki server performs very well), but it is definitely an annoyance which introduce friction in my workflow because I maintain many personal wikis and serve one wiki at a time (when I want to browse or edit it). Thus I often stop and then start Node.js to switch the current wiki I'm working with. In this case, it'd be very useful if it would load fast (which I guess should be possible as TW shouldn't need to deeply process all tiddlers' content just to start up).

Cheers!

On Friday, December 6, 2019 at 1:50:54 PM UTC+2, Jeremy Ruston wrote:

Hi Florian

Do you have a large number of tiddlers in your wiki?

The startup time under Node.js hasn’t had much optimisation attention, and I think there’s quite a lot of scope to improve things.

Best wishes

Jeremy.

On 6 Dec 2019, at 11:33, Florian Cauvin <floria...@gmail.com> wrote:

On my RaspberryPi 3 with Raspian and NodeJs:

$ time tiddlywiki --version 5.1.21 real 1m2,309s user 1m2,727s sys 0m0,511s

Once TiddlyWiki is "loaded" it works well and is responsive.... but >1 minute to start up? Is that normal???

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

Arlen Beiler

unread,

Dec 6, 2019, 8:52:43 AM12/6/19

to TiddlyWiki

In my experience, there are two things which are the main startup bottlenecks.

The first is that evalSandboxed (in boot.js) runs every module in a new context. If you change it to all of them using one separate context which you create outside the function, it should speed things up some. This one may vary between node versions, but it's worth a shot.

The second is loading the hundreds of tiddlers that make up the core. There is a feature of the boot process that fixes this. Replace the contents of the core folder with this file (assuming your using Tiddlywiki5.1.21) https://github.com/Arlen22/TiddlyWiki5-production/blob/master/5-1-21/core/plugin.info. That way it's just loading one file and doing a simple JSON.parse.

That's the main ones I've found.

Arlen

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6513caf7-b1a4-4290-8abb-4718c3c9e292%40googlegroups.com.

Florian Cauvin

unread,

Dec 6, 2019, 10:28:24 AM12/6/19

to TiddlyWiki

Thanks Arlen for the suggestions:

The first is that evalSandboxed (in boot.js) runs every module in a new context. If you change it to all of them using one separate context which you create outside the function, it should speed things up some. This one may vary between node versions, but it's worth a shot.

I located the file and function call you're refering to but I'm not familiar with Javascript, Node, or the internals of TiddlyWiki. Thus I don't feel confident trying to modify that code. Hopefully, a TiddlyWiki developer could look into this?

The second is loading the hundreds of tiddlers that make up the core. There is a feature of the boot process that fixes this. Replace the contents of the core folder with this file (assuming your using Tiddlywiki5.1.21) https://github.com/Arlen22/TiddlyWiki5-production/blob/master/5-1-21/core/plugin.info. That way it's just loading one file and doing a simple JSON.parse.

Alright, I replaced all content of /usr/local/lib/node_modules/tiddlywiki/core with that file you linked to. It seems to have shaved a few seconds off the startup time which remained over 56 seconds, so while it seems to help a bit, there seems to be a larger issue.

TonyM

unread,

Dec 6, 2019, 9:29:38 PM12/6/19

to TiddlyWiki

Florian,

Add a Splash screen to startup if not already and consider the larger part of the load time could be the final render, depending on what is visible at startup. Minimise the tiddlers opened, hide the sidebar and more. Test a load with minimal opening and no features and unusual plugins active to see if this render time is a contributor.

There are also performance indicators that can be activated to investigate this (Which I have not yet used.)

Regards

Tony

Arlen Beiler

unread,

Dec 6, 2019, 10:48:57 PM12/6/19

to TiddlyWiki

I just realized he was timing the runtime of the --version flag, which would literally load only the core plugin and nothing else. So that's almost 90 seconds to just load the core.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/654d317b-af72-4e89-aa71-2ceabb26c17a%40googlegroups.com.

TonyM

unread,

Dec 7, 2019, 12:18:16 AM12/7/19

to TiddlyWiki

that would mean something is broken then?

Arlen Beiler

unread,

Dec 7, 2019, 8:51:42 AM12/7/19

to tiddl...@googlegroups.com

No. It’s a problem I’ve run into before. Slow processor speed is the main culprit I think. What version of node are you using?

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/629be1bd-3763-419a-a5a7-dd94d3679b1f%40googlegroups.com.

Chuck R.

unread,

Dec 7, 2019, 10:54:57 AM12/7/19

to TiddlyWiki

On Saturday, December 7, 2019 at 12:18:16 AM UTC-5, TonyM wrote:

that would mean something is broken then?

Not necessarily broken with TW. The Rpi 3 is limited to 1-2GB of RAM AFAIK and we don't know how much RAM the OP has. One question I have is, if the TW file is a zip file (my assumption), then on average, all the compression of the text (which includes user-entered content and source code) is about 70%. So a 1GB compressed file compressed at 70% (or is 30% of the original size) has an uncompressed size of 1GB/0.30 or 3.3GB, beyond the RAM capacity of the RPI3. So when the Raspbian OS runs out of RAM, it starts paging RAM to a file (possibly on a micro SD card) frantically, and that's why it takes so long.

Also, how big is each tiddler and how many images are in the TW file, and how big are the image files? Those are all variables I'd like to know about.

Also the Rpi 3 is a low end CPU and it's not that fast.

So I'd like to know:

How much RAM does the RPi3 have?
Is the OS on a micro SD card, USB hard drive, or something else?
How big is the TW html file (which is the compressed size)?

Another question is, when TW is opened (and it sounds like the OP is using the node.js version of TW), how often is the disk accessed? Since Rpi could be running off a micro-sd card, or USB hard drive. Throughput of USB is not that great and has never, IME, reached theoretical speeds. USB throughput IME has only reached 10% of theoretical speeds on average, and not much more than that.

Chuck R.

unread,

Dec 7, 2019, 10:56:03 AM12/7/19

to TiddlyWiki

This would be an opportunity to have a FAQ for TW problems, especially for tiny Single Board Computers (SBCs) with limited RAM.

Arlen Beiler

unread,

Dec 7, 2019, 11:41:59 AM12/7/19

to tiddl...@googlegroups.com

You can google the specs for the raspberry pi 3, but it’s about what you described. And yes, it’s either a USB or MicroSD that it’s reading from, but that problem was already eliminated by the core plugin file fix above and it still takes 56 seconds for TiddlyWiki Node to load the core and run the version command, so there’s more to be investigated. It could still be paging to disk, as you mentioned, but the TiddlyWiki boot sequence will only read about 7 files in this case.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f7b1fc7c-68bf-4db5-8cfb-5de854b3e78d%40googlegroups.com.

Arlen Beiler

unread,

Dec 7, 2019, 1:31:11 PM12/7/19

to TiddlyWiki

Ok, here's the evalSandboxed code currently. Well, almost, there is one change inside the function, but just ignore that.

/*
Run code in a sandbox with only the specified context variables in scope
*/
$tw.utils.evalSandboxed = $tw.browser ? $tw.utils.evalGlobal : function(code,context,filename) {
  var sandbox = $tw.utils.extend(Object.create(null),context);
  vm.runInNewContext(code,sandbox,$tw.utils.evalFilename(filename));
  return sandbox.exports;
};

In the first line, change $tw.browser to true, like this:

/*
Run code in a sandbox with only the specified context variables in scope
*/
$tw.utils.evalSandboxed = true ? $tw.utils.evalGlobal : function(code,context,filename) {
  var sandbox = $tw.utils.extend(Object.create(null),context);
  vm.runInNewContext(code,sandbox,$tw.utils.evalFilename(filename));
  return sandbox.exports;
};

See if that improves anything.

Arlen

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/b987363a-17f2-4444-8bb9-d6aa43c7b841%40googlegroups.com.

Florian Cauvin

unread,

Dec 8, 2019, 3:52:24 PM12/8/19

to TiddlyWiki

Arlen,

Yes, wow, this change made a dramatic improvement in TiddlyWiki startup time which was reduced from over a minute to about 3 seconds. Fantastic.

So, can I keep TW running his way, is it somehow "less safe" or ...?

On Saturday, December 7, 2019 at 8:31:11 PM UTC+2, Arlen Beiler wrote:

Ok, here's the evalSandboxed code currently. Well, almost, there is one change inside the function, but just ignore that.
/*
Run code in a sandbox with only the specified context variables in scope
*/
$tw.utils.evalSandboxed = $tw.browser ? $tw.utils.evalGlobal : function(code,context,filename) {
var sandbox = $tw.utils.extend(Object.create(null),context);
vm.runInNewContext(code,sandbox,$tw.utils.evalFilename(filename));
return sandbox.exports;
};

In the first line, change $tw.browser to true, like this:

/*
Run code in a sandbox with only the specified context variables in scope
*/
$tw.utils.evalSandboxed = true ? $tw.utils.evalGlobal : function(code,context,filename) {
var sandbox = $tw.utils.extend(Object.create(null),context);
vm.runInNewContext(code,sandbox,$tw.utils.evalFilename(filename));
return sandbox.exports;
};

See if that improves anything.

Arlen

On Fri, Dec 6, 2019 at 10:28 AM Florian Cauvin <floria...@gmail.com> wrote:

Thanks Arlen for the suggestions:

The first is that evalSandboxed (in boot.js) runs every module in a new context. If you change it to all of them using one separate context which you create outside the function, it should speed things up some. This one may vary between node versions, but it's worth a shot.

I located the file and function call you're refering to but I'm not familiar with Javascript, Node, or the internals of TiddlyWiki. Thus I don't feel confident trying to modify that code. Hopefully, a TiddlyWiki developer could look into this?

The second is loading the hundreds of tiddlers that make up the core. There is a feature of the boot process that fixes this. Replace the contents of the core folder with this file (assuming your using Tiddlywiki5.1.21) https://github.com/Arlen22/TiddlyWiki5-production/blob/master/5-1-21/core/plugin.info. That way it's just loading one file and doing a simple JSON.parse.

Alright, I replaced all content of /usr/local/lib/node_modules/tiddlywiki/core with that file you linked to. It seems to have shaved a few seconds off the startup time which remained over 56 seconds, so while it seems to help a bit, there seems to be a larger issue.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

Arlen Beiler

unread,

Dec 9, 2019, 8:07:30 AM12/9/19

to TiddlyWiki

It is essentially a sandbox for each module. Disabling it is a little less safe but not by much. The only way it matters is if a plugin or module would modify a global like Promise or Buffer, but data folders and plugins also have full file system access, which is much more powerful, so there's actually not much that it can protect from. So for your application the benefits definitely outweigh the risks.

I'm honestly not sure that this offers much additional security at all, especially since the Node docs specifically say not to use it as a security measure.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/2495b4d0-ff89-474e-8c67-ffae52549c9e%40googlegroups.com.

Reply all

Reply to author

Forward