File-based TW on node.js crashes when the number of tiddlers is more than ~2000

[solex]

Hi all,

I am rather new to TiddlyWiki, and I've been doing some experiments recently with the automatic importing of data to a file-based wiki.

But I found that when serving a wiki with a few thousand tiddlers, the node server crashes with the following error, while in the middle of dispatching "load" tasks:

RangeError: Maximum call stack size exceeded
    at Console.(anonymous function) (internal/console/constructor.js:243:9)
    at Console.log (internal/console/constructor.js:283:24)
    at Logger.log ($:/core/modules/utils/logger.js:32:25)
    at Syncer.dispatchTask ($:/core/modules/syncer.js:515:15)
    at Syncer.processTaskQueue ($:/core/modules/syncer.js:437:9)
    at $:/core/modules/syncer.js:445:27
    at $:/core/modules/syncer.js:525:4
    at FileSystemAdaptor.loadTiddler ($:/plugins/tiddlywiki/filesystem/filesystemadaptor.js:193:2)
    at Syncer.dispatchTask ($:/core/modules/syncer.js:516:20)
    at Syncer.processTaskQueue ($:/core/modules/syncer.js:437:9)

My system:

tiddlywiki: 5.1.19

node: 11.9.0

MacOs: 10.13.6

I experimented with the total number, and found that the issue occurs somewhere between 1500 and 2000 tiddlers. It looks like the content doesn't matter: the problem occurs even with the simplest possible tiddlers (title only).

I also disabled the TW's logger just to rule out a problem in the logger itself, and the problem persists.

To reproduce the error:

1. Create a fresh "server edition" wiki
2. Populate with 2000 tiddlers (I used a basic script for that)
3. Start a server with a "listen" command
4. Open the TW's page in the browser

2000 tiddlers doesn't seem like a very large number for an in-browser wiki, nor it should be a problem for a server per se. I believe the bottleneck is somewhere in the synchronization process, but so far I wasn't able to pinpoint it exactly.

Has anyone experienced the same problem and is there is anything that can be done to improve the sync performance?

(Also, please let me know if this sort of question should be posted to TiddlyWikiDev)

Thanks!

[PMario]

Hi solex,

I did create 10'000 tiddlers. They all look similar to this one and are about 800 bytes in size.

created: 20190603094535523
modified: 20190603094535523
tags: t-0
title: a-0-0
type: text/vnd.tiddlywiki

123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789

There is no problem for me.

 - I use node v10.16.0 LTS.

 - FF latest

 - TW 5.1.19

 - win 10

The tw server side needs about 20 seconds to start.

It transferes about 12MByte to the browser, which needs about 4 seconds.

So how do you create your files?

-mario

[solex]

Hi mario,

My tiddlers only have title as an attribute, and the corresponding file name.

e.g.

tiddlers/test-tiddler-1.tid

title: test-tiddler-1

Initially they had other fields as well (including content-type), but I reduced them to a minimum while investigating, and the particular set of fields didn't seem to make any difference. 

Also, when the total number is lower (e.g. <1500), the wiki is functioning as expected, so I'm sure the problem is not in the tiddler generation.

The browser doesn't matter too (as expected).

Apparently the only difference is environment (Node version + OS), so perhaps I will try later on a different machine.

Thanks,

solex

[Jeremy Ruston]

Hi Solex

I run many wikis with over 2,000 tiddlers, so I think there maybe something else going on here.

Firstly, Node.js is by default conservative about memory usage, using only a maximum of 512MB of 32-bit systems and 1GB on 64-bit systems. You can manually specify the total amount of memory in MB to be used with the "max-old-space-size" flag. For example:

node --max-old-space-size=16384 <path-to-tiddlywiki.js> <tiddlywiki arguments>

Secondly, the easiest way to cause out-of-memory errors is to try to render a template that includes recursive elements. Are you using a custom raw markup tiddlers? Do you get the same error if you try to build the wiki as a single file? 

Best wishes

Jeremy

[Oleksiy Solyanyk]

Hi Jeremy,

It didn't occur to me to tweak the Node's memory usage, because I didn't think it was the cause of the problem, and I still doubt that processing 2000 one-line tiddlers consumes all of the default 1GB memory - something else must be the problem. But I'll try to experiment with the parameters to see if that at least increases the number.

I did try to close most of the other apps/processes on my machine just in case, to no avail.

My test wiki doesn't use any recursive templates - or anything besides the built-in functionality - because I created a fresh server wiki just to investigate the problem. All I am doing is generating 2000 simple tiddlers with an external script (I described a sequence of steps in my original post).

Yes, the same error occurs when I am trying to build a static version with:

tiddlywiki tw-benchmark --build index

-

Thanks,

solex

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/cdbfa07c-80bd-4ed9-b378-242ab9180f02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[BJ]

maybe there is a problem with the logging...

you can turn off the logs from the syncer in startup.js

$tw.syncer = new $tw.Syncer({wiki: $tw.wiki, syncadaptor: $tw.syncadaptor, logging : false});

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[solex]

Hi BJ,

No, turning off the logs was the first thing I tried. The error was the same (just without the logger-related traceback)

[solex]

It looks like my original post doesn't appear in the email exchange (while visible in Google groups UI).

That probably explains some some questions and suggestions (which I already addressed in the first post)

[Mark S.]

I'm wondering about the data. Could you zip up your tiddlers and post it?

About the email. Google bounces mail from certain domains (or at least that seems to be the problem). 

-- Mark

[Oleksiy Solyanyk]

Hi Mark,

Sure, attached. 

But like I said, it really is just a fresh wiki with a bunch of generated simple tiddlers

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/c320a3b8-3d90-4bfd-8135-9d7518f0111b%40googlegroups.com.

[Mark S.]

It's always useful to see things exactly the way the user sees them.

I got the exact same error you did, when loading it as

tiddlywiki tw-benchmark --listen port=8090

The error occurs after 1862 "loads".  However, when I check the internal contents, it appears that all tiddler titles have actually loaded (I can check these numbers because the advanced search works even when the server has crashed).

But I was able to load the wiki via Bob just fine. So ... maybe Bob doesn't invoke all the "loads"? Or uses a different loading mechanism?

I'm using Windows 7, so my system is very different from yours, but we're getting the same error.

So anyone testing should use your data and the standard mechanism for using it on node.js.

Good luck

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[PMario]

Hi,

With your attachment, I can reproduce the problem. ... I'm pretty sure it's because there is no text field.

So if you could create your tiddlers with at least 3 line breaks it may work.

-m

[solex]

Hi,

I can confirm that the problem is related to text-less tiddlers (as PMario suggested). 

The problem can now be properly restated as: File-based TW on node.js crashes when the number of tiddlers without text field is more than ~1800

The number and structure of the rest of tiddlers doesn't matter, so long as a total number of text-less tiddlers is over the threshold.

The problem appears to be related to the "load" task dispatched by syncer-server-filesystem, because apparently this task is only dispatched for text-less tiddlers, but not for tiddlers with text.

This looks like a bug to me, because "title" is supposed to be the only required tiddler field, and also because the system fails in a non-obvious way (as opposed to, say, refusing to process tiddlers that don't meet certain requirements)

What is the proper way to submit a bug? Should I open a Github issue?

Thanks,

[PMario]

 - use your bold text as heading.

 - Copy your text above to the issue at github.

 - add a link to this thread. So we can find it easily

issue page: https://github.com/Jermolene/TiddlyWiki5/issues

 

Should I open a Github issue?

Yes

[@TiddlyTweeter]

Great bit of collective detective work in this thread!

YES, GitHub is the way to go.

Also, once written (I assume as an "Issue", but maybe its almost a "PR" a writer here could polish?) you can ask back in this thread for "upvotes" to it so folk can indicate its importance.

Best wishes

Josiah

[Jeremy Ruston]

Hi Solex

This isn’t actually a bug, it’s more of an unexpected consequence of the way that lazy loading is implemented: in client-server mode, if the browser references a skinny tiddler (one that lacks a text field) then it requests the body from the server. It’ll only do it once for each tiddler but in a case like this with thousands of skinny tiddlers it looks like it can overload the server.

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6cc309c6-7c3e-459c-b41c-c15074bc134f%40googlegroups.com.

[solex]

Hi Jeremy,

I just opened an issue on Github. Please feel free to close if you believe it's not a bug.

However, I do believe this needs to be at least documented. Currently there is no reason to expect this crash, and the cause of the problem is not immediately obvious.

Perhaps a note could be added somewhere that skinny tiddlers are treated somewhat differently by the system, and that one shouldn't create too many of them.

(If I knew that, for instance, I could have easily avoided the problem altogether by simply adding a few newline characters to my tiddlers)

-

Thanks

On Tuesday, June 4, 2019 at 3:19:00 PM UTC+3, Jeremy Ruston wrote:

Hi Solex

This isn’t actually a bug, it’s more of an unexpected consequence of the way that lazy loading is implemented: in client-server mode, if the browser references a skinny tiddler (one that lacks a text field) then it requests the body from the server. It’ll only do it once for each tiddler but in a case like this with thousands of skinny tiddlers it looks like it can overload the server.

Best wishes

Jeremy.

On 4 Jun 2019, at 12:44, @TiddlyTweeter <Tiddly...@assays.tv> wrote:

Great bit of collective detective work in this thread!

YES, GitHub is the way to go.

Also, once written (I assume as an "Issue", but maybe its almost a "PR" a writer here could polish?) you can ask back in this thread for "upvotes" to it so folk can indicate its importance.

Best wishes

Josiah

On Tuesday, 4 June 2019 12:36:02 UTC+2, solex wrote:

What is the proper way to submit a bug? Should I open a Github issue?

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[TonyM]

Some

A question. Rather than adding newlines to text, adding nothing but actually something in each tiddler and thus defeating the possibility of using the filter [all[current]has[text]] to test if text has content.

Would populating a field in each tiddler do the same? or must it be the text field?

If you must use the text field perhaps insert something like "batch creation: batchname", then you could list items with a text field with prefix[batch creation:] or prefix[batch creation:batchname] if you needed to identify thousands for batch deletion.

Regards
Tony

[@TiddlyTweeter]

Ciao TonyM

Simple query: Why SHOULD you have to populate anything?

Josiah, x

[@TiddlyTweeter]

Jeremy Ruston wrote:

This isn’t actually a bug, it’s more of an unexpected consequence of the way that lazy loading is implemented: in client-server mode, if the browser references a skinny tiddler (one that lacks a text field) then it requests the body from the server. 

My impression reading the thread is that the "break-down" occurs at a specific point?

I mean, a server overload I can't see happening at a specific number on different machines. That has to be code?

I remain hazy whether this is able to be replicated at a set number? But that was my naive first impression--that it does?

Side thoughts

Josiah

 

[solex]

TonyM,

It has to be text. The crash occurs regardless of the number and the content of other fields.

If I understood Jeremy correctly, a tiddler with no text field is considered a "skinny tiddler" and is treated differently by the sync process, thus causing the memory overload when the number of such tiddlers is in thousands. 

[Mark S.]

I'm not sure what the "1800" tiddlers is based on. In my tests, all the tiddler titles loaded. Behind the scenes, apparently the server was stilll fishing for tiddler text. It must have been using an internal array to save results while waiting for (non-existent) data to load. The server crashed after the server had reported 1862 load attempts. But that's just as far as the reporting got. Internally, it may have had all the load events queued. The output reporting would have just lagged. So, I think,it would be more about machine processing maching. I'm running at 3.2 Ghz. I suspect solex's processing speed is similar.

The way I captured the report is to use the > operator to funnel the output of the node launch into a text file. Then I could look at the file and count how many load entries there were. solex could repeat the process with his own device. It would be a doubtful coincidence if our count was exactly the same. In that case, maybe there's a pre-set number of threads?

-- Mark

[solex]

Mark,

I am not sure the exact number matters much.

So far anyone who tried to reproduce it with 2000 tiddlers got the server to crash, so the issue is reproducible on most systems.

I would expect the exact number to differ depending on the available memory and processing power, but the gist of the problem is that the system cannot handle a few thousands of skinny tiddlers while being quite capable of handling tens of thousands of "normal" tiddlers.

[Mark S.]

I just realized something Jeremy said. This is happening because it's acting as if the tiddlers have been lazy-loaded when they actually

haven't been.  So ... If you use the lazy-load option:

   root-tiddler=$:/core/save/lazy-all

then the Wiki will load fine without crashing. So if you really want to load this set, you can do so.

-- Mark

[Mark S.]

I notice that when TW makes it's own "empty" tiddler, there is one line after the last header field. It might be interesting to see if you get different results if your test tiddlers have that blank line.

[Mark S.]

After lazing-loading the text data, I populated all the text fields. Then I deleted the text fields.

Outcome:

1. The tiddlers created have a different kind of line return then your test data (maybe \r\n vs \r ?)

2. The tiddlers created have a line return after the last header line, adding one blank line on the end

3. The tiddlers can now be loaded without the lazy-load option.

HTH

[TonyM]

Josiah,

My answer to your question is

Should you need to as a workaround

However depending on how the tiddlers are to be used it may provide additional functionality.

More directly the problem at hand its suggested it relates to skinny tiddler handling. A feature one may want everyday but which tests resources in large batches. If its a code bug we fix it, if its external in node perhaps simply a short or configurable pause every 1000 tiddlers could be sufficient.

Regards
Tony