Limits of Tiddlywiki 5.1.19?

[Chuck R.]

Does anyone know of the limits of TW 5.x?

1. What is the max number of tiddlers in a file?
2. What is the max amount of text that can be in a tiddler?
3. Is there a max file size for the html file itself?

Thank you.

[Chuck R.]

Sorry, I'm on 64-bit Windows 10 and 32-bit Windows 7. I can use different browsers like Firefox and Chrome and I'm currently editing a TW 5.1.19 on Tiddlyspot.com. So my question would be in that context.

My intention is to use this TW file for many years. My current note file in Keynote is about 3GB but that's because there are a few large images in it. I don't know how many images I will have in my current TW 5.x file.

[Jed Carty]

The limits are not going to come from TiddlyWiki itself, there may be limits to the size of a wiki on tiddlyspot.com, and you have limits set by your browser and system memory.

[TiddlyTweeter]

Ciao Chuck

There is a problem with how your question is phrased :-) If you are a newbie then I understand it is not so easy to "get it" at first :-)

What is the max number of tiddlers in a file?

There is no limit per se. It depends on what you are trying to do. 

The main thing to understand is TW is dynamically built and the performance will depend on how many dynamic scripts you are running. 

Overall: Size is no more limiting than usual in a web-page.

What is the max amount of text that can be in a tiddler?

Same point. It depends on how much dynamic you have going on in it. Simple text Tiddler size is not usually an issue. Its more of a Browser issue.

Is there a max file size for the html file itself?

No :-) Not much less than a normal web page.

Best wishes

TT

[Chuck R.]

Thanks TiddlyTweeter, but what issues could I expect from the browser side? On a 16-bit system there is a limit on how high an integer will go, it will count up to a 16-bit value. On a 64-bit system there is a limit on the value of unsigned integers, but it's really high, in the billions at least.

We've had some questions here that someone gets some type of error and they have 60,000+ tiddlers in a single file. The max value for an unsigned int is 65536. This is only relevant if TW5 uses unsigned integers to give a unique number to each tiddler. What if it uses a base 36 or base 64 numbering system? Or something else?

So if I'm using the most recent version of Chrome, and it's a 64-bit browser, what issues could I expect, if any, from the browser or OS end of things?

Does the version of Javascript used by TW5 have any limits? Has it been updated to support 64-bit operating systems and 64-bit data types?

Thanks.

[Chuck R.]

Oops, the NUMBER of values in a 16-bit system is 65536, the largest value, starting at 0, would be 65535.

[Qalisto]

Agree that this would be useful information so as to plan multiple TWs based on reasonable data segregation strategy.  For example, I am creating a Contacts TW and a Business DB.

I am particularly keen on keeping as much Data out of the Cloud as possible unless it is an encrypted TW backup.

As much data as I can isolate locally via TW, is my objective.

=== > Q < ===========================================

[Mark S.]

I'm the guy with 60,000 plus tiddlers. That turned out to be an issue with tag optimization. I changed my

filters to not use the tag operator, and now my TW app works reasonably well even on a Kindle Fire.

You might do a search on javascript abilities. A quick search suggested that javascript arrays could hold 4 billion entries. You will be out of browser memory long before that happens.

I think it's hard to document the exact range of abilities of TW because there are so many platforms, hardware, OS, browser differences.

To me the impressive thing is that we have someone in the forum who actually runs a store on TW !

Hardware memory seems to be a major consideration. A file may be only 10 megs, but the javascript it runs

appears to need to spread out and take several times that amount of space. Or at least that's what I've

observed.

If I was seriously considering creating a mammoth app in TW, I think I would start by figuring out the basic

relationships I would want between tiddlers (fields, tags). Create a few sets of tiddlers. Then run some process

to clone thousands of test tiddlers. Then you would have a feel for how things were going to work. At this

point, avoid using the tag operator in filters were there will be huge numbers of tiddlers tagged with that tag.

But 5.1.19 isn't the TW version to be using. Optimizations were made in 5.1.20 that allow for larger TW files. So

it's probably worth upgrading, unless there is something specific about 5.1.19 that you need.

Good luck!

[TonyM]

Chuck,

As you can see from other answers it is similar to "How long is a piece of string?"

• There are multiple tricks and optimisations, you can use a server and have external media, lazy loading and more. 
• Clearly with a really massive tiddlywiki you may need to start making choices. choices that are influenced by if this internet available, is it private on local disk or LAN only. 
• Since I use a lot of tiddlywikis I allow my browsers to use more RAM but I would not expect others to configure their browsers this way.

Marks suggestion of testing it is good, try and make it shareable so people can help. 

Of note when mark raised this issue of the 66,000 tiddlers I found a performance improvement as well which was different from the one Mark finally used. So if he gets a new bottle neck I can help him get further performance improvements. 

Unlike other environments there are a lot of ways to manipulate tiddlywiki to reach a destination, so I think it is worthwhile trusting the platform even although you may still ask questions we have not answered yet.

This is a vital community and you can count on us to help. 

In the unlikely case tiddlywiki proves not to be adequate, you will still have organised, designed, prototyped and studied your needs such that choosing something else will be much easier.

Regards

Tony

[TonyM]

Chuck,

Tiddler titles are a strings with an effectively infinite number of values. No need to ask about these as a limitation unless you have a specific numerical issue in the very large numbers. What in your needs do you feel would challenge this?

For mathematically intensive applications the addition of a javascript maths library as already available in a number of plugins can lift these limits higher, and for the hyper mathematical algorithms I can only speculate on, if you understand the algorithm you could represent it in different ways to overcome almost any barrier with only computational time your restriction (This is my informed speculation).

Even with Javascript limitations you can use methods to define new types and overcome most limits that you may come across, otherwise you may need a super computer.

Regards

Tony

[PMario]

On Monday, November 25, 2019 at 4:44:38 PM UTC+1, Chuck R. wrote:

Oops, the NUMBER of values in a 16-bit system is 65536, the largest value, starting at 0, would be 65535.

Hi,

The number range in JavaScript we use is: (2^53 - 1)

Number.MAX_SAFE_INTEGER
9007199254740991

So that's not a real problem for arrays and counters ;)

I think, the limiting factors are file size and embedded images. ... but images can be "externalized".

-mario

[PMario]

On Monday, November 25, 2019 at 4:43:14 PM UTC+1, Chuck R. wrote:

...

Does the version of Javascript used by TW5 have any limits? Has it been updated to support 64-bit operating systems and 64-bit data types?

Yes!

-------------------

JavaScript is an "untyped" language, which is an advantage and a disadvantage. ... Based on your question, it is an advantage, because it doesn't matter if you have a 32bit system or a 64 bit system. max_int is the same!

We use what the browser gives us and the browser takes care about the code compatibility.

In JS the most important type is an Object. Almost everything is an object. That's why browsers are highly optimized to work with objects.

Internally the tiddler store is object based, which gives us maximum speed and flexibility.

-mario

[PMario]

On Tuesday, November 26, 2019 at 10:56:23 AM UTC+1, PMario wrote:

On Monday, November 25, 2019 at 4:44:38 PM UTC+1, Chuck R. wrote:

Oops, the NUMBER of values in a 16-bit system is 65536, the largest value, starting at 0, would be 65535.

Hi,

The number range in JavaScript we use is: (2^53 - 1)

add link

[Chuck R.]

> JavaScript is an "untyped" language, which is an advantage and a disadvantage. ... Based on your question, it is an advantage, because it doesn't matter if you have a 32bit system or a 64 bit system. max_int is the same!

I'm a Perl programmer, not a JS programmer. Would you mind explaining why the limits for a JS counter are the same for a 32-bit vs 64-bit OS? I would be eager to learn something new today by tapping into your wonder fount of wisdom. :)

Perl comes in several variations: 16, 32, and 64-bit. Counter value limits are based on the lowest common denominator of the OS or Perl bit width. So running 16-bit Perl on a 32-bit OS would mean I'm still limited to 16-bit values via Perl.

Perl only has 3 basic "types" but officially, it is untyped. It has the $a variables (called "scalar" variables), which can hold an integer, float, string or even a memory address (pointer). The other basic types are array and hash. So Perl has limits on the value of integers it can hold in $a scalar based on if Perl is 16, 32 or 64-bit. Which I found out the hard way using a vendor version of Perl which is 16-bit, where I found that I cannot read a 5GB Excel file into memory on a Windows Server 2012 machine with 32GB of RAM because the XLSX read routine must make a LOT of pointers as it expands the XLSX file into memory.

(An XLSX file is actually a zip file with a bunch of stuff in it. So on average, just by unzipping the file, the file doubles in size.)

Thank you!

[PMario]

On Tuesday, November 26, 2019 at 1:03:33 PM UTC+1, Chuck R. wrote:

> JavaScript is an "untyped" language, which is an advantage and a disadvantage. ... Based on your question, it is an advantage, because it doesn't matter if you have a 32bit system or a 64 bit system. max_int is the same!

I'm a Perl programmer, not a JS programmer. Would you mind explaining why the limits for a JS counter are the same for a 32-bit vs 64-bit OS? I would be eager to learn something new today by tapping into your wonder fount of wisdom. :)

From: MDN docs

The MAX_SAFE_INTEGER constant has a value of 9007199254740991 (9,007,199,254,740,991 or ~9 quadrillion). The reasoning behind that number is that JavaScript uses double-precision floating-point format numbers as specified in IEEE 754 and can only safely represent numbers between -(253 - 1) and 253 - 1.

-m

[TiddlyTweeter]

Chuck R. wrote:

I'm a Perl programmer, not a JS programmer. 

Just a footnote & encouragement. Having a Perl background will help, I think, if you want to delve TW code.

Many parts of TiddlyWiki, behind the scenes, use regular expressions extensively. Something Perl-istas tend to be very good at.

Best wishes

TT

[Chuck R.]

Thank you Pmario!