Size of a wiki and namespace philosophy

[joearms]

Do you

1) Put all your tiddlers in one name space (ie one TW)

2) Put them into different namespaces (one TW per project)

Tiddlers should be small (in order to make them reusable)

If they are (say) on average c. 300 bytes

we could get 3 tiddlers per KB of 3K tiddlers / MB

or 3M tiddlers per GB

So size isn't a problem - is performance a problem (when searching etc)

Is choosing the name of a tiddler a problem at scale a problem?

(I have elsewhere argued that naming is difficult)

If we imported large numbers of tiddlers with the same name

could we make a disambiguating mechanism (like the wikipedia)

ie X might mean A B C and let the user decide

I think all tiddlers in one namespace might be overkill.

(counter argument - a dictionary has a flat structure and thousands of words in it)

How about *two* levels of naming for example joe@TiddlerName

or myhost.name@TiddlerName?

(( and please NOT multiple levels joe@stockholm@sweden - I hate these))

and an import declaration (see joe & fred)

which exposes the joe and fred namespaces

I'm thinking out loud here - just wondering about names.

Actually sticking everything in one directory make life easy since you don't

have to worry about which directory to put things in - two levels

make life far more difficult - but if the top level is the author then

the decision problem (as I call it) (ie choosing which directory to store a file in)

is solved! 

Any other distinction (say by project) is tricky - ie X could belong to project A or B

or both - so where do we store it???

/Joe

[PMario]

Hi Joe,

On Friday, March 2, 2018 at 6:52:43 PM UTC+1, joearms wrote:

...

Tiddlers should be small (in order to make them reusable)

If they are (say) on average c. 300 bytes

Tiddlers internally do have a little bit of an overhead.

We use several invisible core fields, which also need a bit memory.

created:  20130822170200000 modified:  20130822170200000

creator: name
modifier: name
title: tiddler title
type: text/vnd.tiddlywiki
tags: a b c

So the "header" is already about 150 byte. ... So I think a good approximation is ~1k Byte per tiddler with user content.

we could get 3 tiddlers per KB of 3K tiddlers / MB

3000 tiddlers is ok, even if they are ~1k
 

or 3M tiddlers per GB

This will definitely be a problem and probably cause browser "long running script" messages. Or it may crash.

TiddlyWiki doesn't use a database in the background. Internally its a "simple" JSON structure. So search doesn't use an index. ... There is a "indexing" plugin ...

I did some tests with about 20'000 tiddlers which are also taged. So backlinks have to be calculated. Tag Pills are claculated ...

The program is still usable, but opening a tiddler takes several seconds. Which is "kind" of slow. Saving and loading is even slower.

From my tests

The result is a 20Mbyte html file.

- no crashes at save fromo tiddlywiki.com  ... but 3 times a slow script warning
- it needs about 13 seconds to load from disk
- and about 25 seconds to save with TiddlyFox

If the "long script warning" is not disabled with firefox setting about:config ... It crashed several times while saving
Disabling the long script warning didn't crash it.

So size isn't a problem - is performance a problem (when searching etc)

As I wrote. No index

have fun!
mario

[PMario]

Hi,

On Friday, March 2, 2018 at 6:52:43 PM UTC+1, joearms wrote:

1) Put all your tiddlers in one name space (ie one TW)

2) Put them into different namespaces (one TW per project)

... It depends :) I personally use several TWs.
 

 If we imported large numbers of tiddlers with the same name

could we make a disambiguating mechanism (like the wikipedia)

ie X might mean A B C and let the user decide

TiddlyWiki uses a very "special" mechanisms, where the same name for a tiddler actually is an advantage.

There are 3 tiddler types:

• Shadow Tiddlers ... used as a fallback, if users mess up default values
• System Tiddlers  ... user editable
• Content Tiddlers .. or just Tiddler  ... user editable

For example: If you open the TW ControlPanel: Info: Basics   tab, you'll see

Title of this TiddlyWiki:  TiddlyWiki  <- This text can be edited and will be updated in the top right of the page

If you click the link in Title of this TiddlyWiki:  it will open a tiddler named: $:/SiteTitle .... This is a SystemTiddler
System Tiddlers are hidden in most "end user" facing lists. They start with: $:/ ... Special filters are used to show them.

If you edit and delete that tiddler, the system will use the "core" default tiddler, which is called a ShadowTiddler

The text will be: My TiddlyWiki

Also $:/SiteTitle now contains this text. ... Editing it, will show a "warning", for users to let them know, that they will overwrite a shadow tiddler

This mechanism allows plugins to overwrite  "core" content. eg: Translations. ... The core always contains a fallback value, and a language specific plugin just overwrites the texts. So if a translation misses a text, TW still shows the core content. ...

See: https://tiddlywiki.com/editions/full/#HelloThere:HelloThere%20%24%3A%2FControlPanel

If you switch between languages, you'll see, that the texts change, but it's still the $:/SiteTitle shadow tiddler, that creates the site title.

We use this "stacking" mechanism quite a lot.

hope this makes sense.

have fun!
mario

[PMario]

About namespaces:

In TW everything is a tiddler. Eg: plugins.

A plugin is a tiddler, with a content-type: application/json and a field: plugin-type: plugin

This tiddler contains several other tiddlers. At startup, those "content" tiddlers are extracted into ShadowTiddlers.

See: ControlPanel: Plugins tab

The first plugin is named: Browser Sniffing:

If you click it, it opens a tiddler named. $:/plugins/tiddlywiki/browser-sniff  As you can see, we follow a naming convention here.

$:/plugins ... global prefix for plugins
/tiddlywiki ... author
/browser-sniff ... plugin name

If I would create the plugin, it would be: $:/plugins/pmario/browser-sniff ... and so on  ...

At startup this plugin is expanded into: 

• $:/plugins/tiddlywiki/browser-sniff/bowser/bowser.js
• $:/plugins/tiddlywiki/browser-sniff/readme
• $:/plugins/tiddlywiki/browser-sniff/sniff.js
• $:/plugins/tiddlywiki/browser-sniff/usage

So this is one naming convention, we follow. There are a lot more. ... So imo we just would need to come up with a "global" scheme, that is flexible enough, to include content from other TWs, in a sensible way, without loosing functionality.

-mario

[PMario]

As I wrote: Everything is a tiddler:

You can explore the core structure using: The right sidebar: More: Explorer  tab

see: https://tiddlywiki.com/#%24%3A%2Fcore%2Fui%2FMoreSideBar%2FExplorer

Clicking a "blue link" will open the tiddler and show the full name.
Clicking the "folder icons" will expand the list.

have fun!
mario

[PMario]

On Friday, March 2, 2018 at 6:52:43 PM UTC+1, joearms wrote:

Any other distinction (say by project) is tricky - ie X could belong to project A or B

or both - so where do we store it???

With TiddlySpace (discontinued) we did explore the possibility to "include" one "space" into an other.
eg:

space/system ... contains the core
space/plugins ... contains plugins
space/content ... contains user content

The further down the list, the higher the priority. ... That means "user content" will always win.
Tiddlers from  space/system can be overwritten by space/plugins  ... and so on

The important thing is: User content wins. ...

BUT if the user overwrites and messes up a plugin setting. Deleting the user tiddler, will nicely fall back to the space/plugin default setting.

See: ShadowTiddler / Tiddler relation from above

hope this makes sense

-m

[joearms]

Goodness - In front of your there are many doors leading to

twisty passages - you pluck up your courage and press on

"core" - your brain explodes ...

Thanks

/Joe 

[PMario]

On Saturday, March 3, 2018 at 10:18:02 AM UTC+1, joearms wrote:

Goodness - In front of your there are many doors leading to

twisty passages - you pluck up your courage and press on

"core" - your brain explodes ...

hihi, ..

Yea, but it's still "just a tiddler" ... a big one. It contains everything. The UI, settings, language ... and so on.

If we exchange it with a newer version and save and reload the ".html" file, we have made a core update, without loosing user adjustments. ... And as long as we keep it backwards compatible, it should - "just work" :)

-m

[PMario]

About namespaces again.

I think, the datproject.org in combination with beakerbrowser.com would provide us with a nice NS.

It uses dat://<hash>/index.html  ...

The <hash> is the public key to access the content. If you know the <hash> you are able to access the content with a p2p network.

It will be possible to make every single tiddler accessible. eg: dat.//<hash>/tiddlers/test.tid  ... or something like this. ...

... So we just needed to include it as a tiddler field. eg: origin. So the internal minimal structure would look like this:

title: my-new-tiddler
origin: <hash>

BUT there is a problem. ... A privacy problem.

Beaker promotes "Secret File sharing" as one of their features. ... So let's say we drag&drop import  a tiddler from a "private" dat:// address to a TW with a "public" dat:// address and set the origin-field as proposed above.

That's a privacy problem, since everyone which has access to "public", would now also know the "private" hash and can access the whole content. :/

So we would need to hash the dat://<hash> again and save the hashed-hash as the origin. So you would need to know the private dat:// address, to compare it with the origin: hashed-hash ....

Which imo would create a new "lookup problem", that doesn't scale well.

just my thoughts
mario