Referencing ~35000 tiddlers - Custom development?

[Cs Molnar]

There is something I want to achieve with TiddlyWiki but I don't know how, so I hope one of you can help me.

My problem, in a generalized way, is the following: I want to take reading notes from a book in tiddlers. The book has chapters (about 60), every chapter has on average 20 sections, and every section has on average 30 subsections. This makes around a total of 35000 subsections.

In my tiddlers I will refer to specific subsections. Later I want to be able to list all tiddlers which refer to a certain subsection. Also, I want my references to be links to the content of the subsection on the internet. (The book's subsections cannot be included into my TW, but they are available on the internet.)

Of course, the easiest solution would be to create one tiddler for every subsection and insert links like [[Chapter4_Section6_Subsection23]] into my notes. This way I could find every reference to a subsection. But this would mean creating 35000 tiddlers (equivalent to 35000 files in my TW on Node.js) to cover all subsections. Even if I didn't create them, I would have 35000 missing tiddlers. This approach would heavily pollute the list of my tiddlers. Every time I would use the $list widget, the 35000 existing or missing tiddlers would come up.

I hope someone has a good idea to save my TW from this pollution, but still retain the possibility to search for references. I thought that I could develop a custom widget in Javascript which would store the reference list in a JSON tiddler. I'm experienced in JS programming but have less knowledge on TW internals, so I'm not sure if this is the best solution or is it overkill.

[maki aea]

hi not a direct answer but i am using a structure for 10000 tiddlers (ported from a different app) and in hindsight didn’t end up needing so many. loosely, it’s 100 tiddlers with 100 subtiddlers per tiddler. references look like [[0030]] (reversed order i.e. subtiddler 00, main tiddler 30). these particular tiddlers are tagged with the same tag, and we can use that tag to exclude them from searches etc. 

[Stobot]

For what it's worth, I have a wiki that's a copy of my company's ActiveDirectory with about 30,000 people in it - one tiddler per person. It is a little slow but it works. If the issue isn't performance, but rather the "polution" one idea might be to "hide" them with a prefix - like $:/book/chapter4/section6/subsection23 per your example. Tiddlers with the $:/ prefix are automatically hidden from most general areas of TiddlyWiki.

I've also played with data tiddlers for similar things and performance is much better, but it makes working with properties/fields more difficult, so there's a tradeoff. If there was such a thing as a datatiddler that allowed multiple fields/columns (like JSON), that would be a game changer, but alas even datatiddlers as they are seem to be on the chopping block here from what it sounds like - still supported but nobody's interested in developing them further. 

[David Gifford]

Sounds like the Bible, which has 66 books, about 20 chapter average per book, and about 30 verses per chapter. Regardless of what your content is, do a search for Bible here and that should turn up instances of people who have the entire Bible in one TiddlyWiki, divided by book and chapter, to see how they organized what they did.

Comments:

1. Is it necessary to create them all at the outset, rather than create them as you take notes on them? I don't see the value in that.

2. You could create a manual list of tiddlers for the next section you plan to read, say 4.6 (with content of

* 4.6.1

*4.6.2, etc),

then when you are ready to take notes on 4.6.1, select it and transcludify it. Then open the link. That way there are no missing tiddlers or existing tiddlers before you are ready to add content to them.

3. Or you could create the subsection tiddlers one by one and have them backlink to the section.

4. You may consider using initial zeroes. 09.04.18, so they line up in order in the All tab and the search results.

5. Consider the Freelinks plugin. Tiddlers with unlinked references to other tiddler titles appear automatically as links.

6. You can add a chapter field to the chapter tiddlers, a section field to the section tiddlers, and a subsection field to the subsection tiddlers, to limit searches in AdvancedSearch or in list-searches (see the ListSearchPlugin) [has:field[subsection]], etc

[Mark S.]

I've made a Bible (twice, I think, but I've misplaced one. Possibly on the car roof top). A 6000 entry nutrition database. A 63,000 entry dictionary.

You'll have to explain what you mean exactly by "pollution".

For your situation, using node.js, there is, as they say "One weird trick." You can create all your reference tiddlers and put them in a plugin. This just means putting them physically in a folder under "plugins" and including a plugins.info file. Now your references will be essentially invisible until you want to link to them.

[Cs Molnar]

Thanks for all your suggestions! By „pollution” I meant that, according to my understanding, it would be a bad idea to create 35000 „useless” tiddlers, which do not have content but only used as references. It would make my wiki slow. TW is optimized for the case that tiddlers store content and not references. Every time TW or a plugin would run its Javascript code, it would have to chew itself through these useless tiddlers, and it would happen at almost every content refresh. Also, because I use git to version control my .tid files on Node, this would put unnecessary burden on my git repository, as well.

That's why I looked for other ways. Not to mention what happens if I read 2 other similarly sized books. That would require over 100 000 tiddlers. It just seems like a really bad idea to me.   

[Mark S.]

I thought the reference tiddlers had content -- external links to file system documents ??

[Cs Molnar]

As I was thinking my requirements the other day, I realized that I basically need a referencing/searching functionality in TW, which is independent from (works alongside of) the existing referencing/searching functionality for tiddlers. After I had realized this, I came up with a simple solution.

I created a macro with a syntax like <<myreference "Chapt01_Sect02_Subsect23">>. This generates a link to the online version of the book, to the referenced Subsection. If I want to find out in which tiddlers I referenced this Subsection, I simply search for "<<myreference "Chapt01_Sect02_Subsect23". This is possible because (as I recently realized), one can search for the invocation of a macro by searching for the text „<<mymacro>>”.

This solution seems to fit my requirements and it does not require the creating of thousands of empty tiddlers just to enable the referencing of Subsections. It is also easy to add another macro with a different name if I need to reference sections of another huge book I will read.

Again, thanks for all your kind suggestions and reactions.   

[Soren Bjornstad]

Might be worth pointing out that you could also write:

<$macrocall $name=myreference parameter="Chapt01_Sect02_Subsect23"/>

...and that macros can contain arbitrary whitespace between parameters, and use different types of quotes:

<<myreference

  'Chapt01_Sect02_Subsect23'

>>

As long as you're consistent with how you call the macro in your wikitext, your method can work fine, but you should be aware that just because you have a correctly displayed macro doesn't mean your search method will find it.