Building a list of tabs dynamically in a performant manner

[Soren Bjornstad]

Hi all,

I want to determine what tabs of the Reference Explorer in my Zettelkasten to show on a template, such that if there aren't any results on that tab, the tab doesn't appear at all. The results of the tab are produced based on a filter (of course), so I figured I would also determine whether the tab appears by running a filter. The filter is stored in a field in the tab tiddler.

That is, I have a series of tiddlers with a certain tag (say Tab), and each of these tiddlers contains a filter in some field (say condition). For each Tiddler tagged Tab, if and only if the filter Tiddler!!condition, run with the current tiddler as input, has more than zero results, I want to display the tab.

I came up with the following:

<$set name="tabList" value={{{ [tag[Tab]] :reduce[<storyTiddler>subfilter{!!condition}then<currentTiddler>addprefix[ ]addprefix<accumulator>] }}}>

    <$macrocall $name="tabs" tabsList=<<tabList>>/>

</$set>

This produces the correct result (well, as long as there are no spaces in the titles of the tiddlers tagged Tab; I'm OK assuming that since there indeed aren't any). The problem is that it is horrendously slow to run all these filters. On my dev machine it is tolerable, but this is a machine specced for serious processing power. On my MacBook Air it now takes 1–2 seconds to open a new tiddler, even without anything currently open!

Probably I am just asking TW to do too much on the fly here, but before I start rethinking the project too hard, can anyone think of obvious optimizations I might be missing here? The filters involved are moderately complex (the basic pattern for each is to gather together links[], backlinks[], and tagging[] for the story tiddler, then filter some things out of that using + and !operator[]'s).

[TW Tones]

Soren,

Its late here but I have done something similar in the past without a performance hit, and will try and create a solution tomorrow, However I think the answer best answer may be through the use of a filter run as in 5.3.23+ however I am sure I succeeded in something similar a few versions ago.

No need for reduce and accumulators I think.

Regards

Tones

[Soren Bjornstad]

Tones,

:filter was my first thought, but I couldn't figure out how the data would flow through it. Perhaps I was missing something, looking forward to seeing your version.

If it helps to see the context, have a peek at https://zettelkasten.sorenbjornstad.com/#%24%3A%2Fsib%2Frefexplorer%2FReferenceExplorer. The snippet above (or a replacement) would go inside the ref-explorer macro definition.

[Soren Bjornstad]

Important note, the condition fields aren't defined in the current public version, so you'd have to add them in to get a working test.

[Saq Imtiaz]

@Soren I recommend enabling performance instrumentation and checking to see which filter runs are the culprits.

https://tiddlywiki.com/#%24%3A%2Fconfig%2FPerformance%2FInstrumentation:%24%3A%2Fconfig%2FPerformance%2FInstrumentation%20%5B%5BPerformance%20Instrumentation%5D%5D

Note that you need to enable it by setting the config tiddler to "yes", then save and reload.

Also suggest posting a public wiki where the issue (and the condition filters) can be seen to facilitate debugging.

[Soren Bjornstad]

Thanks Saq, that view is super useful and I never knew it existed!

Armed with that information, I spent most of this evening tinkering on improving some of the other things that have been slowing my wiki down, including adding an Idea tag so I don't have to define it as the negation of a dozen other tags and replacing TiddlyTables with a similar HTML table that needs fewer filters, macros, and transclusions in the most critical paths. (More work needs to be done to bring the rest of the Reference Explorer in line and refactor it to catch it up with a bunch of other changes I've made recently.)

The tabs filter discussed above was and remains the #1 contributor to filter processing time when clicking around the wiki, often something like 30% of the total time, so I'm still interested in suggestions others may have for improving that. I didn't want to publish the wiki in its present state previously since it was so slow it would be difficult for people to use, but I've ameliorated that enough that I am now comfortable republishing it. As mentioned in my previous post, you can start looking at $:/sib/refexplorer/ReferenceExplorer, but now it is a working example rather than something you would have to tweak into a working example by yourself.

[Saq Imtiaz]

Hi Soren, glad that was helpful. When I worked on performance improvements for Streams a few months ago, the biggest gains were from simplifying the widget and DOM structure, and replacing macros with transclusions, rather than from optimizing filters.

I took a brief look at your wiki just now and the one odd thing I see is that there seems to be an intermittent refresh being triggered every few seconds. Any idea what that is about? To see what I mean, create a new tiddler with the following in the text and then observe the developer console:

<$log t=<<currentTiddler>>/>

This should work on the current pre-release but probably isn't any faster:

[tag[$:/sib/refexplorer/tabs]] :filter[subfilter{!!condition}] +[join[ ]]

I suspect that any real gains would come from optimizing the condition filters, as they likely make up the bulk of the time needed for the above filter expression. For filter operators that are indexed, the first time they run they will take a while but subsequent runs should be faster. 

https://tiddlywiki.com/#Performance

[Soren Bjornstad]

Saq, the intermittent refresh every 5 seconds updates a clock and Pomodoro timer. The reason it's puzzling is that it doesn't appear by default in the public edition. If you're curious, tick the "clock" checkbox in the FeatureFlags tiddler.

I'll give the :filter version a try later. I suspect I didn't think to use filter and subfilter together in one run!

[TW Tones]

Soren,

Given this thread, I am not sure if I am on track here, but reading your Original post it seems so.

As I understand it you want to use a filter in a tab tiddler to determine if it should have the tab displayed.

The thing is the tabs macro is not designed for conditional tab display, it simply wants a list of tiddlers. If this could all be rammed into a single filter then all would be fine. However you not only want to make use of another variable or in this case a field but that field contains a filter. This is asking too much of the filter syntax.

What I have done in my solution was to generate the list of tiddlers that would be tabs according to you conditions and feed only the result to the tabs macro.

• In my small test not performance issues, and I do not expect their to be
• I think what is happening in your case if the reduce etc... is adding unnecessary complexity to the performance of the tabs macro.

Here is the core logic tested on tiddlywiki.com and pre-release incidentally.

\define tabs-with-content()

<$list filter="[tag[Tag]]" >

   <$list filter="[all[current]get[filter]]" variable=found-filter>

      <$list filter="[subfilter<found-filter>count[]!match[0]then<currentTiddler>]">

       </$list>

   </$list>

</$list>

\end

<$wikify name=tabs-with-content text="<<tabs-with-content>>">

<$macrocall $name=tabs tabsList=<<tabs-with-content>> template="tabtemplate"/>

</$wikify>

If this does not make sense I will package the tiddlers in this test for you.

The above example may be a good test case if we wanted to invent a method to make filters more sophisticated basically permit ones that allow you to avoid the nested lists I used, however as stated previously dont be shy using nested lists, because each level of nesting allows to generate a new variable and keep the old ones in this case found-filter while keeping currentTiddler from the first filter.

My preference would be a shorthand method when presenting variables as a parameter, in this case <<tabs-with-content>> such that it is first wikified rather than using the ungainly wikify widget and its need to wrap the macro call. Eg a a parameter ((tabs-with-content)) or in filters (tabs-with-content)

Alternatively TiddlyWiki would detect the need for wikification of that variable/macro and do it itself.

Regards

tones

[TW Tones]

Oops;

I must add

the tab template contains

<$tiddler tiddler=<<currentTab>>>

<$transclude mode="block" />

</$tiddler>

and each tab contains 

<$list filter={{!!filter}}>

</$list>

The tabs work stand alone

Tones

[Soren Bjornstad]

Tones, thanks for putting this together, but your solution is slower than mine as far as I can tell (no easy way to instrument that since it has a very different number of filters, but the difference is apparent from clicking around a bit).

Saq, I can't get your filter to work and I can't see how it would work in theory either now that I look at it – the !!condition in the subfilter is getting the condition field on the tiddler in which the tab list is being rendered, whereas it needs to be the tab tiddler that's coming in on the input of the subfilter operator…right? Or am I just being dense? In my :reduce version it has a different meaning because :reduce actually resets the currentTiddler variable to match the input value currently being processed.

With the additional optimization work I did the other night, including optimizing the filters on each tab as Saq pointed out, I would say it's now performing acceptably with my current version, if only barely on a slower computer. So if nobody has any other ideas, I think I can leave it where it is for now.

[Soren Bjornstad]

[TW Tones]

Soren,

Can you share the data set, or a dummy one for us to look at performance improvements?

I presume you have reviewed you filters in respect of their performance here?

Tones

[Saq Imtiaz]

Hi Soren, I mentioned this when I suggested the filter and explained in a bit more detail in  a follow up reply which seems to never have been posted.

 You will need the 5.2.0 pre-release for that filter to work, as one of the changes is greater consistency in setting the value of currentTiddler for :filter, :reduce and :sort and their operator equivalents.  See  https://tiddlywiki.com/prerelease/#Filter%20Expression

[Soren Bjornstad]

Ah, got it. I'll give it a try when I upgrade, thanks.

[Soren Bjornstad]

Tones,

Here's the reference explorer tiddler, as mentioned earlier in the thread: https://zettelkasten.sorenbjornstad.com/#%24%3A%2Fsib%2Frefexplorer%2FReferenceExplorer

Most of the data set is public in the same wiki…if you just click around in some random non-system tiddlers, they all have the Ideas section of the Reference Explorer showing. The public version may perform just a little better than my private one since it has about a third fewer tiddlers, but won't be too far off.

I had not remembered to look at the #Performance tiddler, but I don't believe any of the suggestions in there are applicable here.