Hitting a barrier at 36,000 tiddlers

[Mark S.]

My project may have been over ambitious. But I thought there were going to be some

improvements in 5.1.20+ that would allow for larger TW files. I currently have a TW file

with about 36,000 short dictionary definitions. Each tiddler consists only of a title

and a field, EN, with usually only 3 or four words in it. The overall size is only 10 megs.

Overall, the performance is impracticably slow. Ten seconds to open a tiddler. 20 seconds or more

to do a simple search box search. As much as 25 seconds or more to close out tiddlers that

need to render a custom search. In most cases, you have to type blind because the

keystroke refresh is also incredibly slow.

I'm wondering if the performance improvements actually made it into the core, or if

my expectations were too high. Actually, the performance noticeably declined

after the first 12,000 entries.

At the moment it seems like I need to start over, using a data dictionary instead of

tiddlers for storage.

Firefox, Windows 7, TW 5.1.21

Thanks!

Mark

[TonyM]

Mark,

To fully address your experience consider if you can show or share the wiki. Alternativy share some test data of an equivalent size. Also consider it from a "what must it render perspective". If for example the tiddlers were all listed in the table of contents and this is displayed in the sidebar this will refresh every change that may effect that toc list. Hide the side bar and see if its the same.

I had 12Mb files working well in TiddlyWiki Classic, so it should be good in TW5

In an App I built there was a considerable difference with 5.1.20 however be warned there was an accidental leaving on of "Performance instrumentation", see Control Panel > Settings > Performance instrumentation - uncheck save and reload.

Also a normal search that is activated for any three letter combination and lists the result below may just be inappropriate for such an application. Perhaps you need a enter search criteria then hit a search button, which renders the results in a list, rather than an instant response popup/list. 

There is also a little gotcha with browser memory. Try with nothing else open by or in a tab of the browser and see if it improves, this will give us a clue to its bottle necks. Browsers also have a maximum memory usage set so they do not overpower the device/computer but this is often a fixed value, when you mostly work in the browser there is value making more ram available to the browser especially if you have 8-16GB available.

The more you share the more we can help.

Regards

Tony

[TonyM]

Post Script

If you a doing word searches, ie words as tiddler titles, not words in phrases or in the test fields changing the search to look only in titles (not in text) or even search first by prefix could improve search outcomes a lot.

Regards

Tony

[Mohammad]

TW-Scripts has 4000+ tiddlers including all shadow tiddlers.

I am on 5.1.21, Firefox 69, Win 10

I see noticeable response time comparing to empty.html

--Mohammad

[@TiddlyTweeter]

Mark S. wrote:

.... At the moment it seems like I need to start over, using a data dictionary instead of

tiddlers for storage. 

I'd be very interested to know, if you go ahead with a data dictionary version, what the performance is like!

Best wishes

TT

[Birthe C]

Hi Mark S,

You are around the size 10 megs where I usually get into trouble. Recent i got a "new" computer with more ram, it did help a little but really not enough for it to be fun to work with these wikies..

Former I sometimes used Firefox, Windows7 and found it slower than Firefox, Linux Mint.

Birthe

[Mark S.]

It's slow even if non-searching tiddlers are all that is rendered.

It's slow with only 6 other tabs open.

It's slow even if FF has just been opened.

It's slow even though instrumentation is off.

It's slow even if doing a single-field match.

It's not just searching that's slow. EVERYTHING you type is slow. You sit there waiting ten to 20 seconds for your last

typed entries to appear.

It could be that it's the number of tiddlers rather than the gross size of the TW that matters. In which case I suppose

I could do some crazy design where a single tiddler contains all the words that start with "a", etc. But this makes

reverse lookups harder.

Thanks!

[Jeremy Ruston]

Hi Mark,

Interesting! I’d echo the request to share the wiki if you’re able to.

Perhaps you could use the performance instrumentation tools to see if any filters are bottlenecks:

https://tiddlywiki.com/#Performance%20Instrumentation

In particular, after turning on performance instrumentation and restarting, please can you post the output of $tw.perf.log() typed in the browser developer console.

If it’s any comfort, I’m working with wikis with over 60,000 tiddlers without issues, so it’s not an intrinsic problem with the number of tiddlers.

Best wishes

Jeremy

-- 
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/2ef5aae4-83c9-411e-bb62-ad97832f6b2c%40googlegroups.com.

[Mark S.]

Thanks for the reply!

It can be retrieved from:

   https://github.com/Marxsal/EODIC

How do we get the output out of the console? It loses all the table formatting when I try to copy it.

Thanks!

Jeremy

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[TonyM]

Mark,

Perhaps if you can't share your wiki publicly perhaps you can privately to a few of us. With privacy assured.

There would be value in looking into browser memory like I suggest in a previous reply.

I find timimi is the quickest but I do use Windows 10 64bit with 16gb of ram. And since 90% of my work is in a browser now, usually in a tiddlywiki, I have lifted the ram that Firefox and chrome can make use of.

I knew of Jeremy's large wiki so thus speculated on other issues.

Can you try another computer/os or version to see how persistent the issue is.

Regards
Tony

[Mark S.]

Hi Tony,

Perhaps you didn't see the prior post. You can download what I have so far at

  https://github.com/Marxsal/EODIC

It could very well have something to do with machine capabilities and memory. My current box only has 8 Gig.

Thanks,

Mark

[TonyM]

I just noticed before you replied - ooops

Looking now

[Birthe C]

Sure enough, Firefox constantly complaining, a webside is slowing, do you want to wait?

Birthe

[TonyM]

Mark,

I am getting some performance issues as well. I did hear my computer waking up to get disk or cpu to operate.

The example you have provides still relies on the sidebar and standard search. Have you tried to implement some of my changes in the previous post in your own copy?

By opening the advanced search > Standard tab and closing the sidebar I do get a quicker response to searches but this is already looking for any partial match of the title and in the content of all tiddlers. A Customised search will surely improve it a lot. 

I am experimenting now.

Tony

[TonyM]

Birthe,

There are timeouts I commonly increased in browsers to stop this kind of error message, at least in earlier versions. I expect they remain in my browser settings.

Tiddlywiki does do more than most things in the browser.

Regards

Tony

[Mark S.]

If you want to try a more optimized search then open up the Dictionary Manager tiddler

and change this line:

<$set name="listfilter" value="""[tag[Vorto]regexp:en[\bcat\b]sort[title]limit[10]]""">

to

<$set name="listfilter" value="""[tag[Vorto]match:title[kato]sort[title]limit[10]]""">

Or you can search for "automate" in the title in whatever form you think most efficient. I think you'll

find it's still plenty slow.

But search is only part of the problem -- I can barely type or operate. Opening any tiddler takes 20

to 30 seconds. Everything is slow, slow, slow. You start to type, and the cursor disappears and you

have to wait 5 to 20 seconds for it to return.

I certainly couldn't continue developing the app in this manner.

And it gets worse. Only half the words from the original dictionary are loaded!

Thanks!

[Mark S.]

Yes, I already have this set to a two minute timeout:

https://support.mozilla.org/en-US/kb/warning-unresponsive-script#w_letting-the-script-run-longer

but of course, that just gets the messages out of the way. You still have to wait ...

Thanks!

[Birthe C]

Mark,

My pc also has 8 gb ram. One of my wikies are about the same size and has nearly the same amount of tiddlers. I have very little shown in the sidebar. The tiddlers a very short text, very little filtering . Very simple wiki indeed. But it is just as slow and difficult to work with as yours. That testet on the same computer with the same amount of tabs open in the browser.

Just so you know before spending a lot of time changing and optimizing.

Hope the best,

Birthe

[Mark S.]

Thanks Birthe.

What TW version are you using? 5.1.20 was/is supposed to have optimizations that allowed larger TW's.

Thanks!

[Birthe C]

I am using the latest 5.1.21 - just upgraded yesterday.

I does not take as long to read as yours in my browser, it is slow to navigate, but as you are experiencing. - terrible to impossible to edit.

I can add, that last time I tried TW translators edition I had the same feeling, I really had to give up only half way through - earlier version and something else I know.

Birthe

[Birthe C]

Mark,

I got to try your wiki on my friends pc with 12 gb ram. That did not give any big difference, it would not like to edit it. (Just thought it would be fun to know)

Birthe

[Birthe C]

Mark,

Did you mention optimizations?

Do you recall an earlier thread concerning Dave Giffords Photoalbum...and the wonderfully humorous answer from Jeremy https://groups.google.com/forum/#!searchin/tiddlywiki/TW5$20image$20gallery/tiddlywiki/PpJsdJr99n8/nY-9eAXGB6YJ

Great, greater, greatest wiki and a contest in hardware,

Birthe

[TonyM]

Mark,

I am continuing to research this here is a little of my work product

Longer time outs and waiting can be better than multiple continue prompts.

I would definitely add a splash screen to this wiki.

I can see what you are experiencing but not as severe. By using a taylored search in the only tiddler displayed and the side bar closed I get further improvements. I have not yet set a submit search that only searches once you hit submit. 

Try this and see if there is a marked improvement or not

Vorto/Title/prefix search

<$edit-text tiddler="$:/temp/search/title" tag=input/>


<$list filter="[{$:/temp/search/title}minlength[3]]">


Number found: <$count filter="[tag[Vorto]prefix{$:/temp/search/title}]"/><br>

<details>






<$list filter="[tag[Vorto]prefix{$:/temp/search/title}]">




</$list>
</details>
</$list>

My hunch is tiddlywiki is rendering something with every keystroke and action that is impacted by the scale of the wiki. Its obvious with a search that changes every key stroke but not before the 3 character limit is reached in my example.

I just tried it 

through a local TiddlyServer and its a little slower but similar.

through TiddlyDesktop and its faster but still not easy to use.

Hey could this be it?

I closed all tiddlers and sidebar and did a firefox developer inspect, to see what is actually presented in this view and possibly damaging performance

• I found a noscript area for when Java script is not available full of static tiddler links (Presumably all tiddlers). I deleted this from the HTML
• Then on inspect again I find the div id=storeArea appears to list every tiddler, this may be normal operation however the inspector lists a few pages then says
• some nodes were hidden. And a button to Show all 72157 nodes, this is a large number and seems to be twice your 36,000 words.
• This looks suspicious.

We now need some serious skills to advise.

Regards

Tony

[TonyM]

Ok Mark,

I have something resembling a solution. These are the steps I took and from this we may draw some conclusions. 

Is there a Tiddlywiki ribbon I get?

It is a solution and a work around but the root cause is not yet known.

• ESPDIC wiki - exported all tiddlers tagged Vorto to json
• New Empty Tiddlywiki
• Imported all tiddlers in above json
• Searching for the words with the default and advanced search is very fast, none of your reported issues here
• Stay away from the side bar recent tab, its too slow

Note: I also cloned the $:/import tiddler into "Word List" after importing the words, because it contains a list of all imported words as links to the tiddlers. Delete the status field.

Not only does the above work well as is, but I believe rejigging the "Word List" into a datatiddler, you could do a custom search inside it and provide a link to the resulting tiddler, thus not searching the tiddler store for words but searching the wordlist data tiddler.

There may be away to bypass this step by importing the Words as an Info plugin, the words will then remain shadow tiddlers unless you overwrite them, then a special search that also looks at the shadow tiddlers would be required to find words but this should keep the words out of the standard search and recent tiddlers tab.

You could also modify the recent sidebar tab to exclude tiddlers tagged with Vorto and create a special one for only Vorto items if needed.

Regards

Tony

[Mark S.]

Hi Tony,

That's very interesting. Why should deleting and then re-importing make a difference? Unless the way I created them bypassed the new optimization technology.

Maybe @Birthe could try exporting and re-importing her data as well, to see if her TW performance improves.

I might mention at this point, to add to my chagrin, I was importing these into the app 6000 at a time. I was using Vim to break up the original word list into

a manageable dictionary. BUT, unbeknownst to me, Vim was changing diacritical characters into plain ASCII characters. I didn't even know that that was

possible. So I have to start over using some other editor (Emacs maybe) that doesn't savage the data. Each set of imports of tiddlers (conversion from

a data dict tiddler to tiddlers) was taking 20 minutes or more, with multiple interruptions to bat the wait dialog out of the way. Ouch

If your method works, it's definitely worthy of a ribbon ;-)

I might not get a chance to work on this until tomorrow.

Thanks!

Mark

[Mark S.]

Hi Birthe,

Strangely enough, I didn't remember a thread from 2013 ;-)

Back then, Jeremy was saying an upper limit was around 100 mb. Now I think he's suggesting 800mb (or was it more?) is feasible.

Thanks!

[TonyM]

Mark,

I have had a chance to work on this today, but may not tomorrow. Because of my long term investment in the tiddlywiki platform, both personally and professionally I want to ensure I identify the limitations and ways to overcome them.

When building a custom search I saw the performance issue return in different circumstances, however the current demo seems to be OK.

Oddly the Wiki is larger than before - perhaps because the noscript section is within it. It would be better to replace this with a message that Javascript is required.

http://tiddlywiki.psat.com.au/ESPDIC

I have installed and activated the local storage plugin, if you change the filter to [all[]] you can work on this in the browser.

Regards

Tony

[TonyM]

Post Script,

I forgot to mention on my demoI have created a info plugin tiddler with ESP words in it, and operating on its shadow tiddlers. this may have the result of keeping them out of other searches and improving performance.

Regards

Tony

[Mat]

I didn't try out anything but here are a few thoughts:

• Do browsers accept any size of a webpage for all it's tools and operations? I.e might some thing kick in or turn off in browsers at a certain stage? Might TW in particular trigger or or off something that browsers rely on? If I understand, we have a special relationship to javascript that is atypical.
• Could perhaps unused fields in the standard templates be removed?
• You've probably already ensured this is not the problem but; When my wikis (that are only a few megs) slow down it is, I believe, typically because of nested loops (e.g nested listwidgets).

<:-)

[Birthe C]

Hi Mark,

I referred to the old thread mostly because Dave Gifford said he could not feel a slow down until 100mb, Jeremy admitted he had to give up at 60 mb then.

That indicate to me, that hardware has to do with it.

Also that the improvements in hardware in the years to come will better this.(That was 2013).

Now ,not everyone gets new hardware constantly. That must be remembered when creating a wiki for others to use, who are the potential users?

Tiddlywiki itself has grown rather big over the years also. A language plugin + a few plugins and I am constantly finding myself starting out with 3.6 mb.

Tony,

Thank you for all your test and explaining. It is interesting and very important.

Birthe

[Birthe C]

Hi Mark,

When upgrading my very old wiki I did export my tiddlers and reimport them after having changed the "old ways" of doing things. It did help just enough for me to divide my tiddlers and export them for use with 3 of my other wikies. (it made more sense in my case).

@Tony just followed your link. The start does not feel that slow. It seems workable, but I kind of wonder if it will not be more difficult for users to use. I will have to look closer when I have more time, it is an important learning experience.

Birthe

[@TiddlyTweeter]

TonyM wrote:

http://tiddlywiki.psat.com.au/ESPDIC

Quick note.

This version worked fine for me on a short test. Mark's original didn't.

Win 10, 64. 4GB memory. Dual core. Tested online version, not downloaded file.

Tested in Chrome & Chromium-Edge.

Best wishes

TT

[Mat]

http://tiddlywiki.psat.com.au/ESPDIC

This version worked fine for me on a short test. Mark's original didn't.

Same experience for me - downloaded file poor performance, online file works quite well.

Win10, 4GB

<:-)

[Birthe C]

Hi Mat,

Firefox or Chrome?

Reading TT. I tried the downloaded file in Chromium, it absolutely didn't run, but it walked. (I mostly find Chromium faster than Firefox, but I like to work in Firefox better.

Linux Mint Mate 19.2  8gb

Tony's work is by far the fastest.

lørdag den 21. september 2019 kl. 12.37.00 UTC+2 skrev Mat

[@TiddlyTweeter]

Footnote.

I just tested TonyM's version on Android 8. Chrome browser. Online version.

Its works, on first look, just as well as via Desktop.

That's really good news for apps like that. 

Take your dictionary with you!

Best wishes

TT

[@TiddlyTweeter]

TonyM wrote ...

It is a solution and a work around but the root cause is not yet known

That is a very interesting comment.

It would be very useful to pin it down. 

Is it about reducing dynamic recursion in wikis with umpteen Tiddlers? 

I have been inhibited from making large scale wikis of zillions of fragments because I hit issues Mark did.

Best wishes

TT

[TonyM]

Actualy I think there may be some fundamental assumptions made in the typically sized wikis that a wrong at scale. For example the recent tiddlers list can get very large. When saving the noscript section can add a 50% to file size.

Somehow Jeremy and others avoid this at scale as did my example but are we avoiding performance issues through techniques we have a gut instinct for but are not quantified or published?

regards
Tony

[@TiddlyTweeter]

TonyM wrote:

Actualy I think there may be some fundamental assumptions made in the typically sized wikis that a wrong at scale. For example the recent tiddlers list can get very large. When saving the noscript section can add a 50% to file size.

Somehow Jeremy and others avoid this at scale as did my example but are we avoiding performance issues through techniques we have a gut instinct for but are not quantified or published?

I'm just an end user. Waiting for some tips. Like a list of things to do and not do. x

TT 

[Aidan Grey]

Recent list has come up many times as an issue in this thread. How can we limit it to the last 10/20/etc. tiddlers? I'm tempted to drop it altogether in my tws.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/50bc5f2c-f80c-4d2f-9485-2d1d3d22664e%40googlegroups.com.

[Mark S.]

Hi Birthe, Tony -- everyone! You've all been very helpful!

I think I know what is causing the problem -- but I don't know why.

There's a view template (WordTemplate) used to expose the en field. The contents are typical for templates:

<$list filter="[all[current]tag[Vorto]]">
Eng: <$edit-text field="en" tag="textarea" rows=2  class="vorto-edit" />
</$list>

Add this to Tony's version (tagged with $:/tags/ViewTemplate) and it will immediately bog down

about the same as mine (maybe slightly faster, since the template isn't being styled).

This follows the standard recipe for view templates. Tiddlers themselves are built from view templates

(or at least the UI we see are built from templates), so why should this short bit of code cause

problems>

I suppose I could jettison the template for this particular project, but I'm sure there are other

projects where having a view template for tiddlers will be very important.

Ideas?

Thanks!

Mark

On Thursday, September 19, 2019 at 9:19:37 PM UTC-7, Mark S. wrote:

My project may have been over ambitious. But I thought there were going to be some

improvements in 5.1.20+ that would allow for larger TW files. I currently have a TW file

with about 36,000 short dictionary definitions. Each tiddler consists only of a title

and a field, EN, with usually only 3 or four words in it. The overall size is only 10 megs.

Overall, the performance is impracticably slow. Ten seconds to open a tiddler. 20 seconds or more

to do a simple search box search. As much as 25 seconds or more to close out tiddlers that

need to render a custom search. In most cases, you have to type blind because the

keystroke refresh is also incredibly slow.

I'm wondering if the performance improvements actually made it into the core, or if

my expectations were too high. Actually, the performance noticeably declined

after the first 12,000 entries.

At the moment it seems like I need to start over, using a data dictionary instead of

tiddlers for storage.

Firefox, Windows 7, TW 5.1.21

Thanks!

Mark

[Mat]

Mark S. wrote:

<$list filter="[all[current]tag[Vorto]]">
Eng: <$edit-text field="en" tag="textarea" rows=2  class="vorto-edit" />
</$list>

What happens if you exchange the operators for "synonyms" like instead of "all[current]" do "[[tiddlertitle]tag[Vorto]]" or [all[current]foofield[Vorto]] etc? Maybe it is one of the operators that doesn't function properly.

<:-)

[Mark S.]

Hi Mat,

I've already tried [<currentTiddler>tag[Vorto]], if that's what you meant by the first part. Alas, it didn't help.

Testing for foofield[Vorto] is a good idea. But I'd have to first modify 36,000 tiddlers. So I'm kind of hoping someone

will pop up with "Oh, that's a known problem with template tiddlers -- all you have to do is reverse the polarity and you're done" (Reversing

the polarity always works in ScFi.)

Thanks!

[Mark S.]

Using "[all[current]has[en]]" did not show any significant improvement.

[Mark S.]

It appears that having a search, list, or list-links that uses a tag operator

running in any currently open tiddler will create a performance problem EVEN if the tag

operator is applied AFTER a previous search operator.

So this

[all[shadows+tiddlers]search:title{$:/temp/search2}has[en]sort[title]limit[10]]

performs fine, but this

[all[shadows+tiddlers]tag[Vorto]search:title{$:/temp/search2}sort[title]limit[10]]

or this

[all[shadows+tiddlers]search:title{$:/temp/search2}tag[Vorto]sort[title]limit[10]]

Perform poorly.

This last one is especially puzzling, since you would expect at this point that the tag

only has to filter a handful of input titles. This suggests that there might be something

wrong with the tag operator, that goes undetected except when there is a large number of tiddlers.

If a process uses the tag operator in any open tiddler, then the performance of EVERYTHING,

including simply typing in some other random tiddler, will be degraded.

Or at least that's what it's looking like to me at the moment. 

Thanks!

[TonyM]

Mark,

Perhaps try these test in a 5.1.19 Wiki given the additional indexing to improve performance was introduced in 5.1.20 and if there was any change that could have had an unexpected impact in this area I think it could be those changes. Keeping in mind the tags field potentially contains multiple values and if different order so how is this sort indexing operating. I believe Jeremy's 60K tiddlers were working before 5.1.20 was introduced.

Its great that you are possibly closing in on this. Are you confident your sidebar is closed and there is nothing that could be interfering? Like my range of suggestions before?

Regards

Tony

[Mat]

Mark, that might be a huge step forward.

Here is a brain dump of hypotheses and thoughts from someone who doesn't really know what he is talking about. You will, for sure, think "what is this fool twaddling about?" but while my ignorance is typically a weakness it occasionally brings an out-of-the-box perspective that is fruitful. I used to think most people were too shy or polite to express ignorant ideas but it turned out they just barely have any ideas to begin with - so, for what it's worth:

• I'm assuming that filtering works by saving the step output in a variable that progressively changes for each new step. However if, to our surprise, the position of the tag operator in the filter doesn't affect the time it takes then maybe it happens to not use the latest instance of the variable but an earlier one? A question here is if the variable status is saved somewhere and the tag op somehow accesses this earlier filter step output instead of the latest instance?
• I believe the tags field is treated differently from other fields. Matching a tag, means iterating over all items in this field. Just maybe the system is made to deal with this in some parallel (asynched?) process and this messes things up when one of the parallel processes needs the input from the other one?
• Maybe the repeated iteration in the tag operator gets stuck in some recursive loop? Are the edge cases properly investigated - a tiddler with no tag? A tiddler with tag fooX when using tag[foo]? Might some tag fields overflow a magic max number of allowed tags? 
• And are tiddlers that DO fulfill the filter properly dealt with? For example "tag[foo]" will get the tiddler with "tags: foo bar baz" but will it effectively avoid iterating over the rest of the tags when "foo" is already found? (This possible behaviour would not explain Marks wikis behaviour but maybe it is relevant anyway)
• Is there some magic number of parallel processes where the browser switches methods for performing calculations? Might specifically the tag operator initiate an atypical number of processes which triggers this?
• Does the tag op - or filters in general - treat pure tags (i.e tags without existing such-named tiddlers) differently from TagTiddlers? 
• Does the system do any preprocessing on the filter before the steps are processes? Perhaps it cleans up the syntax, e.g trimming white spaces or some such, which affects specifically the tag op?

Again, just a brain dump to trigger ideas. I'm not expecting anyone to answer these .

<:-)

[Jed Carty]

Each filter operator only works on the output of the previous operator, so the order matters with later operators in a run generally working with fewer inputs than earlier operators.

To my knowledge none of the operators do anything in parallel, that may be difficult to set up because javascript is single threaded.

The tags field is treated differently, the tags field is sorted and stored as a different type of list than the title lists normally used in tiddlywiki are, this means that accessing or manipulating the tags field introduces an extra step or two.

In the worst case it has to load the field, parse it as the list type it uses, do whatever the operator does, re-encode it as the tags list and then save it.

Reading or searching the tags field doesn't need to re-encode it or save the field, but the extra parsing can add some overhead.

Other than that each filter step takes a title list and returns a title list, there isn't any other memory that carries over between each step.

The tag operator is a pretty heavy operator. It finds every tiddler in the wiki that has the current tag then checks each input title to see if it is on that list and returns the list of input titles that pass that test.

So if you have a million tiddlers with the tag Vorto it is searching through a list of a million tiddlers once for each input tiddler.

I think that the current version has a better worst-case behaviour than many alternatives, but it's speed is based on the number of tiddlers in the input list and the number of tiddlers in the wiki that have the same tag.

It could be changed so that the speed is based off of the number of input tiddlers and the number of tags each input tiddler has which may be worse in small wikis but a lot better in very large wikis. It depends on the cost of checking the tags list of each tiddler in the input list individually vs the cost of getting a list of all tiddlers in the wiki that have the input tag.

Most of the improvements that immediately come to mind don't preserve the order of the input titles. In large wikis resorting the output could be far faster than the current version, but much slower in smaller wikis.

I don't know how much overhead would be added by adding a check to see which one is faster, that check may be a complex task by itself.

The tag operator may be a place where some significant improvements can be made.

[Mark S.]

Seems like the answer might be a separate tag filter operator for use on data with high use counts. Then the

developer could decide when it was most appropriate to use whichever tag filter operator.

[Mat]

Did you try [contains:tags[Vorto]] ?

<:-)

[TonyM]

Possibly related To tag search performance.

Unless you use sort[] list of tiddlers with a given tag will be sorted in the order they are listed in a list field. I would assume thus tiddlywiki needs to look for list field existence on any tag.

Also any tag pill for a tag, one exists for every Word in the dictionary tagged with Vorto if you click on the tag pill it will list all 36,000 words so tagged in the tag pill drop down. Are there dropdown populated only once opened or before hand. If before hand every tag pill with contain a 36,000 item drop down.

I note that in my high performing wiki of ESPDIC https://tiddlywiki.psat.com.au/ESPDIC/ that the More Tags field does not include Vorto, and there is no vorto tiddler. 

Regards

Tony

[A Gloom]

I have created a similiar effect in a much smaller but more complex wiki

single file TW, vers 5.1.20, 4.3 Mb file size, ~1700 tiddlers, latest FF, 16 Gb RAM

I already get about a one sec delay when typing in the tag selection field once I went over 500 tags but otherwise the wiki doesn't show much lagging-- except one tiddler full of filters, lists and transclusions.  Only when that tiddler is open in the sidebar, I get what Mark gets-- everything-- typing, clearing the search box, hiding/reveraling the sidebar, switching sidebar tabs has about a 5 second lag.

It uses a bunch of view widfets but then it uses a lot of many other things (including mixed wikitext/html and in tiddler styling) so narrowing down the culprit has been very time consuming.  Need to use devtools but dev tools unusable for me to do serious troubleshooting with.

[A Gloom]

EDIT:

Only when that tiddler is open in the sidebar, I get what Mark gets-- everything-- typing, clearing the search box, hiding/reveraling the sidebar, switching sidebar tabs has about a 5 second lag.

Only when the tiddler is open in the (sidebar) story river *(not sidebar)

[Mark S.]

Hi Mat,

That was a very good thought. Previously I was thinking I would have to ditch tags, which is a shame since Thomas' beautiful list-reveal makes

it so easy to add and remove tags. But contains:tags[Vorto] seems to work without a noticeable performance hit.

Thanks!

[Mark S.]

If anyone is curious, here is the work in progress (only 36k -- half the target lexicon)

https://marxsal.github.io/EODICT/EO-DICT.html

[Mat]

Mark S. wrote:

[...] contains:tags[Vorto] seems to work without a noticeable performance hit.

Ah, great! I must assume this raises the interesting question whether the tags operator can/should be modified to behave like contains:tags instead, perhaps even replace the tags op all together? And what causes tags to misbehave in the first place? (BTW, is @Jermolene following this discussion?) 

<:-)

[TonyM]

The truth is if a large number of items have one tag it makes sense to me not to use a tag. I would instead use a field. Such as vorto and use has:field[vorto] where only the existence of the field is required. Then if I had a set of different types I may use a field such as objectytpe with the value vorto thus select with objecttype[vorto] where object type only ever has one value of a limited set.

Tags should really be reserved for ad hoc unstructured labeling where it makes sense that one or more tags may be assigned. Fortunately it is easy to transfer values between tags and fields

Because tags are so versatile I am not surprised they are used less appropriately.

Before I use a tag a field etc I always ask my self about the nature of the value I am setting. I Actualy do this intuitively so it may be a job to list the conditions I use to choose.
Regards
Tony

[Mat]

TonyM wrote:

Tags should really be reserved for ad hoc unstructured labeling where it makes sense that one or more tags may be assigned. 

 

If this is so, then TW should feature more variants of "tags" (or whatever the appropriate label would be then). For example, it is not trivial to get "myfield:Vorto" to display as a tag pill, which you might want it to.

Because tags are so versatile I am not surprised they are used less appropriately.

Well, obviously Mark is an absolute TW expert so I'd say your interpretation of how tags are to be used is merely an opinion. The question is, again, if the system meets our needs. 

Fortunately it is easy to transfer values between tags and fields

There's certainly no UI for it so it is not trivial. How do you go about it?

<:-)

[TonyM]

Mat, 

 

Mat

Tags should really be reserved for ad hoc unstructured labeling where it makes sense that one or more tags may be assigned. 

 

If this is so, then TW should feature more variants of "tags" (or whatever the appropriate label would be then). For example, it is not trivial to get "myfield:Vorto" to display as a tag pill, which you might want it to.

This is a useful possible improvement, but of course they will not be tag pills but field pills. Of course it depends on what you want from the pill. I like the idea that the field pill drop down could present possible values for the field and a click will populate that field on the current tiddler. Ideally one could set different filter on each field pill from canned values to the each of values found in the field through out the wiki. I am not so good at writing popups yet.

 

Because tags are so versatile I am not surprised they are used less appropriately.

Well, obviously Mark is an absolute TW expert so I'd say your interpretation of how tags are to be used is merely an opinion. The question is, again, if the system meets our needs. 

This is not a criticism of marks design but a suggestion already made with others for performance, it also highlights this will be common for all tiddlywiki users because the tag system is so versatile. 

 

Fortunately it is easy to transfer values between tags and fields

There's certainly no UI for it so it is not trivial. How do you go about it?

<:-)

Again a good suggestion. A UI for this. Some tools provide this (Mohammads commander?) but this is what one does

• Create a List widget that creates the list of tiddlers you want to change
• Place the actions (Widgets) you want performed inside the list
• Wrap all of it in a button.

Here is one I prepared

;Filter: <$edit-text tiddler="$:/temp/bulk/filter" tag=input/>
;Actions:
<$tiddler tiddler="$:/temp/bulk/actions">
<$transclude tiddler="$:/core/ui/EditTemplate/body/editor"/>
</$tiddler>
:`Warning - most actions may be irreversible` make backups
://consider a two step process for intermediate results//

<details open><summary>Tiddlers to be actioned - click to open/close</summary>

<$list filter={{$:/temp/bulk/filter}}>

</$list>


</details>


<$button>
<$list filter={{$:/temp/bulk/filter}}>
{{||$:/temp/bulk/actions}}
</$list>
Apply actions to selected tiddlers
</$button>

 Perhaps a version of this with the ability to inserts filters (From advanced search) and actions from a drop down should be provisioned?

Regards

Tony

[Mark S.]

Now with 63,000 entrees. Eliminated dates and modifier/creator to save space. Comes in just under 10 megs.

Crashed when attempting to save on tablet. Be interested to know if people are able to save on their small

devices.

https://marxsal.github.io/EODICT/EO-DICT.html

[TonyM]

Mark,

I used tidloid's fork a website and it opened but attempting to create a new tiddler it froze. Presumably saving.

LG V20 Android 7.0 64GB internal memory

turning of autosave made it work much better but touching the save changes icon took forever to demonstrate it was happening.

Reloading from tiddloid menu takes quite some time but it seems to work

Regards

Tony

[TonyM]

Here I attach my "Not Quite Empty" edition as an example.

Do you receive it?

[@TiddlyTweeter]

Mark S.

I can use it on Android 8 successfully in Chrome (ONLINE version). Its slower than TonyM's previous but not unworkable. Loading would need, I think, a message so you know something is happening whilst is gears up.

I'll comment later on opening a saved version on Android.

TT

[@TiddlyTweeter]

Mark S. wrote:

... I was thinking I would have to ditch tags, which is a shame since Thomas' beautiful list-reveal makes

it so easy to add and remove tags. But contains:tags[Vorto] seems to work without a noticeable performance hit.

TW has a kinda implicit philosophy of the Uber-Role of Tags. Tags are doing about 85 jobs.

They are for structure. They are for content signalling. They have sophisticated parsing.

Personally I like the freeform taggery of tools like Twitter where #hashtag is simply an in-text string to match.

I think my underlying question is "are we neglecting the text field?"

Hope this is not too off-topic.

Best wishes

TT

[TonyM]

TT

Tiddlywiki allows you to do as you suggest out of the box. Just go ahead and use #tags in text. And filter on text contains or search. Perhaps a little experiment and publish this method is all that is needed. You could throw in some macros to make it easier or an editor toolbar button to insert existing hash tags. Ideally you could register new hashtags in a data tiddler so you know what you can search for or log their use.

Of course if the wiki has a lot of text or tiddler it may be wise to move such hash tags into a tag like field.

Would it be ok to use $:/#tagname because then you could have the option of creating a tiddler and using it as a tag. You could then have a view template that with [all[current]prefix[$:/#]] displaying hash tag tools when you open a $:/#tagname tiddler. References etc will also work with this.

Imagin if you could make # or /# a new namespace like $:/ is

Regards
Tony

[Mark S.]

What is tidloid? It's not in the toolmap. I think autosave should already be off in EO-DICT.

Thanks!

[Mark S.]

On Wednesday, September 25, 2019 at 3:25:05 AM UTC-7, @TiddlyTweeter wrote:

Personally I like the freeform taggery of tools like Twitter where #hashtag is simply an in-text string to match.

I was going to say that the hashtag would add a performance hit, but now I'm not so sure. Using regular tags

actually created a performance hit. But one problem is that people, left to themselves, will make multiple

variations of the same tag.

#myhashtag #MY-hash #TAG-Hash #hash-tag-de-mio

[@TiddlyTweeter]

Its a very interesting issue, I think.

I have NO idea if #hashtag in text field would be a PITA or not on performance in TW, not having tested it.

I just think semantically it makes sense. I does well for the type of thing I write ...

Mr #Darcy, cognisant of the plebesite,  announces #BottledWater should be banned.

You get the idea.

I SIMPLY like to write and sort out an order later.

TT 

[Mark S.]

Ah, but if your tiddler represented the smallest semantic unit, then it would be no trouble to add the Darcy tag at the top.

And you would notice immediately that you already have #DarcyIsPrideful, #DarcyD'Man, and #DarcyHatethBottledWater.

Later, when producing your final copy, you wouldn't have to scrape out all the little # symbols, like pulling burrs off

a dog that's been wandering too long in the 100 acre forest.

“If I ever go looking for my heart's desire again, I won't look any further than my own back yard.

Because if it isn't there, I never really lost it to begin with… "

Ok, a quote from P&P would be better, but I only read it once and never understood what the zombies were doing ...

[Arlen Beiler]

After stepping through conditional breakpoints and other things, the tag operator is indeed the result of the massive slowdown. 

The first line takes a long time to return, about 3 seconds per invocation. The rest of it is fairly quick. There is a source.byTag operation right before it that would be much quicker but I don't know where it comes from or what uses it. I guess that would take some investigating. But this basically means that using the tag operator for any tags which have many tiddlers is basically out of the question. So everyone's basically on the right track it seems. 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/c1878946-113b-49ac-b66c-6a38c6f1f29e%40googlegroups.com.

[Jed Carty]

It is used here:

exports.getTiddlersWithTag = function(tag) {
 // Try to use the indexer
 var self = this,
 tagIndexer = this.getIndexer("TagIndexer"),
 results = tagIndexer && tagIndexer.subIndexers[3].lookup(tag);
 if(!results) {
 // If not available, perform a manual scan
 results = this.getGlobalCache("taglist-" + tag,function() {
 var tagmap = self.getTagMap();
 return self.sortByList(tagmap[tag],tag);
 });
 }
 return results;
};

which caches the results to make it faster afterward, this works well for when a tag isn't on too many tiddlers, but I have no idea what 'too many' is here.

I haven't dug too deeply into the cache mechanism and there may be some maximum size after which it won't cache results, if that is the case than it may have to build the results every time the tag is looked up instead of being able to use the cached result.

[Arlen Beiler]

When I step over the line it takes six seconds to show up for the next line. However, when I run a performance profile of the page and do exactly one action -- opening the control panel -- it still takes about 6 seconds but the performance profile shows thousands of refresh calls down the widget tree one after the next for six seconds. It blames the whole thing on sortByList, and specifically the replaceItem function inside it, but I'm still investigating it.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/160dda32-67cc-4078-b76a-3e2408239d99%40googlegroups.com.

[Arlen Beiler]

Ok, I found the offending line! And I double-checked that one line using Date.now() and it literally does take six seconds. It varies by about 400 ms from one time to the next, but that's completely accurate! Note that the line is executed many times during the sort and that is the total execution time for that line for one sort operation. 

[Arlen Beiler]

I've been studying this function for a half hour (yes, sometimes that's how I work -- inefficient, I know). Anyway, it suddenly occurred to me that 

if new position is -1 then current position is useless, 

• because right now if new position is -1, then it is set to current position
• immediately after that it checks if they are equal and only replaces them if they are not. 

So if new position was set to -1, then it will be set to current position, and therefore is equal to it, and therefore prevents a splice from happening. So we could totally optimize this if new position is still -1 after the first group of if statements and just exit the function if there is nothing to do without looking up the current position. 

if(newPos !== -1) {

  var currPos = titles.indexOf(title);

  if(currPos !== -1) {

    titles.splice(currPos, 1);

    if(newPos >= currPos) newPos--;

    titles.splice(newPos, 0, title);

  }

}

Of course, you still can't use list-before and list-after on 36000 tiddlers, but who has ever done that?

[@TiddlyTweeter]

Great to see something so clear! 

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[Arlen Beiler]

I opened an issue on GitHub: https://github.com/Jermolene/TiddlyWiki5/issues/4275

[TonyM]

Folks,

Great stuff here collaborating to resolve a performance issue.

I hope also to emphasis something I stated early in this thread and in my own example of a high performing version. Tags have a particular ad hoc classification function. One which can be used so many ways, and is great for small wikis, because you can use the same mechanism for many purposes. Of course with lots of different tags the "tag space becomes polluted or messy" It is possible to have multiple tag like fields, see  Generic Tags Plugin but just as tiddlywiki already has the type field for its own tiddler types the best way for a designer to handle the example of classifying a large number of tiddlers as words/Vorto is with a field be it containing a tiddler-type value or just an if exist "word" field. Why because it does not use a heavily flexible mechanism for tags to do a simpler job, and in large quantities this makes a difference.

Not with standing I have what I consider a good technical and design reason for avoiding tags in this case, it is true that this would not occur to most users/designer. As a result perhaps we can encourage the use of a field such as user-type or object-type which contains a single value from a number of possible values defined by a user. We could even support this by providing a drop down that lets you select from each value found in this field anywhere in the wiki (along with edit text to enter new ones). Its about providing a gentle push towards a design pattern that will serve them well.

Regards

Tony

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[PMario]

On Thursday, September 26, 2019 at 10:21:14 AM UTC+2, TonyM wrote:

 ...

but just as tiddlywiki already has the type field for its own tiddler types the best way for a designer to handle the example of classifying a large number of tiddlers as words/Vorto is with a field be it containing a tiddler-type value or just an if exist "word" field. Why because it does not use a heavily flexible mechanism for tags to do a simpler job, and in large quantities this makes a difference.

I think using the tiddler-type field would be a solution here, since it wouldn't need to use the

<$list filter="[all[current]tag[xxx]]">   .... hack, to create a custom ViewTemplate or / and EditTemplate. So the "WordTemplate" could be much simpler.

The TW tiddler-type field is used to specify a MIME-type. It is possible to create your own "vendor specific" (like text/vnd.tiddlywiki) or "personal" (text/prs.dictionary) types, but you need to follow some rules.

Personal or vanity tree

The personal or vanity tree includes media types associated with non publicly available products or experimental media types. It uses the prs. tree prefix:

type "/" "prs." subtype ["+" suffix] *[";" parameter]

Examples: audio/prs.sid, image/prs.btif.

from Wikipedia

So a text/prs.dict.word;language=en-EN or something similar, would be a possibility to work with a specification as described in the OP.

The problem is, that we don't have good end-user support for content like this in TW at the moment.

just some thoughts.

-mario