Since my original project of 85,000 tweet records ran too slow to make it pretty much worthless I scaled down the project. 8-(
Still have some work to do in it.
You can find the new project - http://wardjhtweets.tiddlyspot.com/ (2,800 Tiddlers)
Old project can be found - http://wardjhtrumptweets.tiddlyspot.com/ (85,000 Tiddlers)
There's no indexing system in TW, and thus no scalability. Indexed databases were able to handle hundreds of thousands of records even with 1980s hardware.
There's a way to send empty tiddlers via the server that only get populated on demand. The problem then is that your search will only see the titles and tags (fields ??) for searching. But if what you wanted was link and tag based (without depending on text searches) then it should work OK. You would need a separate (or enhanced) search box to fetch tiddlers from the server directly.
Or, maybe someone could write a system that would store an index to the current TW's text context. Then a special search would look through the index and use that for creating a list of links of tiddlers. The tiddlers would become populated when clicked on. The index would be updated on saving (harder) or on demand.
My first attempt to import the tiddlers into a pre-release version crashed the browser tab. So I thought that maybe I needed to use node to render out the tiddlers as tids. Now we hit another scalability wall. The process has been running 30 minutes and isn't even a quarter of the way through. Looks like converting to tids will take two hours!
Next question, is the entire master branch on github considered the pre-release?
-- Mark
On Friday, April 6, 2018 at 7:37:19 AM UTC-7, @TiddlyTweeter wrote:BTC
Good thought. I'll point the author to your suggestion.
TT
BurningTreeC wrote:For a good comparison I would like to see that big one using the latest prerelease version, where some performance tweaks have been done
--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/5e037fd3-3d96-49c8-b006-8072e34f7382%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
undefined
. This is excellent news, as it shows that the V8 team is working towards improving the performance of delete
. However, the delete
operator will still result in a significant performance hit on property access if a property other than the most recently added property is deleted from the object. So overall we have to continue to recommend against using delete
.From the second link, requestIdleCallback() appears to be very interesting - if the browser support is given. It reschedules workloads to idle-time like requestAnimationFrame()
....
I have been considering trying to use TaffyDB (http://taffydb.com) with tiddlywiki but I haven't had a chance to try it out yet and I have no idea if it would actually help at all. Another option is to use an in-browser sqlite database and maybe web workers.
... Perhaps we should start a discussion specifically on settings and tools for large wikis?... When it come to scale there are a number of performance issue's, one of which NoteSelf solved by showing the "Loading" message before TW can respond at all. What I quickly realised is when you have a lot of tiddlers, quite a few assumptions made in a regular tiddlywiki's, for features and responsiveness are no longer valid. Such as a search that responds keystroke by keystroke.
There is no reason why we can't supply some tools to support such large data-sets.
... What I suggest is we bundle a range of settings together, perhaps in a plugin designed to reconfigure TiddlyWiki for larger data sets...
I perceive the results-as-you-type thing as more of a toy -- not a usability development.
Not just TW, but anywhere it's used. It may make sense when it's attached to a powerful back-end server database, but not so much when you have a local, non-indexed data source.
In the case of TW, it interferes with usability particularly on small devices. Re-rendering with each keystroke is another related problem that affects all TW's performance. I can only assume most users are not touch typists, have very powerful machines, have small TW's, or are using screen keyboards.
VERY useful info! Who knew but you? :-) the problem with Hidden Settings is they are ... Hidden :-).
VERY useful info! Who knew but you? :-) the problem with Hidden Settings is they are ... Hidden :-).
Since there's a hidden setting to control minimum key length, it must be possible internally to change the mechanism so that it only searches upon enter. Maybe there could be a hidden setting to turn off key-by-key searching?
Yes -- that's what I said. But is there a setting to turn off key-by-key searching and replace it with traditional type-and-enter ?
Thanks!
Mark
<$reveal tag="div" state="$:/config/Search/MinLength" type="nomatch" text="50" default="">
<$button>Hide<$action-setfield $tiddler="$:/config/Search/MinLength" text="50"/></$button>
</$reveal>
<$reveal tag="div" state="$:/config/Search/MinLength" type="match" text="50" default="">
<$button >
<$action-setfield $tiddler="$:/config/Search/MinLength" text="3"/>
Show
</$button>
</$reveal>