Request for help with new filter operator and new tutorial

120 views
Skip to first unread message

Diego Mesa

unread,
Jun 25, 2018, 1:57:54 PM6/25/18
to TiddlyWikiDev
I would like to make a new filter operator: fuzzysearch which makes use of this library:


in the hopes of submitting something to:


The problem is, I'm not at all sure how to do it! I know I can make a new tiddler with:


and then somewhere do

var fuze = require(...)

but thats about it. What I would like to do is ask the community for help! Not just to make the filter operator, but to make it a full-fledged tutorial! (This is the part that I can really help with!). We can then add it to the tiddlywiki.com/dev site! 

What do you all think? Kill two birds with one stone! 

Jed Carty

unread,
Jun 26, 2018, 5:41:25 AM6/26/18
to TiddlyWikiDev
I have looked at this a little and I may be able to help. I am not sure how much time I will have in the immediate future because there are some potentially major bugs in Bob that I need to take care of in addition to my day job but this looks like a good project.

I think that using the way that the sjcl encryption library is included is a good start. The minified sjcl.js file as part of the core plugin along with a .meta file for it. I think that copying that is the way to go. I am not exactly sure how the library is loaded, that may be enough.

The library is used in the crypto function in boot.js here https://github.com/Jermolene/TiddlyWiki5/blob/929b0c98330ad25aa4a32cd201274a6e89cd8902/boot/boot.js#L601

Looking at that it has different methods of loading depending on platform, on node it uses require like you suggested, but that isn't available in the browser but the library is available as window.sjcl. I assume this is done when the sjcl.js library is loaded, either by tiddlywiki or the library itself, but I am not certain.

I hope that is enough to get started because I have to run for a bit.

Jeremy Ruston

unread,
Jun 26, 2018, 9:21:38 AM6/26/18
to TiddlyWikiDev
Hi Jed


On 26 Jun 2018, at 10:41, Jed Carty <inmy...@gmail.com> wrote:

I think that using the way that the sjcl encryption library is included is a good start. The minified sjcl.js file as part of the core plugin along with a .meta file for it. I think that copying that is the way to go. I am not exactly sure how the library is loaded, that may be enough.

The integration with the SJCL encryption library is actually not a good example: the SJCL library has to be available during the boot process, which means that it can’t be packaged as a tiddler in the usual way. Better examples are the following places where third party JS libraries are integrated:

https://github.com/Jermolene/TiddlyWiki5/tree/master/plugins/tiddlywiki/xmldomhttps://github.com/Jermolene/TiddlyWiki5/tree/master/plugins/tiddlywiki/codemirror

(To make that list I searched the repo for files called “tiddlywiki.files” and then pruned out some outdated examples).

Best wishes

Jeremy.

Jed Carty

unread,
Jun 26, 2018, 9:38:36 AM6/26/18
to TiddlyWikiDev
I hadn't realised it could be done that way. That makes including external libraries a lot easier than I thought. I am a bit embarrassed that I didn't notice you could do that.

Diego Mesa

unread,
Jun 26, 2018, 3:45:42 PM6/26/18
to TiddlyWikiDev
Hey all,

Thanks Jeremy for sending that. The following is more pseduo-code than anything, but reflects my quick hacking:

(function(){


/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";


var fuze = require("$:/plugins/fuzzy/fuze.js");


var options = {
  shouldSort: true,
  matchAllTokens: true,
  threshold: 0.6,
  location: 0,
  distance: 100,
  maxPatternLength: 32,
  minMatchCharLength: 1,
  keys: undefined
};


/*
Export our filter function
*/
exports.fuzzysearch = function(source,operator,options) {
    var results = [];


    source = options.source || $tw.wiki.each;


    var searchTiddler = function(title){
        jsonData = $tw.wiki.getTiddlerAsJson(title)
        var fuse = new Fuse(jsonData, options); // "list" is the item array
        var result = fuse.search(operator.operand);
        return result;
   };
    source(function(tiddler,title) {
        if(searchTiddler(title) !== options.invert) {
            results.push(title);
        }
    });


    console.log("e");
    return results;
};


})();

I quickly generated this just by looking at:

$:/core/modules/filters/search.js

which is just a wrapper around:

$:/core/modules/wiki.js

where exports.search = is defined.

The issue is that as you can see from 


Fuze works better if it would have access to the entire json string of the wiki, and it can handle the searching and scoring itself. 

What do you guys think? Any tips to help me move forward? 

Best,
Diego

Jeremy Ruston

unread,
Jun 27, 2018, 3:28:18 AM6/27/18
to tiddly...@googlegroups.com
Hi Diego

Filter operators are often re-executed and so it's important that they perform well. For an expensive operation like a search we need to try to cache as much of the work as we can.

An approach that should work is to introduce a new startup module that tracks changes to the store in order to keep the index up to date, and then have the filter operator call a version of the search function that caches the results (and clears the cache when it is invalidated by changes to the tiddler store).

Having said that, one of the client projects I'm working on for Federatial looks like it's going to need an enhanced search capability. If the project proceeds to the next phase then I'll be looking at adding a third party search index library -- I've been looking at https://lunrjs.com/ and http://elasticlunr.com/. The API that they feature allows the index to be dynamically modified by adding and removing data incrementally; fuse.js looks like it only accepts a monolithic block of data, and would require a re-index each time the tiddler store changes.

Best wishes

Jeremy
--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Diego Mesa

unread,
Jun 27, 2018, 9:47:06 AM6/27/18
to TiddlyWikiDev
Hey Jeremy,

Thanks for your feedback - I agree search needs to performant. Im glad you're also looking at these issues! 

A priority for me is the inclusion of fuzzy searching - a quick look seems lunr does support it: 


but elastic does not. 

Best,
Diego

Diego Mesa

unread,
Jun 28, 2018, 10:14:27 AM6/28/18
to TiddlyWikiDev
Hey Jeremy,

Just wanted to let you know of this plugin:


which incorporates lunr.js into TW for full text searching. I just tried today and am somewhat able to do fuzzy searching with it (does not deal with immediatel searching properly yet). I have let him know about this discussion and the corresponding feature request on github, and have opened an issue on his repo for the fuzzy matching:


Perhaps you can reuse some of hoelzro's work?

Best,
Diego

Rob Hoelz

unread,
Jun 28, 2018, 12:09:21 PM6/28/18
to TiddlyWikiDev
Hi everyone!

Diego was kind enough to bring this discussion to my attention - I plan on fixing up that issue he ran into.

Jeremy - if you decide to go with lunr.js but not use my plugin, please feel free to reach out if you have questions about lunr.js!  One pitfall I ran into while upgrading from lunr 1.x to 2.x is that lunr's index is immutable by default in 2.x - I ended up writing a plugin to add mutable indexes.  Of course, any collaboration on my plugin (along with advice on how to improve it) would be most welcome!

-Rob

Diego Mesa

unread,
Jun 29, 2018, 9:20:18 AM6/29/18
to TiddlyWikiDev
All, 

Rob has fixed the issue with his plugin, so we can now have fuzzy search!!!
Reply all
Reply to author
Forward
0 new messages