Simple fuzzy search plugin TwFusejs beta 0.9.0 ready for test driving

[TheDiveO]

My Fuse.js-based TwFusejs plugin for TW5 has now reached beta status. It's on purpose a lightweight solution, and not a one-size-fits-it-all search plugin. As others have pointed out, there are very promising and powerful search plugins when you can live with managing an search index. This plugin aims at the beginners with smaller wikis who are interested in a maintenance-free fuzzy search entry solution. It's an experiment whether Fuse.js is suitable in such situations.

It neatly integrates with the sidebar search box. It also has two simple control panel settings: one for searching for all or any search term, and another to make the fuzzy results tab your default search result tab.

• Live demo of my TwFusejs plugin
• Plugin description page
• TwFusejs GitHub repo

Constructive feedback welcome, pull requests even much more so ;)

-- TheDiveO

[Mat]

TheDiveO - thanks for sharing!!!

May I ask, what does regular search bring that your fuzzy search does not? I.e what's the point in also keeping regular search?

<:-)

[TheDiveO]

Good question! There are two reasons at the moment for keeping the regular search:

• fuse"s fuzzy search does not necessarily rank exact matches high enough,  that's at least my impression. Sp until we have gained enough experience,  I'm hesitant to replace the regular search.
  
• the plugin adds the fuzzy search result tab, because for replacing the default search the plugin needs to overwrite the core's default search tiddler(s). Not least for the reason above I'm hesitant to do so at the moment.
  

[@TiddlyTweeter]

Its interesting. In your test I entered "frank" &, later, "gerald" searches and got results. I'm not quite sure what to do with them :-)

I think its scope is interesting. Can the user define "how fuzzy" it is?

Just an early comment.

Josiah

[TheDiveO]

Yes, you can tweak the searches using options data tiddlers. This is detailed in the demo wiki.

That's also what I want feedback on, as fuse.js seems too fuzzy to my expectations. However, I played safe with the default options setting in the plugin, sticking nearly to the original defaults, until there's more field experience and feedback.

As for Gerald ... he surely is special. Ok, enough obscure jokes for now.

[Mat]

TheDiveO wrote:

Good question! There are two reasons at the moment for keeping the regular search:

• fuse"s fuzzy search does not necessarily rank exact matches high enough,  that's at least my impression.

Does this mean that you're not personally involved in developing the algos in it but mostly to "package" it for TW? (I applaud either effort, of course)

Otherwise - and I understand this is naive - but isn't it just to rank exact matches at top?

Plus... does fuse's fuzzy search rank title matches and tags matches higher than text matches as is done in standard TW search? 

And, come to think of it, does it search other fields than title, tags, text?

Again, thank you for this!

<:-)

[TheDiveO]

I'm "merely" the packager, or integrator, but I'm not involved in developing the original fuse.js algorithm. I had need for a fuzzy serach in at least one of my work TWs, but didn't dare to dive too deep into the topic of Javascript search libraries, so I started with fuse.js. Maybe we can give the author of the search library more feedback to get it improved.

As far as I understand fuse.js, it does not necessarily give exact matches a better score than fuzzy ones. This might be related to the seemingly strange distance parameter where hits are to be expected. This is something the author of fuse.js might know better about.

There is a way to rank matches in different fields differently, that's also documented in the demo. Unfortunately, that more or less breaks matching a set of search terms across multiple fields, so I'm currently joining fields before the search; this is also documented. You can easily override this behavior, please see the demo wiki.

At the moment, the default is to search title, text, and tags only. This can be changed, and ... it's also ducmented in the demo. I didn't add control panel settings as this requires some work, and if the library turns our to be not good enough in searching, then the effort would be spent in vain. So, it's configuring a data tiddler by hand at the moment.

[@TiddlyTweeter]

TheDiveO wrote: "... the seemingly strange distance parameter where hits are to be expected ..."

The results can be a bit strange. Its likely its very sensitive to small tweaks?

I looked at the "fuzzy settings" ... there is no "in-between." It seems to me that it may operate within a range that needs SMALL STEPS to hone?

Just speculation
Josiah

[TheDiveO]

That's what I suspect; to make it match less, I turned down(!!!) the threshold option value to 0.35; YMMV. The author of Fuse.js claims that's that what fuzzy is about. Oh well. ;)

[Mark S.]

Your documentation seems to suggest that there are other solutions that can do indexed searching. Do you know of any such products?

Thanks!

-- Mark

[TheDiveO]

https://github.com/hoelzro/tw-full-text-search

[@TiddlyTweeter]

UPDATE: This replaces a post I inadvertently sent out before it was finished.

Ciao TheDiveO

I had a closer look at the plugin. It looks interesting. I'm going to see if I can optimise the JSON for a specific use case where I have a lot of tiddlers where only the text field is important. I'm not done on that yet. What I have noticed playing the settings ...

1 - Quite small changes in the settings can have quite big effects on results. That might make it very good for tuning for some situations.

2 - I think to test it properly you need a fairly large set of data that is consistent in some way (like its all plain text and all text fields about the same length). Just for test purposes. But it gets easier to understand what its doing I think.

3 - Out-of-the-box good: it is good at coping with spelling errors; and automating adding plurals when you search for only the singular. For some use cases, like, for instance searching literary fiction, that is very useful indeed.

Best, Josiah

TheDiveO wrote:That's what I suspect; to make it match less, I turned down(!!!) the threshold option value to 0.35; YMMV.

[Thomas Elmiger]

Great, idea – thank you very much for sharing!

Cheers,

Thomas