Improve Search in Tiddlywiki

[Atronoush Parsi]

I think most of you have the same opinion as me that the TW standard search loses its efficiency as soon as your Tiddlywiki gets large. For example in a TW with 2000 notes (tiddler) most of the time I DO NOT get what I want when I make a search using Tiddlywiki standard search. I know Advanced Search is there and I can do more precise search, but at the cost of entering some filters!

So, the question is how we can improve the search quality of TW?

There are some search tools out there. If you look at the https://links.tiddlywiki.com/ and search for search and search tools you will find plenty of links, but you can not find a tool to help you here!

I appreciate sharing ideas, tools, links if they address improving searching in Tiddlywiki!

One idea is:

Can TW rank and index searches to improve its precision?

[Charlie Veniot]

I'll I could ask for is that search as-is remains, and that improved search be a separate thing, ideally a plug-in of some kind.  So the many who do want it can easily add it, maybe easily customize it.

[Michael Wiktowy]

I end up using Advanced Search and using filters (especially [regexp:text[]]) a lot but I always forget the filter syntax for certain things so I add extra filter drop-downs with some pre-set templates using the system tag: $:/tags/Filter

https://tiddlywiki.com/#SystemTag%3A%20%24%3A%2Ftags%2FFilter

I am not sure where the documentation describing this is but you can clone core tiddlers as starting example and tweak:

https://tiddlywiki.com/#%24%3A%2Fcore%2FFilters%2FAllTags

For various projects I mulled adding another search box to be able to use some basic syntax to use in an "intermediate" search like quotes for exact string matches and maybe some boolean operators to combine multiple search terms. It would be probably be fairly simple to add another Advanced Search tab with "SystemTag: $:/tags/AdvancedSearch" with a search box that interprets that extra syntax over to the equivalent filter. The question is where do you end with additional syntax before you run into the complexity that compares with filter syntax. There is always one more thing that you want to add.

/Mike

On Tuesday, May 11, 2021 at 12:22:59 PM UTC-3 Atronoush wrote:

[Eric Shulman]

On Tuesday, May 11, 2021 at 12:41:21 PM UTC-7 mwik...@gmail.com wrote:

I end up using Advanced Search and using filters (especially [regexp:text[]]) a lot but I always forget the filter syntax for certain things

You might like my "PowerSearch", which generates complex search filter syntax using a set of drop-down lists.

Just copy these two tiddlers to your TW and it automatically adds a "PowerSearch" tab to the $:/AdvancedSearch

http://tiddlytools.com/filtergenerators.html#TiddlyTools%2FFilterGenerators%2FPowerSearch

http://tiddlytools.com/filtergenerators.html#TiddlyTools%2FMacros%2Fedit-list

enjoy,

-e

[A Gloom]

@Eric,

While studying your search tools, I noticed a possible bug with your tag cloud and wanted to leave a note before I forget and in case no one else has caught it.  The order by count (but not title) breaks tags with multiple words into separate entries.

As for an improved main search bar, couldn't one modify a copy of the original search bar (I've done it for caption search)-- or in this case with a more refined search filter-- and use it instead of the original search bar by simple swapping of the sidebar segment tag?

[Eric Shulman]

On Tuesday, May 11, 2021 at 10:14:13 PM UTC-7 barro...@gmail.com wrote:

@Eric,

While studying your search tools, I noticed a possible bug with your tag cloud and wanted to leave a note before I forget and in case no one else has caught it.  The order by count (but not title) breaks tags with multiple words into separate entries.

Excellent catch!

FIXED.  tagcloud_sort() now encloses tag text within doubled squarebrackets, like this:

\define tagcloud_sort()

<$list filter="[enlist<items>!nsort[]]">

   <$list filter="[<currentTiddler>split[;]last[]]"><$text text="[["/><<currentTiddler>><$text text="]]"/></$list>

</$list>

\end

enjoy,

-e

[Eric Shulman]

On Tuesday, May 11, 2021 at 10:14:13 PM UTC-7 barro...@gmail.com wrote:

As for an improved main search bar, couldn't one modify a copy of the original search bar (I've done it for caption search)-- or in this case with a more refined search filter-- and use it instead of the original search bar by simple swapping of the sidebar segment tag?

Yes.  You can add $:/tags/SideBarSegment to the PowerSearch tiddler to add it to the sidebar.

To make default PowerSearch layout better fit the sidebar, I have updated the options panel (the "gear" icon) and display of the generated filter syntax:

1) added "input width" setting - was previously hard-coded to "25em"... now defaults to "100%"

2) changed default "maxheight" from "auto" to "20em"

3) changed default "columns" from "3" to "1"

4) generated filter syntax uses a smaller font, aligned left, with "word-break: break-all"

enjoy,

-e

[TiddlyTweeter]

Ciao Eric

That is a really neat tool for complex searches!

TT, x

Eric Shulman wrote:

You might like my "PowerSearch", which generates complex search filter syntax using a set of drop-down lists.

Just copy these two tiddlers to your TW and it automatically adds a "PowerSearch" tab to the $:/AdvancedSearch

http://tiddlytools.com/filtergenerators.html#TiddlyTools%2FFilterGenerators%2FPowerSearch

http://tiddlytools.com/filtergenerators.html#TiddlyTools%2FMacros%2Fedit-list

[TiddlyTweeter]

Atronoush wrote:

I think most of you have the same opinion as me that the TW standard search loses its efficiency as soon as your Tiddlywiki gets large. 

Right, sort of ...

The TYPE of search you need actually is related, really, to the nature of the content of the wiki so I don't think, on developed apps, with larger content, you could ever have a totally generic standard method that would work optimally in all wiki. 

WHY? Because different wikis place more importance on "text" or "title" or other fields.

In my own case, as a liker of Regular Expressions, I tend to use hacks that utilise it for searching. But optimal use of regex requires understanding of its obscure syntax.

JUST a comment!

TT 

[Charlie Veniot]

On Wednesday, May 12, 2021 at 6:37:57 AM UTC-3 TiddlyTweeter wrote:

Atronoush wrote:

I think most of you have the same opinion as me that the TW standard search loses its efficiency as soon as your Tiddlywiki gets large. 

Right, sort of ...

The TYPE of search you need actually is related, really, to the nature of the content of the wiki so I don't think, on developed apps, with larger content, you could ever have a totally generic standard method that would work optimally in all wiki. 

WHY? Because different wikis place more importance on "text" or "title" or other fields.

 That's exactly what I was thinking, without consciously knowing I was thinking it, and expressed way more coherently than I ever could hope to express.  Damn it, that was really good.

[A Gloom]

Exactly, TT has always been an advocate on matters of searching.  With my different wikis, the main search often gets modified for each, depending upon what I want to the search bar to search for (often excluding tiddlers from its search). The advanced search I felt was too much for the end user that isn't working with the wiki but rather just reading/navigating it.

If I was going to suggest anything for the general searchbar, it would be a setting to give it's search filter exclusionary pareameters like the sort parameter for the toc macro.  BTW, does the vertical toc have that parameter?

With my current project, I am going to modify the search bar to do separate searches for words between title and text, additional phrase search*, to reduce/separate the search results.  The phrase search is a feature since it will be used for a poetry collection wiki.  The selection button would be up in the toolbar.  Almost tempted to have the results show in a sidebar tab.

[TiddlyTweeter]

cj.v...@gmail.com wrote

 That's exactly what I was thinking, without consciously knowing I was thinking it, and expressed way more coherently than I ever could hope to express.  Damn it, that was really good.

You are too kind Batman! I'm better being Robin.

TT, x

 

[TiddlyTweeter]

 barro...@gmail.com wrote:

With my different wikis, the main search often gets modified for each, depending upon what I want to the search bar to search for (often excluding tiddlers from its search). The advanced search I felt was too much for the end user that isn't working with the wiki but rather just reading/navigating it.

Right. I think it is an issue in TW public release generally that it is actually TOO sophisticated.

IMO we should hide much more. TW is absolutely brilliant. So featured even God is using it.

BUT end-user "overload" may actually be an issue.

"Search" is one of those issues. IMO, for public release, the maker of a wiki should decide which search filter is best for their content and their users and implement it and, basically, switch OFF everything else on search.

I'm sounding horrendously opinionated! Eeek!

I just have the worry that TW in public is too-featured too-early. 

And that limiting search to be bespoke for it's content is the right way to go.

JUST a comment, I'm not REALLY sure ...

TT

[Atronoush Parsi]

Thank you all for your input!

The PowerSearch by Eric is great as I learn alot from that!

I recommend it for learning filters!

--Atro

[A Gloom]

In trying to leverage what TW already has and to avoid too many home brewed mods (my mods are often questionable), here's what I use to modify the main search box:

https://tiddlywiki.com/#Customising%20search%20results

It details how to add custom search visualizations to the default search bar and set the new visualizaton as default.

I often clone $:/core/ui/DefaultSearchResultList (renaming it so it will go into the directory where you may have other custom system/shadow tiddlers) to make a new custom search visualization.  Often to split the title/tag/text results smaller segments.  Or replace the search filter fields filters with custom search filters to meet my needs, like excluding tiddlers like added sidebar tab tiddlers (that I don't want to $:/ prefix) that don't exclusively hold content.

What would be of use, would be paginated search results.

[bimlas]

This can be useful if you use a lot of fields or tags to indicate different properties:

Locator provides a universal interface for navigating between your tiddlers and searching, filtering them without the need to write filter expressions. The plugin contains a table of contents macro and an enhanced search engine that contains a visual filter builder to filter results by tags and field values. You can organize the tiddlers to any number of trees, even by custom fields or links in text. Unlike classic Table of Contents, standard search, filter search and list of tags, this plugin handles these functions organically.

https://bimlas.gitlab.io/tw5-locator/

[Atronoush Parsi]

Thank you all!

@bimlas

locator is quite powerful! Of course it has a learning curve!

@A Gloom,

Thank you for the link!

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/eb2df6f3-73f4-4d1d-bffe-95db43189acbn%40googlegroups.com.

[A Gloom]

@ Atronoush

Glad to help.

@Eric

Studying PowerSearch for inspiration-- I may created a simpler version tailored to the end user-- yours is definitely for power users : )

Your tag cloud has been adopted (with full credit to you) for a poetry TW I'm helping someone with.  I've been spreading the word of TW in my travels and we will be presenting this poetry edition of TW here in the future.