Searchwikis Plugin: Search multiple wikis from one central wiki

[Mohammad]

Ref: Indexing: a wiki holds search indexes to other wikis

Announcement: Searchwikis Plugin

Date: Mar 3rd, 2020

Release: 1.0.0

Status: Stable

Searchwikis Plugin

3rd March 2020 at 10:27pm

start

Tiddlywiki5.1.21+

 

LicenseMIT

 

Release1.0.0

 

Statusstatble

What is Searchwikis plugin?

Searchwikis plugin uses index (dataTiddler) to search external Tiddlywikis. It has two parts

1. An indexer, to build an index of all tiddlers in external wiki
2. A search tool to search indexes and display a link to external wiki

Then one central wiki can hosts many index tiddlers and lets to search several external wikis through index tiddlers.

How to use

• install $:/plugins/kookma/searchwikis into the host wiki
• drag and drop index and export tiddler to the external wiki
• use the ui and create the index tiddler of external wiki
• export it to the host wiki, or drag and drop to the host wiki
• open $:/AdvancedSearch in the host wiki
• goto External wiki tab and make search
• results will be shown as links and one can click a link and open the external wiki with that result under focus

Install

Packaged plugin

1. Drag and drop $:/plugins/kookma/searchwikis into your wiki
2. Save and reload your wiki

Client-Server installation (separate files)

• For Node.js (server version) download the searchwikis folder from source folder and paste in your Tiddlywiki plugins folder
• For local installation put the searchwikis folder into your local plugins folder under your wiki folder

Code and demo

• Demo: https://kookma.github.io/TW-Searchwikis/
• Code: https://github.com/kookma/TW-Searchwikis

For complete tutorial, examples and use cases, see the demo page.

It is highly recommended to backup your data before trying any new plugin! Do it before installing Searchwikis

Terminology

Host wiki

a Tiddlywiki has index tiddlers containing information of other Tiddlywikis. Host wiki is also called centeral wiki.

Index

a data tiddler contains the information of an external wiki in the form of

title: title|caption|tag

External wiki

another Tiddlywiki for which, an index of information of all tiddlers (like title, tags, caption) is created and is desired to be searched from a host wiki

Search index tiddler

making search in an index (dataTiddler) contains information about an external wiki and show links to the tiddlers of that wiki meet the search criteria

History

Revision 1.0.0

• Date: [3rd Mar 2020]
• [NEW] new index and search mechanism
• [NEW] search multiple indexes at the same time
• [NEW] added interface to $:/AdvancedSearch
• [NEW] added documentation and examples

For all changes see History

Acknowledgement

• Mat
• bimlas
• David Gifford
• Olegh Bondarenko
• Ilya

Authors

• Mohammad
• Mark S

Reference

• http://kookma.searchwikis.tiddlyspot.com/
• https://groups.google.com/d/msg/tiddlywiki/8qyPUgmpDYY/_c3rplBLAgAJ

License

• Distributed under an MIT license

[Mohammad]

For email readers:

 The original post has been edited.

The demo has two index from

1. TW-Scripts

2. Tiddlywiki.com

If you like it star it at GitHb

 https://github.com/kookma/TW-Searchwikis

--Mohammad

[Mohammad]

Announcement: Searchwikis Plugin

Date: Mar 3rd, 2020

Release: 1.1.0

Status: Stable

Revision 1.1.0

• Date: [3rd Mar 2020]
• [FIXED] typo in documentation
• [FIXED] temporary tiddlers are system tiddlers now
• [FIXED] all global macros removed

• Demo: https://kookma.github.io/TW-Searchwikis/
• Code: https://github.com/kookma/TW-Searchwikis

Please send your feedback and star at GitHub, only IF you like it!

--Mohammad

[TiddlyTweeter]

Sometime we need a LIST of your brilliant wikis ... :-)

I lost count of what you have done....

So much of it is good.

I think a list would help get them used!

Best wishes

TT

[Diego Mesa]

I agree!! Available at

https://kookma.github.io/

[Sylvain Naudin]

Wahou ! Impressive so congrats !

My first test find a little bug ;)

Search 'Creden', click on only one result, and fail. (cause ":" I think in tiddler name, first look i think it was about space but no 'cause it work fine with space)

Best regards

Sylvain

[David Gifford]

Hi Mohammad

Wow, this is great! Thank you so much for this.

I not only got it working on my main files, I also fiddled with the CSS and got it looking how I want it to look. (image attached)

Blessings to you today!

[David Gifford]

Updated the Toolmap entry with the new demo address.

On Tuesday, March 3, 2020 at 1:31:24 PM UTC-6, Mohammad wrote:

[Mohammad]

Hi Sylvain,

 Thank you for bug report. 

Colon character in URL address!

It seems this is Tiddlywiki issue in encoding and interpreting the colon(:) character.

See: https://tiddlywiki.com/#PermaLinks

About URL encoding

There are technical restrictions on the legal characters in an URL fragment. To allow all tiddler titles to be addressed, illegal characters are subject to a process called "URL encoding" whereby problematic characters are replaced by their numeric code. For example, the space character is replaced with %20.

Both the target tiddler title and the story filter should be URL encoded (but not the separating colon). TiddlyWiki generates properly encoded URLs which can look quite ugly. However, in practice browsers will usually perfectly happily process arbitrary characters in URL fragments. Thus when creating permalinks manually you can choose to ignore URL encoding.

So, for WebServer Parameter: credentialsWebServer Parameter: credentials title it confuses and TW thinks it is several tiddlers in the story view!

I will have a look to see how resolve this

--Mohammad

[Mohammad]

Hi TT

Thank you for your support.

I did not forget what I promised in another thread.

By the way for the meantime please look at https://github.com/kookma?tab=repositories

All pugings with tag plugin are there!

Hi Diego

For sure, I will do it. I am just learning how to create a plugin library from Andrew Harisson and Jed Carty.

I also promised bimlas to create a tutorial for making plugin library :-)

I will hopefully make it with a dedicated page on GitHub

Cheers

Mohammad

[Mohammad]

Hi David,

Glad you like it. I will add documentation describing how to use it with local wikis and how with wikis on the net.

Thank you for sharing your customization!
I tried to follow Tiddlywiki approach in Advanced Search tool!

So, Searchwikis has a search result tiddler you can change it and use our own search result tiddler and any customization you like!

--Mohammad

[Mohammad]

On Wednesday, March 4, 2020 at 7:14:06 AM UTC+3:30, David Gifford wrote:

Updated the Toolmap entry with the new demo address.

Thanks. I will later remove the tiddlyspot wiki!

Ah, and I do appreciate the great contribution by Mark!

--Mohammad

[Mohammad]

Sylvain,

 I addressed this issue. See next release within an houre!

The TW-Searchwikis can now handle colon in the title of tiddler.

--Mohammad

On Wednesday, March 4, 2020 at 1:09:32 AM UTC+3:30, Sylvain Naudin wrote:

[Mohammad]

Announcement: Searchwikis Plugin

Date: Mar 5th, 2020

Release: 1.2.0

Status: Stable

Ref: Indexing: a wiki holds search indexes to other wikis

 

This is a new release with tons of new features. See history below.

Now you can enable/disable and index, exclude items in indexing external wikis and much more!

If you like it, open GitHub from Code page below and star it!

--Mohammad

Searchwikis Plugin

4th March 2020 at 11:59pm

start

Tiddlywiki5.1.21+

 

LicenseMIT

 

Release1.2.0

 

Statusstatble

What is Searchwikis plugin?

Searchwikis plugin uses index (dataTiddler) to search multiple external Tiddlywikis. It has two parts

1. An indexer, to build an index of all tiddlers in an external wiki
2. A search tool to search indexes and display a link to a tiddler found in an external wiki

Then one master wiki can hosts many index tiddlers and lets to search several external wikis through index tiddlers.

How to use

• install $:/plugins/kookma/searchwikis into the host wiki
• drag and drop index and export tiddler to the external wiki
• use the ui and create the index tiddler of external wiki
• export it to the host wiki, or drag and drop to the host wiki
• open $:/AdvancedSearch in the host wiki
• goto External wiki tab and make search
• results will be shown as links and one can click a link and open the external wiki with that result under focus

Install

Packaged plugin

1. Drag and drop $:/plugins/kookma/searchwikis into your wiki
2. Save and reload your wiki

Client-Server installation (separate files)

• For Node.js (server version) download the searchwikis folder from source folder and paste in your Tiddlywiki plugins folder
• For local installation put the searchwikis folder into your local plugins folder under your wiki folder

Code and demo

• Demo: https://kookma.github.io/TW-Searchwikis/
• Code: https://github.com/kookma/TW-Searchwikis

For complete tutorial, examples and use cases, see the demo page.

It is highly recommended to backup your data before trying any new plugin! Do it before installing Searchwikis

Security note

• Most browsers do not allow to open a local wiki (a wiki file on disk) from a webpage! This is a security reason
• Wikis are opened on Node.js even from local disk, are using http, or https and act as webpage

Tutorial

See Searchwikis Tutorial for details.

History

Revision 1.2.0

• Date: [5th Mar 2020]
• [FIXED] issue in generate links for tiddlers with colon (:) in their title
• [FIXED] the index and export tiddler has a user friendly title Index and Export
• [NEW] Index and Export supports exclude filter
• [NEW] Index summary to show tools for manupulation the indexes
• [NEW] Indexs can be excluded temporaririly from search
• [NEW] improved documentation and tutorial

For all changes see History

Authors

[Mohammad]

Diego, Josiah,

This is my first successful attempt to create a list of plugins and distribute using a plugin library tool

https://kookma.github.io/TW-PluginLibrary/

It contains most stable plugins and installing is the same as install from official library!

The list is no complete, but it will gradually completed to cover all tools under Kookma brand :-)

Kookma means The Sun.

--Mohammad

[TonyM]

Mohammad,

You are so productive and prolific making things we all need, thanks so much. 

I have dropped this on my master bobexe wiki and indexed my tiddlywiki syntax and plugin repository of results.

One issue

However I clicked on one which went to this local address;

http://192.168.1.81/IndexWikis/TW5References/index.html##CSS%20Tools

But the wiki opens with default tiddlers, not the CSS%20Tools tiddler, could the "##" have something to do with it?

System tiddlers

For my plugin repository I would like my index to include the plugin and other system tiddlers and there fields, Is this possible?

Some minor customisations I did on my master wiki

• Tagging $:/plugins/kookma/searchwikis/ui/advancedsearch with $:/tags/SideBarSegment after $:/core/ui/SideBarSegments/search Works well!
• Since I expect my master to have multiple indexes I tagged the below tiddler with $:/tags/SideBar

<$list filter="[tag[$:/tags/SearchWikis]sort[stw_wiki_title]]">
  <$link to=<<currentTiddler>> ><$text text={{!!stw_wiki_title}}/></$link><br>
</$list>

However perhaps a minimal version of "Searchwikis Index Summary" aka $:/plugins/kookma/searchwikis/ui/index-summary with a caption field would be better in the sidebar.

Love your work

Tony

[Mohammad]

On Saturday, March 7, 2020 at 3:27:02 AM UTC+3:30, TonyM wrote:

Mohammad,

You are so productive and prolific making things we all need, thanks so much. 

I have dropped this on my master bobexe wiki and indexed my tiddlywiki syntax and plugin repository of results.

One issue

However I clicked on one which went to this local address;

http://192.168.1.81/IndexWikis/TW5References/index.html##CSS%20Tools

I do not understand this! Can you give an example on tiddliwiki.com? I think TW uses one url#tidTitle to create a permlink so what is ##?

do you mean the tiddler is #CSS Tools? 

If so 

 

But the wiki opens with default tiddlers, not the CSS%20Tools tiddler, could the "##" have something to do with it?

System tiddlers

For my plugin repository I would like my index to include the plugin and other system tiddlers and there fields, Is this possible?

There is a macro called 

Some minor customisations I did on my master wiki

• Tagging $:/plugins/kookma/searchwikis/ui/advancedsearch with $:/tags/SideBarSegment after $:/core/ui/SideBarSegments/search Works well!

Nice! I will add this to tutorial.

 

• Since I expect my master to have multiple indexes I tagged the below tiddler with $:/tags/SideBar

<$list filter="[tag[$:/tags/SearchWikis]sort[stw_wiki_title]]">
  <$link to=<<currentTiddler>> ><$text text={{!!stw_wiki_title}}/></$link><br>
</$list>

However perhaps a minimal version of "Searchwikis Index Summary" aka $:/plugins/kookma/searchwikis/ui/index-summary with a caption field would be better in the sidebar.

You may note that, in search tool, you see a link to all indexes and by clicking you can open summary where you can enable/disable indexes, change the sort order, ...

So, this is another taste and it is up to you!

[Mohammad]

On Saturday, March 7, 2020 at 3:27:02 AM UTC+3:30, TonyM wrote:

Mohammad,

You are so productive and prolific making things we all need, thanks so much. 

Thank you for your feedback and support. 

I have dropped this on my master bobexe wiki and indexed my tiddlywiki syntax and plugin repository of results.

One issue

However I clicked on one which went to this local address;

http://192.168.1.81/IndexWikis/TW5References/index.html##CSS%20Tools

But the wiki opens with default tiddlers, not the CSS%20Tools tiddler, could the "##" have something to do with it?

System tiddlers

For my plugin repository I would like my index to include the plugin and other system tiddlers and there fields, Is this possible?

In Index and Export you have a exclude field, see the tricks in tutorial (https://kookma.github.io/TW-Searchwikis/#demo%2Fcreate-an-index)

The exclude can be used in reveres form e.g instead of -[tag[filters]] +[tag[filters]] OR [all[tiddlers+shadows]prefix[$:/PSAT/tony_stuff]]

The indexing filter is:  <$list filter="[all[tiddlers]!is[system]] $(exclude)$">

So you can modify as you like or use the Exclude field.

[Mohammad]

Announcement: Searchwikis Plugin

Date: Mar 7th, 2020

Release: 1.3.0

Status: Stable

History

Revision 1.3.0

• Date: [5th Mar 2020]
• [FIXED] issue in generate links for tiddlers with hash (#) in their title
• [FIXED] documentation to add Searchwikis as sidebar segment

For all changes see History

Searchwikis Plugin

7th March 2020 at 10:07am

start

Tiddlywiki5.1.21+

 

LicenseMIT

 

Release1.3.0

 

Statusstatble

What is Searchwikis plugin?

Searchwikis plugin uses index (dataTiddler) to search multiple external Tiddlywikis. It has two parts

1. An indexer, to build an index of all tiddlers in an external wiki
2. A search tool to search indexes and display a link to a tiddler found in an external wiki

Then one master wiki can hosts many index tiddlers and lets to search several external wikis through index tiddlers.

How to use

• install $:/plugins/kookma/searchwikis into the host wiki
• drag and drop index and export tiddler to the external wiki
• use the ui and create the index tiddler of external wiki
• export it to the host wiki, or drag and drop to the host wiki
• open $:/AdvancedSearch in the host wiki
• goto External wiki tab and make search
• results will be shown as links and one can click a link and open the external wiki with that result under focus

Install

Packaged plugin

1. Drag and drop $:/plugins/kookma/searchwikis into your wiki
2. Save and reload your wiki

Client-Server installation (separate files)

• For Node.js (server version) download the searchwikis folder from source folder and paste in your Tiddlywiki plugins folder
• For local installation put the searchwikis folder into your local plugins folder under your wiki folder

Code and demo

• Demo: https://kookma.github.io/TW-Searchwikis/
• Code: https://github.com/kookma/TW-Searchwikis

For complete tutorial, examples and use cases, see the demo page.

[Mohammad]

New update of Searhcwikis

Those uses the Plugin Library below

No need to do anything just open the Get More Plugin and reinstall the Searchwikis

--Mohammad

[TonyM]

Mohammad,

The "##" is no longer occurring. Thank you

The "exclude filter" of [is[system]prefix[$:/plugins]] works nicely to include the installed plugins, but perhaps I need to go further to index the plugins shadow tiddlers?

I am sorry I did not review and feedback earlier in the history of this solution. Once again "love your work".

I also learned something by accident, I had not provided the wiki url correctly to the index and Export tool, In some ways this was because of my "special" setup.

• http://192.168.1.81/IndexWikis/TW5References/index.html finaly worked

But the thing I learned its trivial to change the stw_wiki_url: field in the "master" wiki where I loaded the index. This highlighted the ease of changing this as needed to point not to a local wiki but one with a different address on the internet, one which may be a copy of the local wiki.

I do not want to ask for new features because this is already a 100% solution but I had some ideas you could consider in the future. I can do some of it it myself and share if you want.

• Include in the Index and Export tiddler a button that appears in Page Control buttons that on clicking exports the local index
  This would allow a Wiki to freely publish an index for import elsewhere if desired (eg tiddlywiki.com)
  
• An option to the above for non static websites could also make the button first generate a fresh index then export the file
  This allows a wiki owners who makes changes worthy of a fresh reindex to click and re-export whenever needed
• It may often be convenient to drag and drop the index from the source wiki rather than go through the export import process
  I wonder if a functional button such as described by me above could also be made draggable to drop the index tiddler on the master wiki?

Be it an option or standard behaviour the truth is I would be keen to see the possibility of search results listed together from all indexes

• However this may make indicating the source difficult, perhaps the index file could have an icon which would prefix each result
• Another alternative may be being able to close the search results for a given indexed wiki, so for example I may only review the results for my plugin wiki etc...

I always dreamed for a solution like this, and it has become a reality in your hands. The reason is I have a multitude of wikis, some with single purposes reflected in the name, some with multiple purposes now captured by the indexing and search you have developed. Hopefully I will no longer loose stuff.

I believe Bob may have a way to search across its wikis, perhaps Jed may be able to provide an add-in to your tool so the search can extend not only to wikis which have used the index and export but those in the bob "network" automatically. This would make bob a premier location for one to "index all ones tiddlywiki resources" including those of other publishes eg tiddlywiki.com

In closing I have developed a process for "Registering" wikis in a central database, and some helpful ways of keeping them uptodate, given the structure of this search solution they would integrate quite well with or without the search index so I will provide another reply in relation to this, all of which could be combined with your solution in the future. Perhaps a new thread is advisable so keep an eye out for it.

Regards

Tony

[Mohammad]

Hi Tony,

 Many thanks for your feedback.

On Sunday, March 8, 2020 at 2:03:18 AM UTC+3:30, TonyM wrote:

Mohammad,

The "##" is no longer occurring. Thank you

The "exclude filter" of [is[system]prefix[$:/plugins]] works nicely to include the installed plugins, but perhaps I need to go further to index the plugins shadow tiddlers?

I am sorry I did not review and feedback earlier in the history of this solution. Once again "love your work".

I also learned something by accident, I had not provided the wiki url correctly to the index and Export tool, In some ways this was because of my "special" setup.

Good! The url is essential as it is used to find the external wiki. 

• http://192.168.1.81/IndexWikis/TW5References/index.html finaly worked

But the thing I learned its trivial to change the stw_wiki_url: field in the "master" wiki where I loaded the index. This highlighted the ease of changing this as needed to point not to a local wiki but one with a different address on the internet, one which may be a copy of the local wiki.

I do not want to ask for new features because this is already a 100% solution but I had some ideas you could consider in the future. I can do some of it it myself and share if you want.

I bookmark this post and hopefully I will try to implement those I can!

 

• Include in the Index and Export tiddler a button that appears in Page Control buttons that on clicking exports the local index
  This would allow a Wiki to freely publish an index for import elsewhere if desired (eg tiddlywiki.com)
  
• An option to the above for non static websites could also make the button first generate a fresh index then export the file
  This allows a wiki owners who makes changes worthy of a fresh reindex to click and re-export whenever needed
• It may often be convenient to drag and drop the index from the source wiki rather than go through the export import process
  I wonder if a functional button such as described by me above could also be made draggable to drop the index tiddler on the master wiki?

This has been implemented already! When you create an index, below the input boxes in Index and Export you see the list of all index ttiddlers with date of creation and delete button. You can drag the link to your wiki.

 

Be it an option or standard behaviour the truth is I would be keen to see the possibility of search results listed together from all indexes

• However this may make indicating the source difficult, perhaps the index file could have an icon which would prefix each result
• Another alternative may be being able to close the search results for a given indexed wiki, so for example I may only review the results for my plugin wiki etc...

This is a bit complex! I myself interested to show to most relevant results first from all indexes, but then I think the standard Tiddlywiki search tool cannot handle this and we need a new search tool.

Also like Google, if master wiki learns from our search, then it can show those we select frequently first!

But these future dreams.

 

I always dreamed for a solution like this, and it has become a reality in your hands. The reason is I have a multitude of wikis, some with single purposes reflected in the name, some with multiple purposes now captured by the indexing and search you have developed. Hopefully I will no longer loose stuff.

Yes, for me also Searchwikis helped to collect all my stuff in one place :-) 

I believe Bob may have a way to search across its wikis, perhaps Jed may be able to provide an add-in to your tool so the search can extend not only to wikis which have used the index and export but those in the bob "network" automatically. This would make bob a premier location for one to "index all ones tiddlywiki resources" including those of other publishes eg tiddlywiki.com

I do not use Bob yet effectively in my own work, but Mark in another thread explained Bob is very powerful in handling multiple wiki searches. Hope Jed explain this.

 

In closing I have developed a process for "Registering" wikis in a central database, and some helpful ways of keeping them uptodate, given the structure of this search solution they would integrate quite well with or without the search index so I will provide another reply in relation to this, all of which could be combined with your solution in the future. Perhaps a new thread is advisable so keep an eye out for it.

In my opinion, one big item in the wish list is a way the update the indexes when the external wikis have changed! Right now, you have to open the external wikis time by time and reindex.

This is a little tedious! So, updating the indexes is very important and is manual right now. By the way for personal wikis I have less problem.

I use also https://fraidyc.at/s/ to track wikis are not mine like tiddlywiki.com and when I see enough changes I start a new index and overwrite the old one.

--Mohammad