[TW5] Community Search

[Erwan]

Hi everyone,

The Community Search wiki keeps growing, slowly but surely, with now
3383 indexed tiddlers from 36 different wikis (half of them thanks to
Tobias!):
https://rawgit.com/erwanm/tw-aggregator/master/tw-community-search.html

It would be nice to make it cover a larger set of wikis, as this is what
a search engine is for, obviously! So if you are a wiki author and are
fine with your content being indexed, please let me know! I can't do it
without your agreement.

Recently I added some documentation, and made some improvements intended
to give wiki authors more control over what is indexed from their wiki.
So if you have concerns about that, please have a look at
https://rawgit.com/erwanm/tw-aggregator/master/tw-community-search.html#AuthorDocumentation
And of course don't hesitate to tell me about any bug you find or
feature you would like.

Regards,
Erwan

[Tobias Beer]

Thanks for all the work, Erwan.

I certainly appreciate that you've come to automate that indexing mechanism via the fields

• follow=yes
• url=http://foo.bar.com

at reference tiddlers such as those listed here...

• Tobias Beer @ tb5
• or here respectively: $:/follow @ tb5

Best wishes, Tobias.

[Felix Küppers]

Great job. Really useful.

[Danielo Rodríguez]

I have to agree with the congratulations.

The new features makes much more powerful this awesome tool.

Thank you.

[Erwan]

Happy that you like it, thank you!

btw these new features (especially the whitelist one) haven't really been thoroughly tested, so if you use them please check that it works as expected for your wiki. and of course let me know in case it doesn't!

Erwan

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

[Jed Carty]

Adding a new wiki worked, but it only used the first word of the title with spaces.

[Felix Küppers]

Hi Erwan,

would it be possible to introduce a tag like e.g. "#news" or "#release notes" that can be used by wiki-owners to force some tiddlers to be listed chronologically as a separate category in your search wiki? Mat already suggested a similar thing but then it would be possible for plugin developers to tag release note tiddlers with #news and people can see what is happening - this is actually pretty close to a RSS-feed.

Edit I see that you have a tiddler "CommunityTags". Maybe you could make a tiddler for every tag ("Tag - <Tagname>") because then I could create a permalink to this tiddler and stay informed, otherwise a big huge taglist opens with many tags I am not interested in. Also maybe we can as a community decide on special tags (like e.g. #news) that have a special meaning.

-Felix

[Erwan]

Hi Felix,

Yes, I'm thinking about something like that as well. Originally I was going to simply use the date of the tiddler to automatically generate a "news event".

I think the difficulty lies mostly in how this features fits with different cases:

• automatic "news" when a plugin is updated based on the date
• some authors might not want to use it although their plugins are useful, so it's important that users don't assume that this kind of news will show up for all plugins
• people could forget to remove their "news" tag when it's not news anymore, so the system problably has to do some cleanup based on the date
• possibly storing the news to keep track of the history?

Also it's probably better to propose a special tiddler where the author can write their own filter instead of a predefined tag, since it's more generic and gives more freedom to the author.

I'm trying to clarify for myself how to make all this work nicely together, anybody is welcome to give their suggestions!

Erwan

On 06/02/15 08:17, Felix Küppers wrote:

Hi Erwan,

would it be possible to introduce a tag like e.g. "#news" or "#release notes" that can be used by wiki-owners to force some tiddlers to be listed chronologically as a separate category in your search wiki? Mat already suggested a similar thing but then it would be possible for plugin developers to tag release note tiddlers with #news and people can see what is happening - this is actually pretty close to a RSS-feed.

-Felix

[Tobias Beer]

Also it's probably better to propose a special tiddler where the author can write their own filter instead of a predefined tag, since it's more generic and gives more freedom to the author.

I'm trying to clarify for myself how to make all this work nicely together, anybody is welcome to give their suggestions!

Not sure if that works, but if I can specify a filter that works for me and works for you (+ the site constraint), then that would be it, I believe... e.g. "news is everything tagged foo not older than x".

Best wishes, Tobias. 

[Erwan]

Hi Jed
This is fixed now.

Erwan

On 06/02/15 05:41, Jed Carty wrote:

Adding a new wiki worked, but it only used the first word of the title with spaces.

[Tobias Beer]

Hi Erwan,

I'm curious to see how your aggregator will handle all the redirects and renaming I'm pushing though right now. ^^

FWIW, most importantly, please change tb5 to this location...

http://tobibeer.github.io/tb5

(How) do you handle deletions / changed urls?

Best wishes, Tobias. 

[Jeremy Ruston]

Hi Erwan

Well done for your work on this, it's good to see this sort of experimentation.

Something that's been cropping up over in GitHub discussions is that it might be useful to be able to search across all community wikis to see if certain constructions are being used heavily in the wild. For example, we've realised that the `[is[missing]]` filter has the wrong semantics, which we'd like to fix but it would be useful to have some idea if it is widely used it in its present form.

So I guess I'm asking for the ability to search system and shadow tiddlers, and perhaps to broaden the search to all publicly accessible TW5 wikis. If only there was a way to get Google/DuckDuckGo to return a list all TW5 documents that it has crawled...

Best wishes

Jeremy.

On Sat, Feb 7, 2015 at 1:04 PM, Tobias Beer <beert...@gmail.com> wrote:

Hi Erwan,

Will be curious how your aggregator will handle all the redirects and renaming I'm pushing though right now. ^^

Best wishes, Tobias. 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

--

Jeremy Ruston
mailto:jeremy...@gmail.com

[Erwan]

Hi Jeremy,

The system/shadow tiddlers were excluded because originally I wasn't renaming the tiddlers, thus causing unwanted interactions on the search wiki. I guess that with the renaming (simple prefix with $:/<wiki id>/) there is little risk of interaction, but I'm not sure to know about all the possible kinds of "interactions" in TW. Anyway I can do some tests with them, but if you or any other TW expert could tell me which aspects to be careful about when merging all the tiddlers, that could be helpful. I know at least that I can't include plugin tiddlers, and currently I also exclude theme and javascript tiddlers to be safe (?).

Indexing all public wikis seems more problematic to me, as some authors might not want that: additionally to my emails on the TW list I have sent an individual email (and later a reminder) to every known author in my list, so at this point I have to assume that those who did not answer are not interested. Since my system requires storing their tiddlers in the search wiki, it wouldn't be appropriate to do it without their consent. It would be different if the content was not made public (even hidden as system tiddlers), but I don't see how to do differently since it is the TW search system which does all the work.

About the particular case of is[missing], I did a manual grep search on my local machine among the 29 wikis in my list (tagged CommunityWikis, including the non-indexed wikis; 5715 tiddlers in total) and found 25 occurrences (more precisely lines containing at least one occurrence); see the detailed results attached. By comparison I found:
- 135 is[current]
- 19 is[image]
- 5 is[orphan]
- 67 is[shadow]
- 389 is[system]
- 44 is[tiddler]
- 7 is[tag]

I was even imagining that this could be automatized, actually: some kind of asynchronous search triggered by some particular tag/tiddler in an indexed wiki, for which you would see the results the day after in a special tiddler of the search wiki... But that seems to me like a terribly complicated system for a very specific use case! (and that's not even addressing the potentially excessive workload on my machine)

Regards
Erwan

[Erwan]

Hi Tobias,

To be honest I wouldn't be surprised at all if you find bugs ;)
Let me know what happens.

Regards
Erwan

[Tobias Beer]

To be honest I wouldn't be surprised at all if you find bugs ;)
Let me know what happens.

It works pretty well, actually showing me my own bugs and dead links ;-)

By the way, perhaps you can also create a simple redirect

such as this to make for a shorter url:

http://tobibeer.github.io/tw/search/

the bit of code...

https://github.com/tobibeer/tobibeer.github.io/blob/master/tw/search/index.html

it's really easy to get github pages up and running...

https://pages.github.com

Best wishes, Tobias.

[Jeremy Ruston]

Hi Erwan

That's very helpful, thank you.

It looks as though most of the hits are actually Tobias's experiments with reworking is[missing].

> The system/shadow tiddlers were excluded because originally I wasn't renaming the tiddlers, thus causing unwanted interactions on the search wiki. I guess that with the renaming (simple prefix with $:/<wiki id>/) there is little risk of interaction, but I'm not sure to know about all the possible kinds of "interactions" in TW. Anyway I can do some tests with them, but if you or any other TW expert could tell me which aspects to be careful about when merging all the tiddlers, that could be helpful. I know at least that I can't include plugin tiddlers, and currently I also exclude theme and javascript tiddlers to be safe (?).

I've been dealing with the same problem for work I've been doing on the plugin library. The code I've got looks like this:

$tw.wiki.addTiddler(new $tw.Tiddler($tw.wiki.getCreationFields(),tiddler,{

title: event.data.cookies.infoTitlePrefix + event.data.cookies.url + "/" + tiddler.title,

"original-title": tiddler.title,

text: "",

type: "text/vnd.tiddlywiki",

"original-type": tiddler.type,

"plugin-type": undefined,

"original-plugin-type": tiddler["plugin-type"],

"module-type": undefined,

"original-module-type": tiddler["module-type"],

tags: ["$:/tags/RemoteAssetInfo"],

"original-tags": $tw.utils.stringifyList(tiddler.tags || []),

"server-url": event.data.cookies.url

},$tw.wiki.getModificationFields()));

As you can see, it's mangling the incoming title by applying a prefix (which I believe you already do), and also neutralising the plugin-type and module-type fields, and moving the tags to another field. The last is probably overkill - the key thing is to remove $:/tags/* tags.

Best wishes

Jeremy