TW Community Search

[Erwan]

Hi everyone,

Here is my new "TW community search":
https://rawgit.com/erwanm/tw-aggregator/master/tw-community-search.html
(it's not actually a search engine, rather an index; just thought
"search" was clearer).

The offline process downloads a predefined list of publicly available
wikis, then merges their content as system tiddlers into the new wiki.
The rest is customized TW search.

* The user cannot see the tiddlers directly on the search wiki: every
link (in the search results or by clicking on a tag pill) is converted
to an external link to the original tiddler (this solves the main issue
which appeared in the first version, where some content was broken due
to incompatible plugins/system tiddlers).

* Currently all non-system tiddlers are scraped from the indexed wikis,
but we can consider a more subtle system with special tags, fields or
whitelist/blacklist (see https://github.com/erwanm/tw-aggregator/issues/6)

The current list of indexed wikis can be found at
https://rawgit.com/erwanm/tw-aggregator/master/tw-community-search.html#CommunityWikis.
If you are the author of one of these, please tell me if you agree to
this use of your wiki: unless there is an explicit licence in you wiki
(and I didn't find any in the current list of wikis), I need your
explicit agreement for this reuse of your content. So please reply,
otherwise I will have to remove your wiki from the list... and obviously
the shorter the list the less useful the search will be. (I will also
send individual emails to the authors in case they don't check the TW
mailing list regularly).

Also if you are the author of a wiki which is not in the list and you
want it to be referenced, please tell me.


Mario had raised a few issues for the first version in
https://groups.google.com/d/msg/Tiddlywiki/qGk3aH731Pc/fZ0RvvsEyqAJ: the
first one is solved, and I hope to solve the second one about the
copyright issue soon. However the third one about google referencing I'm
not sure: I think (?) that the "link farm" problem is solved by the fact
that now links are built dynamically by some widget/macro code: there is
only one permanent link to every referenced wiki (i.e. not in every
tiddler anymore). However the duplication of the content might still be
an issue? If anybody knows how this works and what I can do about it,
please tell me!


Regards
Erwan

[Scott Kingery]

I'm the author of TW5 Tribal Knowledge (https://dl.dropboxusercontent.com/u/118970/wiki/tw5tribalknowledge.html) you may use for this purpose.

Scott

[Ton Gerner]

Hi Erwan,

That's fine with me (http://tongerner.tiddlyspot.com/)

Cheers,

Ton

[Felix Küppers]

Hi Erwan,

nice idea, no problem.

Maybe a suggestion. When crawling the wikis, check for the version number as permalinks where introduced with the release of 5.0.11. So unfortunately stephan's http://tw5magick.tiddlyspot.com/#Geography%2006 doesn't navigate to the tiddler.

-Felix

[Jed Carty]

You can use any of my stuff.
I like the update.

[Tobias Beer]

Fine with me and any of my stuff, e.g. tb5, or whatever you find listed here...

http://tb5.tiddlyspot.com/#Tobias%20Beer

...and worthy being indexed.

Best wishes, Tobias.

[David Gifford]

Hi Erwan,

I don't mind anything I make to be used in any way by the TiddlyWiki community. But since much of my information in TW5Mall (and TiddlyQuickly) is surely outdated, perhaps it would be better if you guys grabbed the still useful stuff from it and posted it on your own sites, rather than posting links to TW5Mall and TiddlyQuickly.

Dave

[andrew.j.harrison84]

You are welcome to use or include in your search anything you find on my site t5a.tiddlyspot.com. I am always working on it. Any suggestions welcome.

Sent from my Samsung Epic™ 4G Touch

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

[AcD]

SeptemberTW (http://SeptemberTW.tiddlyspot.com) is meant for public consumption, with no restrictions on intended readership, hence indexing by your project is most welcome.

Cheers,

AcD.

On Tuesday, December 23, 2014 11:08:32 PM UTC+1, Erwan wrote:

[Roma Hicks]

Hello Erwan,

You may index and use my content on the GSD5 project.  Thanks.

On Tuesday, December 23, 2014 4:08:32 PM UTC-6, Erwan wrote:

[Alberto Molina]

Hi Erwan,

No problem for me.

Alberto

[James Anderson]

no problem from me either.

James

[Tobias Beer]

Hi Erwan,

I added a tiddler with my preferences for indexing.

$:/follow @ tb5

All entries are sorted chronologically,

so you can realize if there are any updates.

If you think a different format may be better, let me know.

Best wishes, Tobias.

[Erwan]

Hi Tobias,

I like your approach, I'm very interested in taking into account the authors indications in an automated way like that. I created a ticket here: https://github.com/erwanm/tw-aggregator/issues/23. It might take a little time before it is implemented though, I'm quite busy these days.


You are probably the only one who has so many wikis so I don't know if the approach is generalizable, but I had vaguely started thinking about doing something similar at the level of tidders: https://github.com/erwanm/tw-aggregator/issues/6


Regards
Erwan

[Mat]

@Erwan

You are probably the only one who has so many wikis so I don't know if the approach is generalizable, but I had vaguely started thinking about doing something similar at the level of tidders: https://github.com/erwanm/tw-aggregator/issues/6

In addition to the two ideas you mention there, here's a third. I hope it's not pie in the sky, in not a programmer, so FIWW:

Note that permalinks can contain filters in TW. This could perhaps be used for some kind of "matching mechanism" so that in the call for the source TW you not only specify which tiddlers you would like to index but also send some kind of identifier for who you are. The source TW can read this and use the identifier as an allow/dismiss criteria.

In another context, (iframes), here's a description of using hash based 'additions' to urls for communication between domains.


Another idea: One of your ideas in your issue brings up haveing whitelists/blacklists in a tiddler. Maybe it would be desirable to allow individual tiddlers to specify this. I.e having the individual tiddler accept or reject when someone attempts to include it.
Perhaps fields
"whilelist: exclusively if domain as tiddler foo"
"blacklist: username=bar OR domain=frotz.com"

This latter idea would be particularly useful in collaboration situations where you want to give different persmissions to different people.


Do I make sense at all?


<:-)

<:-)

[Mat]

Browsing on in your github it seems at least that last idea is already brought up or at least something similar. Sorry for 'repeating' or whatever it would be classified as :-)

<:-)

[Erwan]

Hi Mat,

Your suggestions are certainly interesting, but far beyond my pay grade I'm afraid! For example, the iframe trick with JavaScript is beyond my level of expertise, to give you an idea.

I can see how my "community search wiki" can give the illusion of some form of dynamic sharing of tiddlers between different wikis, but it's actually not the case I'm afraid: behind it there is a program (not a TW program, an independent script) which applies a very crude treatment to the wikis; it is just a standard text file manipulation process, using the TW node.js version precisely because tiddlers are simple text files in this version.

This means that it is not dynamic, because there is an intermediate "worker" which does all the menial work between the source wikis and the aggregator wiki. I don't know whether the kind of dynamic sharing that you envision is technically possible with TW, Jeremy or the other developers are certainly more in a position to answer than I am.

About the blacklist/whitelist marking at the level of tiddlers: it was also my first idea, but I think doing it as a a list in a special tiddler gives more freedom to the author: this way instead of having to mark each tiddler with a specific tag or field, they can use their own filter; for instance if you already have some categories IdeasForTW, TipsForTW, MyGroceryList and MySecretDiary, you can do a whitelist filter "[tag[IdeasForTW]] [tag[IdeasForTW]]" or a blacklist filter "[tag[MyGroceryList]] [tag[MySecretDiary]]".

Regards
Erwan

[Danielo Rodríguez]

Hello Erwan,

It's fine for me if you want to index braintest.tyddlyspot.com, You can also index http://mathmacros.tiddlyspot.com/ and http://contextplugin.tiddlyspot.com/

Let me know if you implement some mechanism to configure how you tool index tiddlers.

Regards.

[Astrid Elocson]

Hi Erwan,

Feel free to index http://ae-railroad.tiddlyspot.com/ if you wish, although there may be little point, as that plugin will be directly documented on tiddlywiki.com in a few days' time when 5.1.8 is released.

The other tiddlyspots I've published are temporary and not worth indexing.

– æ

[Tobias Beer]

Let me know if you implement some mechanism to configure how you tool index tiddlers.

see: https://groups.google.com/d/msg/tiddlywiki/bR9ZozLNadw/lxR8KDZ2l58J

or:  https://github.com/erwanm/tw-aggregator/issues/23 — follow url

Best wishes, Tobias.

[Erwan]

Hi Danielo,

you can also see this: https://rawgit.com/erwanm/tw-aggregator/master/tw-community-search.html#WhitelistFeature
does this answer your question?

Regards
Erwan

[kixam]

Hi Erwan,

OK to index my contributions into your wiki:

http://kixam.github.io/TW5-visjsTimeline/
http://kixam.github.io/TW5-datePicker/

and moment.js plugin, which is a dependency of the above, and a compiled version of which is available from either of the above demo wikis

kixam