ExtensionsExplorerPlugin: introducing collections (alpha)

[Yakov]

Hello everyone,

today I'd like to introduce a feature which is still under development, but which, I believe, will help build really nice flows of extensions publication, exploring, and updating.

As you probably have noticed, up to certain point ExtensionsExplorerPlugin worked nicely for detecting plugin updates and updating plugins, but didn't really much for exploration: although there's some extensions inside the default AvailableExtensions (which one can install via sidebar), that list is either

• the default one, so is only updated with EEP itself (slow, too centralized);
• or a custom one, which can even have a custom Source to update from, and which is fine for flows like "as a developer, I create a custom AvailableExtension, and my fellow users get updates from me"; but which is still centralized and bound to one developer only.
  

What are the collections and what they help with?

The idea is simple: allow additional tiddlers with syntax like AvailableExtension to be recognized by EEP (version 0.6.0 and above) when they are tagged with systemExtensionsCollection. Here's an example: my personal extensions collection, which now only has one plugin, but which I'll be populating with my other extensions. If you "install it", you'll be a) shown extensions from it in backstage available to install and b) notified on collection update, so you'll know when new plugins are released.

I invite extensions authors to give it a try.

Next step, the collections themselves should be explorable, so I've created a "central" collection inside EEP repo which already has my collection listed and I'm glad if you create a PR suggesting to add your collection.

Soon, I'll add autoinstalling of the central collection in EEP, so that other collections will be visible and installable without any manual steps.

I'm now figuring the way to nicely list extensions created by "retired" authors, especially given that many of them don't have "correct" url in the Source slice (example) and also that new "recommended" forks may be created in the future. Also, I'm going to support "TiddlyWiki as a collection" functionality for repositories like TiddlyTools.

In the end, this will allow to list all the available extensions in one place (which will require search/filters-like functionality, though), but also let authors provide their work directly, without need of any PRs (once their collection is added).

I'll be also glad to hear feedback from early adopters, both UI and status messages still require much polishing, and it's important to know what's most confusing or which features are missing. Like always, I have lots of ideas, but limited resources.

Let me know what you think!

Best regards,

Yakov.

[Yakov]

PS there's a number of scenarios which I haven't fully explored yet, like "install all extensions from this collection", or packing all translations into a collection, or collections that can be used for grouping existing plugins by functionality (like custom search plugins), I'm open for discussing similar applications, too

вторник, 4 июля 2023 г. в 15:20:50 UTC+3, Yakov:

[Reto]

Sounds very interesting, Yakov! I will look into this in more depth in the next days.

Cheers, Reto

[Reto]

It looks as if collections could be a way to publish not only plugins but also other tiddlers which are part of an "application". Like what I have (called ExtendedBrain): some custom code in plugins as well as for instance lists (based on ForEachTiddlerPlgin) ot tiddlers with text only which are integral part of the "application". Also those tiddlers can be further developed (or fixed) and new version gets published.

As it looks, whatever the URL points to it is installed as tiddler. At the moment I did not understand, how tags like "systemConfig" are applied. This happens when the source file is like *.js? And *.txt gets no tag?

[Yakov]

It looks as if collections could be a way to publish not only plugins but also other tiddlers which are part of an "application". Like what I have (called ExtendedBrain): some custom code in plugins as well as for instance lists (based on ForEachTiddlerPlgin) of tiddlers with text only which are integral part of the "application". Also those tiddlers can be further developed (or fixed) and new version gets published.

Indeed! That's one of the applications in mind, although there's only basic functionality implemented, and things like "install all tiddlers from collection" should be added.

 

As it looks, whatever the URL points to it is installed as tiddler. At the moment I did not understand, how tags like "systemConfig" are applied. This happens when the source file is like *.js? And *.txt gets no tag?

Well, EEP knows some "extension types", currently they are "plugin", "collection", and everything else. The type can be specified in a collection ("type": "plugin"), otherwise EEP tries to guess it (currently: a plugin either: is from TW and has systemConfig tag, has Type slice of plugin, or the title ends with "Plugin"). Based on type, EEP does additional stuff on installing. So file extension doesn't really matter for now.

[Reto]

Sounds promising.

I am tempted to make a prototype ... just to learn this git(hub) stuff ... I guess there is no "market" for my application. Or more precisely: it is very difficult to find the people who could benefit.

[Reto]

By the way: there is a spelling mistake (in the mouse-over text):

[Yakov]

Thanks, I've fixed the mistake.

> I am tempted to make a prototype ... just to learn this git(hub) stuff ... I guess there is no "market" for my application. Or more precisely: it is very difficult to find the people who could benefit

Regarding the "market", I think the opposite way: while TWC itself can't currently compete for new users with tools like Notion, TW5, etc, by itself (still enough serious UI/UX/presentation problems), solutions based on it may attract new users if such solutions provide enough value. For instance, I'm now using TW for learning languages (storing vocabular, grammar, generating quizes etc) and I'm considering releasing the quiz functionality, may be as a Tiddlyhost template. This way, one could easily try it for their purpose and use it if it's good enough. But I'm tempted to work on improving Tiddlyhost's functionality (saving TWC) and UI (responsive) first, as this is a kind of application that one will need on mobile, too (including myself). Anyway, I think that this model (show a solution to friends and see what they say) can be suitable for you, but if you're afraid to invest too much time in something that can attract too little attention/bring value to too few people, may be you could try to show it to a few people and check what they say first (keeping in mind that people closer to you are tempted to encourage even if they don't see any value in the product for themselves).

As for learn git(hub) stuff, may be we can start a new thread for this, and I'll guide you step-by-step. I think this will work easier than me trying to sit and write a whole tutorial (which can be made after our actual journey to build something real). Just start a thread, ask some question like "I want to do this, where to begin?" and a title which clarifies that the thread is about using git/github for sharing plugins/contributing/whatever.

среда, 12 июля 2023 г. в 08:30:19 UTC+3, Reto:

[Reto]

Anyway, I think that this model (show a solution to friends and see what they say) can be suitable for you, but if you're afraid to invest too much time in something that can attract too little attention/bring value to too few people, may be you could try to show it to a few people and check what they say first (keeping in mind that people closer to you are tempted to encourage even if they don't see any value in the product for themselves).

Main blocker for me to share what I have is that there was no bullet proof way to update instances of other people. It seems that with collections and EEP this exists for the first time. And yes I am also a little afraid of the effort that comes with it: answer user questions, fixing problems, proper docs, translations and so on.

[Yakov]

Sure, I see those potential concerns, it's up to you to decide; I'd say that sharing and helping people is fun, so if you have some spare time that you're ok with to spend on support (you can always limit that time and just not respond immediately) and you want to try, then it's worth trying. Also, be prepared to the fact that most of "extended brain" software is opinionated, and finding those who share your opinions may be not that quick. Sometimes, showing what you do to your friends is a good start, sometimes it's more fruitful to present it in communities that may find it helpful. Giving is an art :)

вторник, 18 июля 2023 г. в 12:21:35 UTC+3, Reto:

[sam paul]

Hi, Yakov,

This plugin is so nice. It would be even nicer if it could export a plugin list .txt file from installed plugins of a wiki. I have hundreds of plugins installed.

I don't think TWC can't compete with other tools. TWC has far more plugins than TW5. And if loaded with suitable plugins, it is a static version of WordPress for those who favor retro web design. However, most of them are in the internet archive. I hope this plugin can make every wiki into a plugin repo server. This will make finding and installing plugins much simpler.

Best.

[Yakov]

Hi Sam,

> It would be even nicer if it could export a plugin list .txt file from installed plugins of a wiki.

sounds as an interesting approach, is it for sharing your setup with others, or for something else?

> I hope this plugin can make every wiki into a plugin repo server. This will make finding and installing plugins much simpler.

Yeah, the indexing process is not yet established, and also the more extensions are indexed, the more important it is to enhance the plugin with search and other means to navigate between them. 🚢 0.7.0 🚢 I've made one step further and added tabs to "Explore extensions" section of backstage, separating the installed ones, the available ones (with some not yet indexed repositories and more), and added a tab about contributing. Several more updates are needed, but at least now we can add existing repositories one-by-one.

Best regards,

Yakov.

суббота, 23 ноября 2024 г. в 04:45:36 UTC+3, samp...@gmail.com:

[sam paul]

Hi Yakov.

Thanks.

I have a few suggestions. Could you turn off the self-update notification? It should be pretty common to have outdated TWC, and that notification would be quite annoying.

Best.

[Yakov]

Hi Sam, could you clarify what you mean by self-update notification? Do you want to avoid notifications about EEP updates, but see those for other plugins? Or are you talking about the notification about new TWC (core) version available? In the latter case, this has nothing to do with EEP as that notification comes from SimplifiedUpgradingPlugin and is controlled by the chkAutocheckUpgradeOnStart option (so you can set chkAutocheckUpgradeOnStart: false in SystemSettings if you like).

суббота, 28 декабря 2024 г. в 18:47:12 UTC+3, samp...@gmail.com:

[sam paul]

Dear Yakov,

Yes, I think it is better a add an option to disable the notification. Anyway, it's so important.

Another question I want to ask is, can I put the extension list txt inside a tiddlywiki classic? For example, the url will be somewikiurl#extensionlist.

Best,

Sam

[sam paul]

Dear Yakov,

I came up with a method to quickly index currently available plugins. Using http://tiddlywiki.abego-software.de/#ForEachTiddlerPlugin, the following macro index all the installed plugins in your tiddlers:(replace https://xyz.github.io by the URL of your wiki.)

[
<<forEachTiddler
where
'tiddler.tags.contains("systemConfig")'
sortBy
'tiddler.title'
write
'"{"
+"\n"
+"\"url\": \"https://xyz.github.io#"
+tiddler.title
+"\","
+"\n"
+"\"description\": \""
+tiddler.title
+"\""
+"\n"
+"},"
+"\n"'
>>]

Then, you can remove the last "," and use tools like https://jsonlint.com/ to convert it to json format. For example, if you perform this action on tiddlytools's html file, you can index all the plugins there. I can't find a method to add whitespace or tab using forEachTiddler macro.

It seems that the ExtensionsExplorerPlugin is unable to load plugin from tiddlywiki file, for example, https://wangyenshu.github.io#ModelViewerPlugin. This is the index of all plugins I installed obtained from the method above: https://github.com/wangyenshu/TiddlywikiClassic_YS_ExtensionsCollection/blob/main/YS_Installed_Extensions.txt . When I load it using ExtensionsExplorerPlugin, it doesn't fetch the plugin name. When I install one plugin, it says:

loaded undefined, about to import and install...

Successfully installed undefined (reload is not necessary)

Imported undefined v1.1.2

I think the format of YS_Installed_Extensions.txt. Could you point out where the problem is? I have attached a sample tiddlywiki file that adds the source https://github.com/wangyenshu/TiddlywikiClassic_YS_ExtensionsCollection/blob/main/YS_Installed_Extensions.txt and the screenshot.

Best,

Yanshu(sam)

[sam paul]

Loading extensions from github blob works correctly. I created a TiddlerInFile fork that allows exporting all tiddlers of a certain tag. That greatly simplifies the work. https://wangyenshu.github.io/#TiddlerInFilePlugin-ByTagFork

But one issue I met when applying the above method is that I can't save old wiki (like abego-extension, tiddlytools) in the modern browser. There is no upgrade option in the backstage as well.

I think it is good to have a centralized TiddlyWiki classic plugins registry.

Best,

Yanshu

[sam paul]

I finished indexing all the plugins (systemConfig tag) from the system server:

• AbegoSoftwareServer
• BidiXTWServer
• BobsPluginsServer
• MartinsPluginsServer
• MonkeyPirateTWServer
• PeachTWServer
• PrinceTiddlyWikiExtensionsServer
• RedMountainVistaServer
• TiddlyStylesServer
• TiddlyToolsServer
• VisualTWServer

Here is the collection txt file: https://github.com/wangyenshu/TiddlyWikiClassicPluginsArchives/blob/main/TiddlyWikiClassicPluginsArchives.txt . Give it a try!

Best,

Yanshu

[sam paul]

I added a demo page https://wangyenshu.github.io/TiddlyWikiClassicPluginsArchives/demo.html .

Best,

Yanshu

[Yakov]

Hello Yanshu,

first of all, I'm very glad that you care about the plugin indexing and created those collections, I appreciate help with this stuff!

There's a lot of things to reply to, I'll do my best to answer in a day or two.

Best regards,

Yakov

четверг, 10 июля 2025 г. в 05:38:47 UTC+3, samp...@gmail.com:

[sam paul]

Dear Yakov,

Is it possible to install the theme and CSS using ExtensionExplorerPlugin? Will all installed tiddlers be tagged as systemConfig?

Best,

Sam

[Yakov]

It is possible, in fact there's an early version of ThemesCollection in the TiddlyThemes repo, and it's also possible to add any tiddler into a collection; the difference is, 'plugin' is one of the 2 tiddler types that's handled in a special way when installing with EEP:

• for plugins, the systemConfig tag is added (and also they are attempted to be installed at once, without reload);
• for collections, the systemExtensionsCollection tag is added.

So, a non-plugin tiddler won't be tagged with systemConfig. Note: aside reading the specified type in a collection itself, EEP also does some type guessing, which is simple for now: if a tiddler name ends with "plugin" (case-insensitive), it is considered as a plugin. Another thing to note is EEP doesn't support mutli-tiddler installations yet (which would be helpful for some themes and complex extensions, like TiddlyTools' QuickEditPackage).

>  can I put the extension list txt inside a tiddlywiki classic?

Yes, you can. That's not optimal for the end user in terms of performance (at least for now), but EEP supports links to tiddlers inside a TW; however, CORS must be enabled on the hosting of such a TW for EEP to be able to load it (for instance, it is enabled on Github Pages, but disabled on Tiddlyhost). You may want to take a look at the examples of extensions listed inside EEP itself: FieldsEditorPlugin is loaded from a TW, and a collection can be loaded from a TW, too. However, there's no support for in-TW filter-based collections (i.e. you can't tell "every tiddler with the tag MyPlugin is included into this collection" yet).

(I'll reply to some other points later)

суббота, 12 июля 2025 г. в 20:29:47 UTC+3, samp...@gmail.com:

[Yakov]

Now, there's an important issue that I'd like to discuss. There are 2 main kind of efforts to indexing the extensions:

• Efforts to index as many existing extensions as possible and
• Efforts to make them easy to explore/find/navigate.

You did a great job in the first area, thanks for that!

With current functionality of EEP, though, this will be difficult to use, unfortunately. Take, for instance, abego plugins:

• The current collection has only plugin names in their description;
• Some plugins are irrelevant for most of the users (for instance, LegacyStrikeThroughPlugin and probably AutoRefreshPlugin), some are outdated and/or have improved forks (for instance, IncludePlugin.. ouch, there's something wrong about your copy, and not only that plugin; anyway, SharedTiddlersPlugin is basically the "latest version" of IncludePlugin);
• The overall number of old and poorly documented extensions is large (not in abego, I mean all the collections).

Finally, imagine that you want to find a plugin to encrypt things or to somehow handle tons of tags and you're not sure what kind of tool you need. Try to find it in your demo, and you'll realize it's hard.

It's not your fault by any means; moreover, this great work will help us shape some features of EEP so that one can find things (or even explore, I hope). For instance, we can implement some "deep search" with it. But I'm not 100% sure it will work nicely without manual annotating things. Here's a specific suggestion: could you implement indexing plugins in a way that "description" is calced in the way below?

• If tiddler has slice Description (or description), use it;
• Otherwise, do the same for the slice Summary (when indexing TWs one by one, you may find other slices author used);
• Otherwise, use tiddler title.

You may wonder what's the "Efforts to make them easy to explore/find/navigate." thing and can I give an example? Well, that's a much more time-consuming process, but it gives some nice results, too. In my abego "fork", I've created the maintained folder, and there I've created the collection where I add the plugins one by one, and only those which I'm sure can be relevant these days. I've also updated the repository to 2.10.2, updated the Source slices so that they will play well with EEP (i.e. if I update any of them, EEP will notice, even if they are installed by copy-pasting), fixed various links and other stuff. Yeah, the result is, there are only 3 plugins indexed in the collection yet, but at least nobody who adds the collection will have to review XHTML10Plugin to understand that they won't need it ever.

Another kind of efforts to simplify search/exploring is to improve EEP functionality, like implementing search and more.

Anyway, that's a good achievement (fun fact: I've never knew of the Minesweeper until noticed it in your collection). I will be really helpful if you try adding descriptions to the items of collections and also if you review the results and fix bugs like with abego that I've linked above. And I'll consider using your collection as a base for plugin fixes (TiddlyTools has various plugins that are buggy and Eric is somewhat silent about accepting fixes/updates), because this sounds somewhat convenient. Once the archive in a better shape, I think we can put it into the TiddlyWiki organization.

Thanks again!

Yakov

воскресенье, 13 июля 2025 г. в 22:38:25 UTC+3, Yakov:

[sam paul]

Dear Yakov,

Thanks for your feedback.

I have corrected the bug link in abego. This is weird. Probably there are some conflicts between abego's plugin and the plugin https://wangyenshu.github.io/TiddlyWikiClassicPluginsArchives/base.html#TiddlerInFilePlugin-ByTagFork.

I updated the script to generate JSON for extensions: https://wangyenshu.github.io/TiddlyWikiClassicPluginsArchives/base.html#Installed_Plugins .

It now uses regex stored in https://wangyenshu.github.io/TiddlyWikiClassicPluginsArchives/base.html#JSONRegexPattern to scrape description.

Since the extension JSON description file depends on the URL where the plugins are stored, I think it is better that we first have a repository in TiddlyWiki organization to store these plugins. It saves time.

Currently, the plugins are unorganized once inside the extension explorer interface. Maybe we should group them in tabs. Also, searching plugins is not implemented.

Best,

Sam

[sam paul]

Dear Yakov,

I create a github workflow file to archives plugins and themes from a list of systemServer stored in systemServer.txt, read the current repository URL and generate the description file automatically, store them in the commit folder and commit to the branch twcArchives everyday. For description, if there is a description in the block like | Description | something |, it will use that description, otherwise, the description can be empty or something else. It is able to archives almost all systemServer found in tiddlyvault (51 systemServers out of 55 systemServers). The repository is here https://github.com/wangyenshu/twcArchives . If you want to use it, you just need to clone it and enable the GitHub action. The generated extension description file will use the current repo URL automatically.

Current version is usable. As for the issues:

- This workflows file use the original tiddlywiki cooker (https://github.com/TiddlyWiki/cooker), which relies on ruby1.8. So, I had to use a Ubuntu 16.04 docker (root account) to extract plugins and themes. After I exit the docker,  I cannot use git init due to permission issue. So, I had to use https://github.com/stefanzweifel/git-auto-commit-action. And it commits everything, including workflows file to the branch twcArchives. That's not very good.

- It use an aggresive approach to find the description. For example, if the extension has the following text:

!Description

xxxxx

It will use '!Description' as the description. However, this approach works if their are description block like | Description | something |.

I don't have so much experience in writing github workflows. I will appreciate that if anyone could solve these issues or rewrite the 

cooker (https://github.com/TiddlyWiki/cooker) in ruby3.

For more details, read the README at https://github.com/wangyenshu/twcArchives .

The extension description file for this repo is https://github.com/wangyenshu/twcArchives/blob/twcArchives/commit/twcArchives.txt .

The archived plugins and themes and description file can be found at https://github.com/wangyenshu/twcArchives/tree/twcArchives/commit .

And I made a demo site at https://wangyenshu.github.io/twcArchives/demo.html .

Best,

Sam

[sam paul]

Currently, it archives and create extension description file for the following systemServers:

https://tiddlywiki.abego-software.de/
https://web.archive.org/web/20130801155706/http://tiddlywiki.bidix.info/
https://web.archive.org/web/20171113084110/http://bob.mcelrath.org/plugins.html#tag:systemConfig
https://mptw.tiddlyspot.com/
https://web.archive.org/web/20130926071240/https://ptw.sourceforge.net/ptwe.html
https://tiddlytools.com/Classic/
https://classic.tiddlywiki.com/#VisualTWServer
https://web.archive.org/web/20100527192535/http://wiki.osmosoft.com/alpha/
https://web.archive.org/web/20181110203611/http://coderswiki.jenskillus.de/
https://web.archive.org/web/20080125164944/http://conflation.org/#BlogUserPlugin
http://web.archive.org/web/20120321101358/http://www.martinswiki.com/
https://bradleymeck.tiddlyspot.com/
http://solo.dc3.com/tw/
https://web.archive.org/web/20140127010925/http://tiddlystyles.com/
https://web.archive.org/web/20240126134323/http://aiddlywiki.sourceforge.net/wikibar_demo_2.html
https://web.archive.org/web/20070915152316/http://dahukanna.net/wiki/tiddlywiki.htm
https://web.archive.org/web/20120703000235/http://danielbaird.com/tiddlywinks/
http://wordcountmacro.tiddlyspot.com/
http://linktomemacro.tiddlyspot.com/
http://devpad.tiddlyspot.com
http://moodle.tiddlyspot.com/
http://discerningtags.tiddlyspot.com/
https://web.archive.org/web/20090311042726/http://end.com/~speth/HaloscanMacro.html
https://web.archive.org/web/20110316090138/http://fire.atspace.org/
https://web.archive.org/web/20080411233439/http://genisis329.googlepages.com/TWMath.html
http://ido-xp.tiddlyspot.com/
https://web.archive.org/web/20090201071059/http://jackparke.googlepages.com/jtw.html
https://web.archive.org/web/20130827021950/http://journal.conal.net/#%5B%5Bsite%20map%5D%5D
https://web.archive.org/web/20110228140439/http://www.tiddly-twab.com/
https://web.archive.org/web/20090928220707/http://yann.perrin.googlepages.com/twkd.html
https://web.archive.org/web/20100418173404/http://www.zagware.com/tw/plugins.html#DcTableOfContentsPlugin
https://web.archive.org/web/20090918230537/http://knighjm.googlepages.com/knightnet-default-tw.html#PlugIn-Knightnet
http://labwiki.sourceforge.net/
https://web.archive.org/web/20090525121320/http://lewcid.googlepages.com/presentation_empty_full.html#Documentation
http://mptw2.tiddlyspot.com/#Plugins
https://web.archive.org/web/20100810151108/http://paularodrigo.homeip.net/stickydate.html#StickyDatePlugin
https://web.archive.org/web/20160224131156/http://project.dahukanna.net/tiddlywiki/timeline/sampletimeline220b5.html
http://remindermacros.tiddlyspot.com/
https://web.archive.org/web/20121113193541/http://remotely-helpful.com/TiddlyWiki/ImprovedSlicesPlugin.html#ImprovedSlicesPlugin
https://web.archive.org/web/20140623152642/http://remotely-helpful.com/TiddlyWiki/LaunchApplication.html#LaunchApplicationPlugin
https://web.archive.org/web/20150531055530/http://remotely-helpful.com/TiddlyWiki/TiddlerEncryptionPlugin.html#TiddlerEncryptionPlugin
http://rgplugins.tiddlyspot.com/#%5B%5BhideIfshowIf%20plugin%5D%5D
http://soloport.tiddlyspot.com/#ExcludeTimelinePlugin
http://syncoutlooknotes.tiddlyspot.com/#SyncOutlookNotes
https://web.archive.org/web/20090209220502/http://tw.lewcid.org/#%5B%5BExtensions%20Overview%5D%5D
http://weave.tiddlyspot.com
https://web.archive.org/web/20080928120640/http://www-static.cc.gatech.edu/~dellaert/dhtml/tiddly.html
https://web.archive.org/web/20101021093750/http://www.anshul.info/blogwiki.html
https://web.archive.org/web/20140121045309/http://www.blogjones.com/TiddlyWikiTutorial.html#smiley
http://coursebook.tiddlyspot.com/#RevisionsPlugin
https://web.archive.org/web/20121004025926/http://www.math.ist.utl.pt/~psoares/addons.html#Plugins

You can find this list at https://github.com/wangyenshu/twcArchives/blob/main/systemServer.txt .

The wayback machine has rate limit. So, after archving one URL, it will sleep for 2 minutes before starting to archives the next URL. Now, it takes roughly 1h 50 minutes to complete the job.

Best,

Sam

[sam paul]

A nice thing to mention is that you can view the traffic in this repo by https://github.com/wangyenshu/twcArchives/graphs/traffic . So if there are enough people using it. We can know which plugins are more popular.

Best,

Sam