Feature Request: high-priority "bootloader" module-type

75 views
Skip to first unread message

Evan Balster

unread,
Feb 12, 2018, 2:05:06 PM2/12/18
to TiddlyWikiDev
Hey, all —

(I had a Twitter discussion with Jeremy yesterday about this topic which quickly overflowed the boundaries of that medium.)

To facilitate development of my Mod-Loader plugin, I'm requesting a new module-type that runs during boot with maximal priority.  This module-type would be used for modules that need to add, remove or modify items in the tiddler store which become "permanent" during the boot process (IE, other JavaScript modules).

While the modloader is my main concern, I've given some thought to other use-cases for boot-time modification of the tiddler store.  The obvious one is "loader extensions":  That is, any plugin that loads tiddlers from a non-default source.  A boot module could load some content from a network or file source at boot-time, from the client-side.


Boot modules could work in a fashion similar to startup modules, running first.  I suspect it might be preferable for boot modules to be loaded and run one-by-one and prioritized by field values rather than JavaScript-defined properties, though.  This would allow them to modify one another or inject other boot modules into the sequence (potentially by adding or modifying plugins).

Jeremy suggested yesterday that high-priority code like this could be installed as raw markup tiddlers, but I'm a bit skeptical of this approach.  I think it's important that modules like this have access to tiddler store functionality, and remain portable across the many different formats TiddlyWiki can be packaged into.  I also think it should remain possible to package these boot modules into plugins.

Evan Balster

unread,
Feb 12, 2018, 2:08:49 PM2/12/18
to TiddlyWikiDev
Addendum:  If startup modules aren't collectively preloaded before they are run, boot modules may be unnecessary.  However, this would force a change in the prioritization mechanism (using tiddler fields rather than export variables).

I think there might also be a meaningful difference in scope between bootloader functionality (modifying the tiddler store) and startup functionality (preparing functionality), but that's a loose opinion.

Jeremy Ruston

unread,
Feb 12, 2018, 3:00:10 PM2/12/18
to tiddly...@googlegroups.com
Hi Evan

I worked up the technique I mentioned, and it may be suitable for you.

Working in Node.js, create a file called “patchcore.js" in a test wiki with the following content:

/*\
title: $:/PatchCore.js
type: application/javascript
library: yes

Demo of patching the core plugin

\*/
(function(){

/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";

var preCore = document.querySelector("div[title='$:/core'] > pre"),
pluginText = preCore.textContent;

pluginText = pluginText.replace(/a non-linear personal web notebook/mg,"this is a patch");
preCore.textContent = pluginText;

window.$tw = window.$tw || {};
$tw.preloadTiddlers = $tw.preloadTiddlers || [];
$tw.preloadTiddlers.push({title: "HelloJeremy",text: "Hello there Jeremy"});


})();

What I was suggesting would take a little bit more code: instead of writing the modified plugin text back to the DOM, you could instead neutralise the DOM element (by targetting the DIV and removing the title attribute) and insert the tiddler via $tw.preloadTiddlers.

Best wishes

Jeremy

-- 
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/2a76bfeb-b709-46a9-9797-ab4acb82c9a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jeremy Ruston

unread,
Feb 12, 2018, 3:09:17 PM2/12/18
to tiddly...@googlegroups.com
Here’s the .tid of the $:/PatchCore.js tiddler, too.

Best wishes

Jeremy

$__PatchCore.js.tid

Evan Balster

unread,
Feb 12, 2018, 3:57:33 PM2/12/18
to TiddlyWikiDev
Hey, Jeremy —

So, as mentioned in the OP I'm pretty leery of the "pre-boot" approach here.  The modloader process uses quite a lot of $:/boot library functionality to identify and inspect shadow tiddlers as part of the patching process, and to generate the output plugin.  I also worry that working on the pre-boot DOM would create incompatibilities with certain TiddlyWiki environments (but I don't understand this low level stuff very well).  The current modloader can manipulate almost everything in the wiki except for startup modules and the modules they require, hence the request.

In most cases, bootloader modules (including modloader) are going to want to generate any output to $:/temp or similar, where they won't create persistent changes in the wiki.

I expect we'll end up discussing in more depth during tomorrow's call.

Jeremy Ruston

unread,
Feb 12, 2018, 4:10:16 PM2/12/18
to tiddly...@googlegroups.com
Hi Evan

We’ve got to the point where the changes you are proposing to the core are quite deep, and have lots of implications for the rest of the core. It’s going to mean a lot of work on verification and testing from me, even if somebody else writes the code. I don’t think that burden is justified at present.

Better to work with the plugin for a while, and explore the capabilities and consequences before we commit to a new mechanism in the core.

So, as mentioned in the OP I'm pretty leery of the "pre-boot" approach here.  The modloader process uses quite a lot of $:/boot library functionality to identify and inspect shadow tiddlers as part of the patching process, and to generate the output plugin.

At this point, I don’t see any problem with your plugin duplicating a few hundred lines of boot.js; eliminating such duplication would come firmly under the heading of optimisation.

I also worry that working on the pre-boot DOM would create incompatibilities with certain TiddlyWiki environments (but I don't understand this low level stuff very well).

You can see all the code that depends on this stuff. The approach is solid.

The current modloader can manipulate almost everything in the wiki except for startup modules and the modules they require, hence the request.

Yes, but your proposed solution is just an escalation in an ongoing arms race to be the first module to run.

In most cases, bootloader modules (including modloader) are going to want to generate any output to $:/temp or similar, where they won't create persistent changes in the wiki.

But equally, there’s no limit to the functionality in the boot kernel that experimental plugins might want to tweak. Better to be based on a pre-boot hook, and have the capability to hook anything, than to be dependent on the hooks anticipated and provided by the core.

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

Jeremy Ruston

unread,
Feb 12, 2018, 5:31:04 PM2/12/18
to tiddly...@googlegroups.com
Hi Evan

Apologies, the end of my reply got garbled in copy and pasting:

In most cases, bootloader modules (including modloader) are going to want to generate any output to $:/temp or similar, where they won't create persistent changes in the wiki.

My reply to this paragraph was missing:

> That’s OK, any $:/temp tiddlers in $tw.preloadTiddlers will still have the normal behaviour of not being persisted.

I should also have added that I think we can enhance the behaviour of library:yes tiddlers so that they can reside in shadow tiddlers.

Best wishes

Jeremy.

Evan Balster

unread,
Feb 12, 2018, 5:33:42 PM2/12/18
to TiddlyWikiDev
Hey, Jeremy —
 
Yes, but your proposed solution is just an escalation in an ongoing arms race to be the first module to run.

This is almost, but not quite, the issue.  My modloader is the first module to have any method called on it.  However, TiddlyWiki currently loads (via $tw.module.execute) a very large number of modules prior to making any method calls.  There is no control over the ordering of these executions, and they are subsequently impossible to change at boot-time without overwriting the core.

As I mentioned before, I think there's potentially a meaningful difference between startup modules and "bootloader" behaviors.  Loading content and code into the tiddler store, versus setting up APIs, DOMs and run-time functionality.  It's much more likely that a loader module would want to affect startup behavior than another loader module.


We’ve got to the point where the changes you are proposing to the core are quite deep, and have lots of implications for the rest of the core. It’s going to mean a lot of work on verification and testing from me, even if somebody else writes the code. I don’t think that burden is justified at present.

A minimal implementation of bootloader modules would look something like this:

// Run bootloader modules
$tw.modules.forEachModuleOfType("bootloader",function(title,module) {
if(module.boot) module.boot();
});

This would go right before "startup" modules.  Lack of prioritization and "one by one" execution is less than ideal but still preferable to the current constraints.

Anyway, the modloader doesn't need this functionality, strictly speaking.  It just extends its capabilities a bit, so that the whole core can be patched by installed plugins.  The modloader currently issues a warning when trying (and failing) to apply a patch of this nature so it shouldn't catch developers by surprise.


At this point, I don’t see any problem with your plugin duplicating a few hundred lines of boot.js.

The design intent of the modloader is to maintain compatibility with future versions of TiddlyWiki.  Duplicating bootloader behavior (which is subject to change) would detrimental enough to that goal that I'd rather stick with the current constraints.

Jeremy Ruston

unread,
Feb 12, 2018, 5:53:53 PM2/12/18
to tiddly...@googlegroups.com
Hi Evan

Thanks, I’ll give your arguments more thought, and hope you can join the hangout tomorrow to discuss further.

This is almost, but not quite, the issue.  My modloader is the first module to have any method called on it.  However, TiddlyWiki currently loads (via $tw.module.execute) a very large number of modules prior to making any method calls.  There is no control over the ordering of these executions, and they are subsequently impossible to change at boot-time without overwriting the core.

Aren’t those executions just the cascade of modules requiring one another?

As I mentioned before, I think there's potentially a meaningful difference between startup modules and "bootloader" behaviors.  Loading content and code into the tiddler store, versus setting up APIs, DOMs and run-time functionality.  It's much more likely that a loader module would want to affect startup behavior than another loader module.

Bootloader modules feel a bit like an admission of defeat with respect to the startup module mechanism.

It might be a useful exercise to try to write the documentation for bootloader modules, and see whether one can explain the difference clearly without referencing the particular use case that has prompted this discussion.

I worry that they are a narrow solution to your specific problem. We could also be looking at something buggyj has already started: making it easier for plugins to splice in raw markup code, and possibly adding more hooks to the boot kernel (eg, an explicit “patch-tiddler” hook).

Anyway, the modloader doesn't need this functionality, strictly speaking.  It just extends its capabilities a bit, so that the whole core can be patched by installed plugins.  The modloader currently issues a warning when trying (and failing) to apply a patch of this nature so it shouldn't catch developers by surprise.

Excellent.

The design intent of the modloader is to maintain compatibility with future versions of TiddlyWiki.  Duplicating bootloader behavior (which is subject to change) would detrimental enough to that goal that I'd rather stick with the current constraints.

But I think the code that you’d need to duplicate is code that’s only going to change if the underlying serialisation formats change? If that happens all bets are off with respect to future versions of TiddlyWiki maintaining backwards compatibility with this February 2018 incarnation of the plugin.

Best wishes

Jeremy.

Evan Balster

unread,
Feb 12, 2018, 6:30:44 PM2/12/18
to TiddlyWikiDev
Hey, Jeremy —

Bootloader modules feel a bit like an admission of defeat with respect to the startup module mechanism.

Potentially this functionality could be an extension of the startup mechanism.  Trouble is, in order to load and execute them one-by-one you'd need to be able to prioritize startup modules without executing them, meaning that their dependency information would need to be migrated to fields.   This would break compatibility with any plugins that define startup modules.

The other problem is that my plugin's requirement — "minimize the number of modules executing before me" — is at odds with the dependency-oriented nature of startup modules.  Modloader doesn't depend on any functionality other than what's provided by the bootloader.


Aren’t (the executions before startup) just the cascade of modules requiring one another?

Yes.  The requirement for my plugin is "run before most modules execute".  There's no best practice for startup modules to discourage them from loading modules at execution time, and I'm not sure there should be(?), hence the conundrum.

You'll notice my modloader plugin eclipses startup.js to move its require() call into its startup() method.  If I don't do this, it's impossible to modify a large portion of the core (including all widgets).  Unfortunately, this eclipsing is exactly the kind of thing the modloader is designed to avoid.


It might be a useful exercise to try to write the documentation for bootloader modules, and see whether one can explain the difference clearly without referencing the particular use case that has prompted this discussion.

"bootloader modules run during the boot process, allowing them to add, remove and modify tiddlers in the store prior to the wiki's startup process.  This is useful for injecting plugins, code and content that TiddlyWiki isn't capable of refreshing at runtime.  Injected content may be dynamically generated or loaded from an external source.  These modules are encouraged to minimize their dependency upon other modules in the wiki and to avoid any dependency on the TiddlyWiki core.

bootloader modules may safely revise any content in the tiddler store with the exception of other bootloader modules and their dependencies."

Jeremy Ruston

unread,
Feb 13, 2018, 6:57:07 AM2/13/18
to TiddlyWikiDev
Hi Evan

The other problem is that my plugin's requirement — "minimize the number of modules executing before me" — is at odds with the dependency-oriented nature of startup modules.  Modloader doesn't depend on any functionality other than what's provided by the bootloader.

“Minimize the number of modules executing before me” is the key phrase that triggers my concern about an arms race; I’ve seen it so many times before from CSS to TSRs, and introducing another category is never a good long term answer. The pre-boot approach is attractive precisely because it’s outside the normal mechanisms of TiddlyWiki, and hence can be used to patch/hack those mechanisms. It is there to help avoid the kind of patching-the-rug-you’re-standing-on situation we face here.

Aren’t (the executions before startup) just the cascade of modules requiring one another?

Yes.  The requirement for my plugin is "run before most modules execute".  There's no best practice for startup modules to discourage them from loading modules at execution time, and I'm not sure there should be(?), hence the conundrum.

You'll notice my modloader plugin eclipses startup.js to move its require() call into its startup() method.  If I don't do this, it's impossible to modify a large portion of the core (including all widgets).  Unfortunately, this eclipsing is exactly the kind of thing the modloader is designed to avoid.

I’ve no objection to moving existing require() calls into a different scope.

It might be a useful exercise to try to write the documentation for bootloader modules, and see whether one can explain the difference clearly without referencing the particular use case that has prompted this discussion.

"bootloader modules run during the boot process, allowing them to add, remove and modify tiddlers in the store prior to the wiki's startup process.  This is useful for injecting plugins, code and content that TiddlyWiki isn't capable of refreshing at runtime.  Injected content may be dynamically generated or loaded from an external source.  These modules are encouraged to minimize their dependency upon other modules in the wiki and to avoid any dependency on the TiddlyWiki core.

bootloader modules may safely revise any content in the tiddler store with the exception of other bootloader modules and their dependencies.”

And that sounds pretty much like a description of the existing $tw.preloadTiddlers mechanism. The preboot approach offers the interesting possibility of patching boot.js, too.

I’m happy to discuss further but my position remains:

* The proposed modifications don’t have any anticipated benefit beyond the usage by the modloader plugin
* The proposed modifications are relatively expensive in terms of time and attention from me and the rest of the core team
* The modloader plugin already exists and works, with some limitations
* The preboot approach already exists as an alternative mechanism that could be used instead, with different limitations

Best wishes

Jeremy

Evan Balster

unread,
Feb 13, 2018, 5:20:14 PM2/13/18
to TiddlyWikiDev
Hey, Jeremy —

With what you've told me about the preboot approach on the call, it's starting to sound a bit better for modloader's needs.  Learning that the serialization DOM format is fixed across all TW5 platforms helps me to visualize a solution.

I'm wondering if this pre-boot approach will be incompatible with encrypted wikis, though.  I have at least one of those that I'd really like to use patches with, if possible.  Are plugins (including core) encrypted, or are they excluded from that mechanism?


* The proposed modifications don’t have any anticipated benefit beyond the usage by the modloader plugin

I will continue to voice my dissent to this argument.  There are other useful mechanisms that might want to add code before startup.  For example, a plugin could fetch some widget script via HTTPS in order to ensure that client wikis are always using the latest version.  Arguably this could be done in pre-boot like the modloader, too, but it's an example of something that needs to deal with similar constraints.
Reply all
Reply to author
Forward
0 new messages