Node.js server non-core plugins deleted silently during upgrade

[David Nebauer]

I'm primarily using node.js client-server TW5. I've just finished migrating my plugins from individual client wikis to the server (under the default location of /usr/local/lib/node_modules/tiddlywiki/plugins). Today I upgraded node.js TW5 from 5.1.20 to 5.1.21 and got a rude surprise when it silently deleted all plugins from this default directory and (re)installed only the core plugins. I very thankfully had a backup so this was trivial for me to recover from, but it was still unexpected and potentially very inconvenient.

The best way to avoid this problem would be to quarantine non-core plugins during the upgrade process. The next best solution would be to build a warning into the upgrade process if non-core plugins are detected in the default location, with an option to abort. Failing either of those, I suggest a prominent warning be added to documentation tiddler [[Upgrading TiddlyWiki on Node.js]].

Since this near miss I have moved my plugins out of the TW5 server install tree and set environmental variable TIDDLYWIKI_PLUGIN_PATH accordingly. It might be worth explaining in documentation tiddler [[Environment Variables on Node.js]] that a major benefit of relocating non-core plugins (and themes, etc.) from their default locations is to avoid silent deletion during upgrades.

Regards,

David

Note: By 'non-core' plugins I mean plugins other than those installed under /usr/local/lib/node_modules/tiddlywiki/plugins/tiddlywiki. I realise they are not 'core' in the sense of being necessary for ordinary TW functioning. There is probably a more accurate label I could use.

[Jeremy Ruston]

Hi David

I'm primarily using node.js client-server TW5. I've just finished migrating my plugins from individual client wikis to the server (under the default location of /usr/local/lib/node_modules/tiddlywiki/plugins). Today I upgraded node.js TW5 from 5.1.20 to 5.1.21 and got a rude surprise when it silently deleted all plugins from this default directory and (re)installed only the core plugins. I very thankfully had a backup so this was trivial for me to recover from, but it was still unexpected and potentially very inconvenient.

Very pleased you had your backup!

You had used npm to install TiddlyWiki globally. npm treats the node_modules directory as private data, and doesn’t expect other processes to interfere with it.

The best way to avoid this problem would be to quarantine non-core plugins during the upgrade process. The next best solution would be to build a warning into the upgrade process if non-core plugins are detected in the default location, with an option to abort. Failing either of those, I suggest a prominent warning be added to documentation tiddler [[Upgrading TiddlyWiki on Node.js]].

Note that the upgrade process is handled by npm, not TiddlyWiki. I don’t think the problem of upgrades overwriting local changes is unique to TiddlyWiki.

Since this near miss I have moved my plugins out of the TW5 server install tree and set environmental variable TIDDLYWIKI_PLUGIN_PATH accordingly. It might be worth explaining in documentation tiddler [[Environment Variables on Node.js]] that a major benefit of relocating non-core plugins (and themes, etc.) from their default locations is to avoid silent deletion during upgrades.

Indeed, the best way to make your plugins globally available with that configuration is to store them somewhere else and use environment variables to pass their location to TW:

https://tiddlywiki.com/#Environment%20Variables%20on%20Node.js

Best wishes

Jeremy

Regards,

David

Note: By 'non-core' plugins I mean plugins other than those installed under /usr/local/lib/node_modules/tiddlywiki/plugins/tiddlywiki. I realise they are not 'core' in the sense of being necessary for ordinary TW functioning. There is probably a more accurate label I could use.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/5683bfe7-6d46-42fe-b87d-76ba60b0e604%40googlegroups.com.

[David Nebauer]

Thank you for your rapid response, Jeremy. Everything you said was technically correct, as I would expect from the creator/guru of tiddlywiki. I'm afraid, however, it showed little insight into the needs of ordinary users of tiddlywiki who won't appreciate the subtleties of npm's ownership assumptions or plugin install locations until they discover one day, shortly after following tiddlywiki's own upgrade instructions, that tiddlywiki can no longer find its non-core plugins, and then discover their plugins have disappeared. (As just happened to me.) I can appreciate that modifying the npm upgrade process may not be possible (though, has anyone actually checked?). Surely, however, this scenario could be prevented in some cases by simply adding some information to the [[Upgrading TiddlyWiki on Node.js]] tiddler, and possibly also the [[Environment Variables on Node.js]] tiddler. (It would certainly have done so in my case.) Regrettably, not every current and future user of tiddlywiki is going to read the excellent explanation you provided in your post...

Regards,

David

On Saturday, 19 October 2019 20:50:15 UTC+9:30, Jeremy Ruston wrote:

You had used npm to install TiddlyWiki globally. npm treats the node_modules directory as private data, and doesn’t expect other processes to interfere with it. 

Note that the upgrade process is handled by npm, not TiddlyWiki. I don’t think the problem of upgrades overwriting local changes is unique to TiddlyWiki.

[Jeremy Ruston]

Hi David

Thank you for your rapid response, Jeremy. Everything you said was technically correct, as I would expect from the creator/guru of tiddlywiki. I'm afraid, however, it showed little insight into the needs of ordinary users of tiddlywiki

I’m not sure what point you want to make by saying that my response lacks insight. I just answered your questions and commented on your assertions/suggestions, admittedly somewhat briefly. I didn’t have the time to go into potential improvements to the documentation, but welcome suggestions for how to do so.

who won't appreciate the subtleties of npm's ownership assumptions or plugin install locations until they discover one day, shortly after following tiddlywiki's own upgrade instructions, that tiddlywiki can no longer find its non-core plugins, and then discover their plugins have disappeared. (As just happened to me.)

I don’t expect ordinary end users to understand those subtleties. But, I’ve never before come across the assumption that an end user should be able to modify an artefact downloaded from an app store or package manager, and expect those modifications to survive an upgrade. In general, it’s hard for documentation to explicitly cover things that shouldn’t be done, because there are so many possibilities; instead we try to describe the things that should be done, and anticipate as many of the mistakes and misunderstandings that we can.

I can appreciate that modifying the npm upgrade process may not be possible (though, has anyone actually checked?).

As it happens, npm has lots of hooks and it’s highly likely that the upgrade process can be modified. But the main point I was making was that the problem of npm upgrades overwriting any custom modifications isn’t unique to TW, it’s part of how npm works. It isn’t TW’s job to “fix” how npm works.

Surely, however, this scenario could be prevented in some cases by simply adding some information to the [[Upgrading TiddlyWiki on Node.js]] tiddler, and possibly also the [[Environment Variables on Node.js]] tiddler. (It would certainly have done so in my case.) Regrettably, not every current and future user of tiddlywiki is going to read the excellent explanation you provided in your post…

I think the documentation failure was that you couldn't find out how to install custom plugins globally. Beyond that, the moment you decided to modify the node_modules directory was a bit like getting out of the jeep in Jurassic Park, and it’s not clear that the specific things that happened to you are particularly universal.

I can see that a note in the upgrade tiddler might have helped you with the specific journey you were taken on, but by then you were already past the point where the mistake had been made.

Best wishes

Jeremy

Regards,

David

On Saturday, 19 October 2019 20:50:15 UTC+9:30, Jeremy Ruston wrote:

You had used npm to install TiddlyWiki globally. npm treats the node_modules directory as private data, and doesn’t expect other processes to interfere with it. 

Note that the upgrade process is handled by npm, not TiddlyWiki. I don’t think the problem of upgrades overwriting local changes is unique to TiddlyWiki.

Indeed, the best way to make your plugins globally available with that configuration is to store them somewhere else and use environment variables to pass their location to TW: https://tiddlywiki.com/#Environment%20Variables%20on%20Node.js

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/b7563084-bec3-444d-aa06-5ef396b9635f%40googlegroups.com.

[David Nebauer]

I apologise. I see we were at cross-purposes. I thought installing plugins in the server plugin directory was a legitimate mode of tw operation. In fact, I thought a major advantage of the node.js version was that plugins could be managed (i.e., installed and updated) in a single location rather than having to manage plugin installation and versioning across multiple individual wikis. I had no idea this was considered the equivalent of getting out of the jeep in Jurassic Park.

I don't think my experience is anywhere close to universal, but nor do I think it unique. A number of plugins announced in the tw gg include source files and instructions for installing them in tw server (or, at least, that's how I interpreted the instructions).

Anyone intending to manage their non-core plugins centrally should absolutely avoid the server plugin directory and keep them elsewhere, letting tw know this location with TIDDLYWIKI_PLUGIN_PATH. I support modifying the tw documentation to explain why this is a good idea and why use of the server plugin directory should be avoided at all costs.

Thanks for your thorough explanation and your patience.

Regards,

David.

[Jeremy Ruston]

Hi David

Apologies for the delayed response.

On 20 Oct 2019, at 06:36, David Nebauer <davidn...@gmail.com> wrote:

I apologise. I see we were at cross-purposes. I thought installing plugins in the server plugin directory was a legitimate mode of tw operation.

It depends how TW has been installed. If one clones the GitHub repo and then adds some custom plugins then one can use git to cleanly upgrade when the repo changes. We added support for the environment variables to support some of the other possible configurations.

In fact, I thought a major advantage of the node.js version was that plugins could be managed (i.e., installed and updated) in a single location rather than having to manage plugin installation and versioning across multiple individual wikis. I had no idea this was considered the equivalent of getting out of the jeep in Jurassic Park.

The Jurassic Park moment was deciding to modify an asset downloaded by the package manager.

I don't think my experience is anywhere close to universal, but nor do I think it unique.

It’s probably not unique, but it’s not something that’s been reported before.

A number of plugins announced in the tw gg include source files and instructions for installing them in tw server (or, at least, that's how I interpreted the instructions).

I suspect it’s quite common practice for developers to work with a fork of the TW5 repo, which as noted above does work as expected.

Anyone intending to manage their non-core plugins centrally should absolutely avoid the server plugin directory and keep them elsewhere, letting tw know this location with TIDDLYWIKI_PLUGIN_PATH. I support modifying the tw documentation to explain why this is a good idea and why use of the server plugin directory should be avoided at all costs.

I’ve made a few changes to the docs that will be active on tiddlywiki.com in about 10 minutes:

https://github.com/Jermolene/TiddlyWiki5/commit/df85281420cccc06870e7dce2aafddf2327708a1

Best wishes

Jeremy

Thanks for your thorough explanation and your patience.

Regards,

David.

On Sunday, 20 October 2019 00:19:44 UTC+9:30, Jeremy Ruston wrote:

I think the documentation failure was that you couldn't find out how to install custom plugins globally. Beyond that, the moment you decided to modify the node_modules directory was a bit like getting out of the jeep in Jurassic Park, and it’s not clear that the specific things that happened to you are particularly universal.

I can see that a note in the upgrade tiddler might have helped you with the specific journey you were taken on, but by then you were already past the point where the mistake had been made.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/2ae9e0b0-b08e-4cdb-986e-9f958a51e868%40googlegroups.com.

[David Nebauer]

Thanks, Jeremy. Anyone reading that would not make the same mistake I did.

You might want to make some complementary changes to the TiddlyWikiFolders tiddler as well, as it currently states in relation to tiddlywiki.info only that, "Plugins names refer to plugin folders listed in TiddlyWiki5's root plugins folder," and makes no reference to other variable-defined plugin folders. Perhaps just add a link to the Installing custom plugins on Node.js tiddler as you did in the Installing a plugin from the plugin library tiddler?

I also noticed that it is not always made clear that in the tiddlywiki.info file the "plugin names" are actually the plugin subfolder path under the plugin directory. This is made clear in the TiddlyWikiFolders tiddler listed above but not, for example, in the tiddlywiki.info Files tiddler which states only, "It should contain a JSON object comprising the following properties: plugins - an array of plugin names to be included in the wiki." The Installing custom plugins on Node.js tiddler you added also refers to plugin name rather than subdirectory: "The plugins can be referenced in tiddlywiki.info by their name (e.g. tiddlytools/magic)."

I realise the overwhelming custom and practice in the tw community is to use the same values for plugin directory as plugin name, so the distinction will not matter in practice most of the time. It only caught me out when I decided to rename a plugin folder name from camel-case to lower-case dash-separated, and found suddenly tw could not find the plugin even though its name remained unchanged.

Regards,

David.