Plugin Request: Use sqlite db to store tiddlers and load them using sql.js

[sam paul]

Hello. This GitHub repo https://github.com/simonw/datasette-tiddlywiki inspired this idea. Suppose we can save tiddlers in an external SQLite database when editing, and some plugins allow us to use https://sql.js.org to load them when viewed over the Internet. In that case, the migration between TiddlyWiki and other CMSs will be much simpler. Also, this can reduce the size of the wiki if you have too many tiddlers.

Best,

Sam

[Okido]

Hello Sam,

SQLite is indeed a good choice. I did some testing already and it is just working well.

The only issue is that in a browser environment it will not be easy to get it running smooth, file access is still complicated.

I use it SQLite with node and NW.js, SQLite is included in Node.js since a few months and works now without any extra installs on node.

When I have some news to report I will certainly do so.

Remember to share good fortune with your friends, Okido

Op zondag 20 juli 2025 om 19:12:30 UTC+2 schreef samp...@gmail.com:

[Yakov]

Hi, couple of notes about potential implementation:

• It's probably best to start from TiddlyWiki.prototype.getLoader and TiddlyWiki.prototype.getSaver, implement another loader (current: TW21, base) and saver (TW21, base) and amend the .getLoader and .getSaver methods. I'm not 100% sure (TiddlyWiki.prototype.loadFromDiv doesn't look generic), but it probably will work. There are also config.adaptors, but they are only used for importing and Story.prototype.loadMissingTiddler, which seems to be kinda improper design (I'd expect that they are used at least in loaders, too, and in fact also in savers), so you can ignore them for now, but for better core extensibility this should be researched.
  • Ok, loader and saver only define how the tiddlers are serialized ("externalized") and deserialized ("internalized"), for the simplest implementation you can probably leave .externalizeTiddler and .internalizeTiddler as is (although you may want to use JSON instead of HTML) and ignore externalize, but you'll need to change saveMain and TiddlyWiki.prototype.loadFromDiv (saveMain should be a method of TiddlyWiki.prototype, too, this is a design flaw that may be corrected in the future).
• Communicating to a DB directly is a bit insecure, since you'll need to store secrets in the TW itself. It can be ok for a prototype or a TW that's only used locally, but generally the approach with an intermediate Node.js server is preferable (nice catch on SQLite being included in Node.js, Okido!)

понедельник, 21 июля 2025 г. в 21:51:38 UTC+3, Okido:

[milind...@gmail.com]

  Just another input to think about. I have always wanted this but haven't checked recently on it. Maybe the database link can be standardized using a server client layer in between. Although it will require running of a sqlite server which I see is possible (for example https://github.com/forchid/sqlite-server), the huge advantage that comes with it is then you can have tiddlywiki be served from a web server with any kind of database in the back. And then it can act as a personal wiki and store tiddlers just as it does right now and also be used as a shared wiki by marking tiddlers from a remote source. 

In my opinion this would make it extremely useful and scalable to groups from a personal wiki level. Although it will bring new issues like permissions, conflicts and all, I think just having this first step would be a big step, and those issues can be resolved in small teams using other means if they want to use it.

Milind

[Yakov]

Here's another thing to consider. Node.js servers (and cli-s) that deal with saving TW or its storage structure currently have to reproduce the core functionality (parsing TW content and formatting it back). They would be much easier to implement if the core code can be reused (e.g. TWC can be downloaded as a dependency and its methods can be imported). The necessary code bits can be always copy-pasted, but for the future development it's desirable to restructure core code so that it uses exports/imports (so that the code can be reused) and those are stripped when building TW itself (so they don't generate any errors). Maybe I'll try to implement it myself, but currently I can't give any solid time estimations.

среда, 13 августа 2025 г. в 22:58:18 UTC+3, milind...@gmail.com: