syncadaptor design for sqlite

[Abilio Marques]

Hi,

As a long time user of tiddlywiki, I adapted the classic version into a server version using sqlite and nodejs a couple of years ago. I never published the code as I think it was more of a hack than real software. After tiddlywiki5 was published, I found that it already has a server running in nodejs. Now I want to add the sqlite storage as an alternative to plain text files.

Been trying to create a syncadaptor plugin for sqlite. Read these before I started:

http://tiddlywiki.com/dev/#Syncadaptor
http://tiddlywiki.com/dev/#SyncAdaptorModules

It made me believe that it was possible.

Then, as a quick test, I went into the directory called, tiddlywiki/plugins and copied the filesystem directory into a new one called sqlite. Refactored the entire code to replace "filesystem" by "sqlite".

Created a test wiki with the --init server parameter, then modified the plugins list inside tiddlywiki.info accordingly (replaced filesystem with sqlite).

Tested it and still ran... Then tried a few console logs, and it printed text when saving or deleting a tiddler, but... loadTiddler didn't run. I was suspicious from the start, as it runs a dummy callback and the comments read:

We don't need to implement loading for the file system adaptor, because all the tiddler files will have been loaded during the boot process

Tried to find where, and I must say I got lost in the code... Would anyone please point me in the right direction on how to make the plugin be the responsible for loading all the tiddlers? (in other words, allow loadTiddler to run)

This time, I want to release my code, so I don't want to solve it in a "replace this code in the core" fashion.


Thanks in advance,
Abilio

[Astrid Elocson]

Hi Abilio,

The code for loading tiddler files under Node.js is in /boot/boot.js.

That file contains a $tw.boot.startup function. One of the many things it does is call $tw.loadTiddlersNode. That in turn calls $tw.loadWikiTiddlers, which performs a recursive exploration of the folder structure and calls $tw.wiki.addTiddler for each tiddler it finds.

$tw.boot.startup is called by $tw.boot.boot, which is called from tiddlywiki.js (in the repository's root folder) when you launch the Node.js server.

Apologies if I'm telling you things you know already.

– æ

[BJ]

Hi Abilio,
when runninng a tiddlywiki with node.js, a complete tiddlywiki html file is built and then served, it includes all the tiddlers.
It is a two stage process, first tiddlywiki(nodejs) builds itself from the directories that contains the tiddlers as files. Then depending upon which 'edition' is specified different plugins are included (controlled by tiddlywiki.info as you probably know). see also the editions directory for tw5.com-server/tiddlers/ which is where the 'user' tiddlers are loaded from with the server edition.
The nodejs instance of tw will then will build the html file using a set of templates.
There is a template that is used for the store, look for the line
<$list filter=<<saveTiddlerFilter>> template="$:/core/templates/

html-div-tiddler"/>
in the source files - this is creating the text of tiddlers for the 'store' area of the html file.

If you are only interested in having 'user' tiddlers in the database you could replace this with a new widget to extract the tiddlers from your database.

Cheers

BJ

[Abilio Marques]

Recomended way to intercept $tw.loadWikiTiddlers from inside a plugin?

[Abilio Marques]

BJ,

That's pretty much close to the solution (hack) I gave to the classic wiki a few years ago. Back then I was a bit worried about mobile access (yeah, I use my wiki everywhere) because each time the wiki loaded (and I have several wikis) the entire html was served. I went into dissecting the stock HTML and putting the entire javascript in a separate file. That way it was cached by the browser, and only a small(er) HTML file was transferred, containing only the tiddlers.

I've seen tiddlywiki 5 boasting 1.5 Mb of data transfer for an empty wiki. That's quite a lot even nowadays. Refreshing my wiki 10 times a day, it means 450 Mb at the end of the month (ouch).

I now have around 6 different wikis (I use them as notebooks for different subjects). And the url is used internally to select the database. I even have a coarse access control list (for the entire wiki), so some users can access some specific wikis.

I wish I can add all these features into tiddlywiki 5. That will be the only way for me to switch. I still don't understand much of the massive piece of code the new tiddlywiki has become.

I want to keep as much of the original code as possible (both in the server and the client side). I really like your idea, as long as I'm able to intercept load, save, and delete. I would also like to expose the functions from the same http server that is already running.

But I have a doubt, what will I loose if I leave the system tiddlers to be loaded at start? Would I gain something if I delay the load of specific system tiddlers until needed? I'm really new to tw5.


I know tiddlywiki tries to store everything in a single file, but that only seems useful when you're editing local (the only way available at tw genesis). For a server version it has no use, and actually, it makes it harder to cache. Would it be too hard to modify the code so it will serve a few javascripts and the HTML?

[Tobias Beer]

I've seen tiddlywiki 5 boasting 1.5 Mb of data transfer for an empty wiki. That's quite a lot even nowadays. Refreshing my wiki 10 times a day, it means 450 Mb at the end of the month (ouch).

I also think that's an ouch and it would be a worthwhile undertaking to have the TiddlyWiki core act more like a scaffolding into which the actual data, as well as themes and plugins are loaded through whichever channel. This way, browsers could more efficiently cache things and not have you round-trip a few mb each time you wish to add and save a few bytes.

Best wishes, Tobias.

[Tobias Beer]

a worthwhile undertaking to have the TiddlyWiki core act more like a scaffolding into which the actual data, as well as themes and plugins are loaded through whichever channel

Skipping the singlefile paradigm for a moment, different folders in a repo, even remote bags served via some tiddlyweb instance could either be pushed into TiddlyWiki by referencing external, individually cached, json collections of tiddlers, or pulled via XmlHttpRequests of much the same.

So, the core would mostly be just that, the core of version x, everything else referenced one way or the other.

Best wishes, Tobias.

[Dániel Molnár]

Hi,

can you please post your hack/code?

I am reallly excited to see it!

Daniel

[Tobias Beer]

Fwiw,

I don't think the filesystem adapter is what you're after with sqlite. It's not that you want to build a single tw on the fly from a sqlite db but only during startup — as the nodejs --build command would do with those *.tid files — only to forget all about it until you hit save to then dump all tiddlers into the same local db, effectively overwriting the entire db on every TiddlyWiki save.

In other words, SyncAdapter possibly means: individual (lazy?!?) loading of (skinny) tiddlers on startup and storing changes to tiddlers "on demand", i.e. individually, whence changed. One way where a db could majorly improve things is if skinny tiddlers where batched up into say buckets (or bags as they're called in TiddlyWeb) and whenever I would first access a tiddler from the bag I'd have the option to either load that tiddler only (via the db connector) or the entire bag so as to speed things up later.

The bag concept would help to more effectively manage batched up chunks of data that are of interest to me in a given section ...which could actually be spread a across different sources / connected via different connectors. So I could have a FAQ bag that would only be loaded if anyone bothered to access a FAQ tiddlers.

I could even have bags that are not loaded at all on startup, not even skinny... and if I was opening that "not-existing" tiddler, my sync adapter could notice that there is such a tiddler in bags x y z and ask me if it should load one of these instead... and indicate the source bag and connection it all came from.

From what I see, there is a bit of an architectural disconnect between the concept of an edition as we see it in TW5 and that of bags in TiddlyWeb, which makes it all the harder to conceptualize a more advanced, well, server architecture. And to me that's really what it is.

I envision TiddlyWiki to allow me to define firstly:

• a number of (server) connectors (or sync adapters) and to introspect and configure a number of either
• simple bags
• or more advanced editions
  
  ...to pull from and to configure as to how I want them accessed, e.g.:
• fully loaded on startup
• lazily loaded
• not loaded but queried for when missing
• there could actually be a querying mechanism by which TiddlyWiki would poll connector for versions of a given tiddler and allow you to load those alternatives / variants instead or even compare, perhaps be notified of changes that way

As for the "ServerMechanism", the development documentation unfortunately leaves us with a bit of a blank.

Best wishes,

Tobias.