A client-side datafolder - Working Prototype

[Arlen Beiler]

I have a working prototype for the client-side data folders. The reason I call them client-side is because they are stored on disk exactly as they appear on the client, and are deserialized directly from disk when requested. 

The client tiddlers are lazy loaded, which TiddlyWiki has supported for several years. The browser side of things literally worked "out of the box", but the server-side took quite a bit of thought and structure to work out the details. 

I've also decided to bring back more of the TiddlyWeb protocol, although I don't know how far I will go with it yet. So the server side supports multiple bags and recipes but the details are not finalized yet. 

It's really simple to test and it should just work. Just download the repo at https://github.com/Arlen22/TW5-storage-plugin/tree/7ef9625eb796e115282b15180262f6a2c6d9cc19 and follow the instructions in the readme. 

I expect to add more features later once I hear from you all what your thoughts are. 

Arlen

[Mohammad]

Hi Arlen,

 Tested on 

• Windows 10 
• Chrome 78 
• TW 5.1.22pre

Works fine!

I tried to install a plugin by drag and drop! I got the red window Internal JavaScript Error.

--Mohammad

[A Gloom]

Will this work for single file TW/local drive or is it node.js?

[Mark S.]

What is the advantage over regular data folders? Or is it just an interim step on the way to something else?

Thanks!

[TonyM]

Arlen

Thanks for experimenting and innovating. Unfortunatly you have used some terminology that was common in the past such as bags and recipes.

A short dummies guide in plain English would help us dummies understand more without having to go looking for definitions. Or perhaps you can reference some doco on these.

If armed I can contribute more.

Thanks
Tony

[Arlen Beiler]

A bag is a group of tiddlers. Currently each TiddlyWiki5 is one bag, for example. A recipe loads a list of bags in order with higher priority overriding lower priority. Saving works in a similar fashion. It's something from the TiddlyWiki classic days that newer users are probably not familiar with but long-time users have mentioned it to me quite a few times. 

The advantages of this format are numerous, but the primary reason for this is that we don't need to load a TiddlyWIki instance on the server. So instead of having a big tiddler store in memory on the server (times x number of wikis when using TiddlyServer), we can keep that all on disk, which saves a lot of memory for large wikis. There are a couple problems I need to check yet related to saving and loading individual tiddlers, but I'm quite excited about the state of things so far. 

The problem this is trying to solve is that we can never assume that a data folder supports loading disk tiddlers directly into the browser. The TiddlyWiki server has configuration options that let it act more like a CMS than a data folder. In other words, the tiddlers could be changed majorly between the client and the disk, or the data folder could contain configuration tiddlers that are only meant for the server and are excluded from the client. Server startup modules could change anything and everything about the way tiddlers are stored on disk, and so on. So we needed to come up with a new format that bypasses that and guarantees that we are looking at a client-side representation of the tiddler without loading the datafolder environment. 

What this then allows us to do is generate an HTML index file which is the TiddlyWiki generated by the server as the page itself, then use whatever mechanism we want to use to lazy-load the rest of the tiddlers into the browser. 

• For TW Cloud, this means we can download a zip of the tiddlyweb folder and load it into the browser directly, then save changes back to Dropbox. You get all the benefits of data folders, without having to load a second "server" instance into the browser to run along side the "client" and translate the tiddlers between the two. That's what I would have had to do to support data folders on TW Cloud. 
• Noteself already uses client-side tiddler storage. Hopefully this concept can be expanded on for multi-user scenarios and the like.
• If it works with TW cloud, it can easily work with most other cloud connector out there, including GitHub. 
• The concept of recipes will make it a lot easier to setup multi-user wikis with shared spaces and user spaces. 
• And the most important of them all: TiddlyWiki client-side data folder handling can be translated into PHP. So we can run this on any shared hosting, and seemlessly transfer it between the PHP and NodeJS realm, because it will be the same data folder. We will still need to generate the initial page using NodeJS, but that's pretty much expected anyway, I think. 

Sorry, I know it's a lot. I hope it's all clear. I'm pretty excited about the possibilities.

Arlen

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/63076535-69c9-45c0-98c5-0d628ddfc2d5%40googlegroups.com.

[A Gloom]

A little bit beyond me but from what I can understand, it sounds very promising

[TonyM]

Arlen,

It seems extremely promising. Do I understand the PHP version would still need a node to load? Do I understand then the PHP would serve as the host/database and the Node loaded wiki would then save back to it? If so I would be keen to build in a checkout method if its not multi-user/access to enable serial editing.

I look forward to helping test, love your enthusiasm

Tony.

On Saturday, December 7, 2019 at 12:29:19 AM UTC+11, Arlen Beiler wrote:

A bag is a group of tiddlers. Currently each TiddlyWiki5 is one bag, for example. A recipe loads a list of bags in order with higher priority overriding lower priority. Saving works in a similar fashion. It's something from the TiddlyWiki classic days that newer users are probably not familiar with but long-time users have mentioned it to me quite a few times. 

The advantages of this format are numerous, but the primary reason for this is that we don't need to load a TiddlyWIki instance on the server. So instead of having a big tiddler store in memory on the server (times x number of wikis when using TiddlyServer), we can keep that all on disk, which saves a lot of memory for large wikis. There are a couple problems I need to check yet related to saving and loading individual tiddlers, but I'm quite excited about the state of things so far. 

The problem this is trying to solve is that we can never assume that a data folder supports loading disk tiddlers directly into the browser. The TiddlyWiki server has configuration options that let it act more like a CMS than a data folder. In other words, the tiddlers could be changed majorly between the client and the disk, or the data folder could contain configuration tiddlers that are only meant for the server and are excluded from the client. Server startup modules could change anything and everything about the way tiddlers are stored on disk, and so on. So we needed to come up with a new format that bypasses that and guarantees that we are looking at a client-side representation of the tiddler without loading the datafolder environment. 

What this then allows us to do is generate an HTML index file which is the TiddlyWiki generated by the server as the page itself, then use whatever mechanism we want to use to lazy-load the rest of the tiddlers into the browser. 

• For TW Cloud, this means we can download a zip of the tiddlyweb folder and load it into the browser directly, then save changes back to Dropbox. You get all the benefits of data folders, without having to load a second "server" instance into the browser to run along side the "client" and translate the tiddlers between the two. That's what I would have had to do to support data folders on TW Cloud. 
• Noteself already uses client-side tiddler storage. Hopefully this concept can be expanded on for multi-user scenarios and the like.
• If it works with TW cloud, it can easily work with most other cloud connector out there, including GitHub. 
• The concept of recipes will make it a lot easier to setup multi-user wikis with shared spaces and user spaces. 
• And the most important of them all: TiddlyWiki client-side data folder handling can be translated into PHP. So we can run this on any shared hosting, and seemlessly transfer it between the PHP and NodeJS realm, because it will be the same data folder. We will still need to generate the initial page using NodeJS, but that's pretty much expected anyway, I think. 

Sorry, I know it's a lot. I hope it's all clear. I'm pretty excited about the possibilities.

Arlen

On Thu, Dec 5, 2019 at 11:44 PM TonyM <anthon...@gmail.com> wrote:

Arlen

Thanks for experimenting and innovating. Unfortunatly you have used some terminology that was common in the past such as bags and recipes.

A short dummies guide in plain English would help us dummies understand more without having to go looking for definitions. Or perhaps you can reference some doco on these.

If armed I can contribute more.

Thanks
Tony

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.