TWX Front end back end

[TonyM]

Dear enthusiasts especially Jeremy,

Having made the move to TW5 over the recent year or so, and sharing the journey towards alternative savers etc... the desire for "federation" and some of my own work in this regard, I had an idea the other night, and I wanted to raise it to see if is is novel and sane or some version of it may be so. It is merely a concept, yet it is based on a lot of what I have seen and used of late ,with TiddlyWiki5. Pleasde provide some feed back or let me know of prior work.

As you know TiddlyWiki is a master of Tiddlers, The various saving methods are about storing tiddlers and Node and NoteSelf store these in database(s) or we can save them in the single HTML model. Tiddlers allow this radical customisation of the user interface along with providing a rich ecosystem on which to build solutions.

TiddlyWiki is in fact both the Front End Gui and the backend server in one. In fact in some ways the server based implementations do in some ways separate the front and back already, Bob does this even more so with messages to the server.

What if TiddlyWiki X (an imagined future version) took this a lot more seriously?

Imagine if every tiddlywiki could act as a back-end to any other tiddlywiki, this would allow one tiddly wiki to store its tiddlers in one or more other tiddlywikis. Or another TiddlyWiki may host more than one other TiddlyWiki's tiddlers for it.

That is the front and backend functions could be sub/divided internally and externally.

Some may immediately see the potential of this but here are a few examples;

• We could add security (authentication) the the connections to other wikis rather than for each tiddler. If you have not authenticated you cant see the tiddlers being served.
• Maybe every tiddler is owned by a wiki and only it can change that tiddler however the owner can pass ownership to another wiki. If designed correctly the ownership could be passed back. The receiving wiki could choose if it wants to accept or reject the tiddler. This may be different if its a macro, plugin or style-sheet.
• Otherwise a wiki can ask another to store its tiddlers for it, either with an encryption key or without thus permitting access or denial of access of the tiddlers by the containing wiki.
  
• Containing wikis can consolidate tiddlers from multiple wikis, and display them as well.
• A wiki can choose to use an external wikis tiddlers through a pseudo import process and the external wiki owns and can update them if permitted.
• One TiddlyWiki may host tiddlers on behalf of another wiki through an encrypted exchange, but may even host tiddlers in an encrypted form such that the server Wiki cant even look inside the tiddler, just serve them.
• Other front ends could be designed to interrogate the tiddler server, like display a single tiddler content etc...

In this model there would be no contention for tiddlers, as the ownership exchange process handles this however we could pass a tiddlers ownership to a server, which can be set up to respond with actions to transform the tiddler and return it or another back to the original owner.

Regards

Tony

[Jed Carty]

Before I start asking questions I want to say that all of these are good ideas and I want to implement them.

When you say that every tiddlywiki could act as the backend of another wiki I am not sure what you mean. The node process that serves a wiki can server other wikis as well, Bob already does this with one node process and TiddlyServer does it by spawning new processes for each wiki.

This whole idea sounds a lot like federated tiddler servers, which can (and in terms of coding complexity and maintainability probably should) be created separately from tiddlywiki (I think that the back-end of noteself is an example of something similar to this already).

I think that this would actually be a separate project from the core of tiddlywiki. The public-facing wikis I have up using Bob right now have a custom expressjs backend that does some of what you are describing, and I am planning on adding the federation aspects (storing tiddlers for other wikis, aggregation, etc.) to it.

Tiddlywiki works well as a single html file and adding all of the overhead needed to do what you are describing would make that a lot harder. So while these things should be available for tiddlywiki I don't think that they should be part of the core of tiddlywiki.

[TonyM]

Jed et al

Thanks for your critical analysis.

I do not disagree with you in any substantial way. If, as we do today, alternating the save mechanisium over different platforms, I believe it can be done, and in fact I belive this idea in part stems from inspiration from bob. Bob offers an even tighter solution allowing shared tiddlers without a wiki owning tiddlers. A network of wikis on top of node or bob offers a very powerful solution to share and federate, yet I like you, value maintaining the single file model indefinatly.

It seems to me permitting node, bob and nodejs to also manage the sharing of tiddlers as in my suggested model, effectivly at a logicaly higher level of abstraction, where tiddlers are owned, can be stored elsewhere, but whos ownership can be relinquished to another wiki and/or backend, as bob would do very well, would allow even more loosly coupled networks of wikis. As I pointed out one wiki can relinquish ownership to another which may transform it and return ownership to the first. In this model there should be no contention at all.

Critical also is this interwiki connection can have security or encryption applied at the connection not per tiddler, because the ownership model will handle that.

The abstraction that any wiki can be the front end and backend (a single file wiki) or have multiple back ends (tiddler stores) or conversly be a front end to multiple tiddler stores, which node and bob can already excel at, is in some ways at a higher logical level above the concerns you express.

There is little doubt such a solution would demand some accomodation in the core, but this need only be the facility and the specific details can remain in a combination of savers, optional key plugins or other (external) server components such as node, couch and pouch currently do. What is needed is tiddlywiki to recognise ownership of its own tiddlers where ever they reside, and respect tiddlers it does not own (as read only).

The conseptual leap that tiddlywiki has already taken to be a form of quine where its user interface is totaly tweekable would be extended into its very storage mechanisiums itself. Thus we can say a tiddlywiki can be front and back end and any combination there of.

One mechanisium I belive we do not yet have is the ability to include a single tiddler in one tiddlywiki found in another tiddlywiki without it existing in the same node network. Bobs messaging could facilitate this but we also want such tiddlers to be available at any url, file or in an addressable wiki found elsewhere.

These ideas come from expierience as a cross diciplinary solutions designer, whilst my skill set is wide, and deep in many places it depends on a team or community behind it. It needs others, specialists, experts and diverse inputs. I recognise our community or team is wider and deeper than any individual can ever be. It is this I dream of engaging, and it can only evolve from communication.

I hope this makes the ideas I am trying to express clearer, and utlimately propel tiddlywiki into the stratosphere, where it is already headed whilst maintaining its place in the two worlds of developers and users. Few users will understand database models, transactions or contention but they could understand how tiddlers can be owned and shared.

I love tiddlywikis current capabilities but also its potentials.

Regards
Tony

[Jeremy Ruston]

Hi Tony

Sorry for picking this up so late.

TiddlyWiki is in fact both the Front End Gui and the backend server in one. In fact in some ways the server based implementations do in some ways separate the front and back already, Bob does this even more so with messages to the server.

To be clear, if you run TW5 under Node.js as an HTTP server then there are two distinct instances of the same TW5 code, one running in the browser, and one running on the server. They speak to each other via HTTP, and of course Bob adds a websockets channel of communication.

What if TiddlyWiki X (an imagined future version) took this a lot more seriously?

Imagine if every tiddlywiki could act as a back-end to any other tiddlywiki, this would allow one tiddly wiki to store its tiddlers in one or more other tiddlywikis. Or another TiddlyWiki may host more than one other TiddlyWiki's tiddlers for it.

This is already possible. There is a well defined API through which the browser instance talks to the server instance, and there's nothing stopping experimenting with the browser calling multiple serverside APIs, nor with the server instance of TW5 itself using the API to speak to another instance.

That is the front and backend functions could be sub/divided internally and externally.

I think we have already exactly that level of encapsulation between the two parts.

Some may immediately see the potential of this but here are a few examples;

• We could add security (authentication) the the connections to other wikis rather than for each tiddler. If you have not authenticated you cant see the tiddlers being served.

The API does already support wiki-based security (it's enhanced in the latest v5.1.18). I've no current plans to explore tiddler-based authentication, but it remains open to investigation by others.

• Maybe every tiddler is owned by a wiki and only it can change that tiddler however the owner can pass ownership to another wiki. If designed correctly the ownership could be passed back. The receiving wiki could choose if it wants to accept or reject the tiddler. This may be different if its a macro, plugin or style-sheet.

That's pretty much how the API works: tiddlers get synced back to the server that they came from.

• Otherwise a wiki can ask another to store its tiddlers for it, either with an encryption key or without thus permitting access or denial of access of the tiddlers by the containing wiki.
  

That would be an enhancement to the API as things stand. As it happens I try to keep encryption and security components outside TW5 so that it is possible to re-use proven, off-the-shelf implementations, but it's still an interesting area to explore.

• Containing wikis can consolidate tiddlers from multiple wikis, and display them as well.

• A wiki can choose to use an external wikis tiddlers through a pseudo import process and the external wiki owns and can update them if permitted.

• One TiddlyWiki may host tiddlers on behalf of another wiki through an encrypted exchange, but may even host tiddlers in an encrypted form such that the server Wiki cant even look inside the tiddler, just serve them.
• Other front ends could be designed to interrogate the tiddler server, like display a single tiddler content etc...

These are the sort of features that could be implemented by the server side instance of TW5 invoking the API to talk to a different server.

In this model there would be no contention for tiddlers, as the ownership exchange process handles this however we could pass a tiddlers ownership to a server, which can be set up to respond with actions to transform the tiddler and return it or another back to the original owner.

Yes, this is exactly the sort of thing that would be interesting to explore. As I say, none of this is really about TWX. TWX is about revisiting design decisions that are too late to change in TW5. The classic example is that in TWX I want to explore a slightly different data model for tiddlers that unifies singleton fields with list fields. There's no way we can change something as fundamental in the context of TW5. But there's lots of other interesting experiments that can be done with TW5: we can literally take it apart and rebuild it into multiple different things (as the recent work on externalising the JS showed).

Best wishes

Jeremy

Regards

Tony

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/712f0a17-f1d4-4c94-8ddc-9789ab6593dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[TonyM]

Jeremy,

Responding to such a question at all are great, no need to be concerned about the time it takes to respond on larger speculative  ideas.

I now understand this is not a TWX issue. 

Very exciting your response, it sounds quite do able if not already done. I suppose the trick here is to convert this design model into something that can be easily communicated, well documented, a little testing and inviting feedback.

I am keen to make this a reality, I can put in the effort to do so, however how should I gain the support or help where I do not have the skill?. It is a project, that needs a vision to guide it as the end result aims to be an easy conceptual model that experts and newbys alike can apply. I need a project framework of sorts around it, and acceptance? to promote the vision.

I look forward to exploring the 5.1.18 auth changes

So would I be correct in thinking that we can/or will be able demand login to edit a wiki (the front end) that also carry's a key or other credentials we could use to authenticate to one or more backends, an example may be login, then when reviewing history tiddlers we have to connect to a history wiki, unless I have the wiki credentials I can not see the tiddlers served by the history wiki? In someways this is an intermediate between secure wikis and secure tiddlers, in effect a secure groups of tiddlers (as served by a wiki), The advantage being I can remove access to select users on the front end wiki at any time, without stopping them access the front end?

If the above were so, I would be happy to build a tiddlerlevel process from this to do (as mentioned before) because I would have the infrastructure I need to do so.

In this model there would be no contention for tiddlers, as the ownership exchange process handles this however we could pass a tiddlers ownership to a server, which can be set up to respond with actions to transform the tiddler and return it or another back to the original owner.

Thanks again

Tony