TWX or sooner - User overlay namespace mapping

[TonyM]

Folks,

Please take this suggestion seriously. I think it could resolve some serious multi-user requirements. Please also forgive me for my speculation and dreams.

Given a number of multi-user scenarios and the ability to selectively save tiddlers to a server;

I was wondering if it may be practical to have a namespace eg $:/username/tiddlername such the the active user creating or saving tiddlers will actually write to their namespace only. When the wiki loads the tiddlers in the namespace are loaded/reloaded rather than the tiddlernames they replace. For all intents and purposes it would look like they are using "tiddlername" however this will actually be saved as S:/username/tiddlername

The save mechanism can then be set to only save their name space tiddlers to the server, meaning they can not overwrite the servers shared, core or system tiddlers (but they can overide them with a replacement version). It is a bit like how the local-storage plugin works loading over the existing tiddlers, loading the users modified tiddlers from local-storage.

A version of this for single file wikis could permit the users namespace tiddlers to operate in local storage, and be saved/exported out for re-import so that they may be loaded back into local-storage. Well designed this would permit an independant save process (an alternate filename such as username.html) making this a possible feature of single file wikis as well. 

In both server and singlefile configurations username.html could also be imported into a master server wiki to enable a loosely coupled multi-user environment. Tw-Receiver or other save mechanisms php or auth could then control access to the username.html, so only the authorised user can load/save or export the whole wiki with there namespace tiddlers applied or stored independently.

Additional features could be introduced so the owner of the server can import the user name space tiddlers, review and selectively replace the server versions of the tiddlers. Thus users can submit proposed changes in their own name space for selective inclusion in the master namespace. Tiddlers in the user namespace could be tagged "pull request" so they appear as change requests to the master namespace.

I have no certainty what complexity is involved in what I propose but given how the local storage plugin now works my guess is the ability and features are only a small step away.

PS I know some may say that TW-Federation or similar already does this. If it is the case please publish it, whenever I look at tw federation and similar solutions, I get lost in developer discussions and can't find an actual solution, or go beyond some betas.

Thanks for considering this seriously

Tony 

[TonyM]

Folks,

Just a little more. This discussion is what triggered this idea, trying to use cardio as a multi-user solution.

• An important point here is my proposed solution would allow existing TiddlyWiki's to be quickly converted to multi-user solutions.
• I would also add this would be a candidate for a plugin, if it can work on top of the core, rather than needing to be in it. Thus it could be in a 5.1.X version rather than TWX
• Although using such a namespace method would help with multiple users, it could also be redeployed for other interesting applications, where by users can change namespace on the fly.
• For example change namespace and you are looking at one edition, change and you can see another edition and download the custom edition you want. ie an editions name space, that only contains the delta between editions.
• This may overcome performance issues by being able to load only the required namespace, so you could have a database for each team, you select the team namespace and only that is contained in memory, yet you still remain in the same tiddlywiki. You could load a minimal wiki, then on selecting the namespace, load the balance ONLY
• Under node we could publish namespace tiddlers at \namespacefolder\tiddlername making them visible (if desired) to all (authenticated) users. Ideally the user can control what is published using a filter. This could effectively treat such tiddlers as external tiddlers, and could even exist on another server "other.com\namespacefolder\tiddlername". These could be published as static tiddlers or as actual tid files that the server can render.

Perhaps another flight of fantasy?

Regards

Tony

[William Armstrong]

This looks decent from the user perspective. 

TW has the ability to deny the ability to read and/or write to the wiki when served on node, but does it have this ability on a per tiddler basis? That is a separate feature you kind or hint at or just gloss over. 

Otherwise, this doesn't look much different than serving several wikis like Bob currently does. With the added benefit to being able to search through all the wikis.

So, breaking down the feature requests, we can say:

1. Add ability for TW to choose to save to multiple subdirectories. It is already able to save to the Files directory in addition to the Tiddlers Directory, but with the specific use case you are talking about, you now need to interface user names to the filesystem. I don't know enough about what usernames are limited to in TW, but the FS definitely has limitations. Older FS default to other than UTF8 encoding, and EXT can support different ones depending on the server. I don't know what the situation is on Windows. Node will probably handle that, but there will be cases where the User Name and the Directory don't match. Also, if the Username is a First Name and Last Name, should TW make one folder, two folders First Name first or two folders Last Name first? I tend to favor the last, which can reduce the number of folders in the top level directory.
2. Add the ability to have r/w permissions on the Folders. This is the simplest method, and give nobody the right to edit the main Tiddler top level. This gives you all the benefits, without having to have a canonical source for all wisdom, the top level tiddlers.
3. Additionally, add r/w permissions to tiddlers. This should not be too difficult, in terms of implementation. Each tiddler would get a hidden field with predetermined values. In the wiki model, we could take a page from Facebook and include group permissions, which would allow for, say, adding receipts that are viewable from anyone in the accounting group, or if writing articles, anyone in the editor group. That makes it a set of fields, however.
4. The ability to version control the tiddlers.
5. The ability to diff the tiddlers.
6. A UI for those functions.
7. Extrapolating: The ability to modify username on import. Possibly the ability to change multiple usernames at import.
8. Create namespacing for tiddlers. I have no idea if this is in TW currently. If this were done, I can foresee a lot of performance use cases in large wikis.
1. You could could namespace each connection to the server, and only serve the tiddlers necessary for the current view. This is almost certainly needed if you have 100 people using the same wiki. Other wise, your bandwidth cost will grow extremely quickly out of control. This would force most searches be done server side.

Alright, see if this is close to the idea you are expressing? Correct anything that is wrong with it, or add to it.

Thanks William.

[TonyM]

William - thanks for responding.

 

TW has the ability to deny the ability to read and/or write to the wiki when served on node, but does it have this ability on a per tiddler basis? That is a separate feature you kind or hint at or just gloss over. 

Access to the wiki needs to be read/write, however a filter can be applied to the save process to only permit $:/namespace/* tiddlers to be saved. A bad actor could defeat this of course. So in effect it does have this ability on a per tiddler basis.

 

Otherwise, this doesn't look much different than serving several wikis like Bob currently does. With the added benefit to being able to search through all the wikis.

Bob place may overlap with this, but this is much simpler and can also add value in single file tiddlywiki cases. I believe there is a way to search through all wikis with Bob. It is not serving several wikis it is serving one and a number of name spaces.

 

So, breaking down the feature requests, we can say:

1. Add ability for TW to choose to save to multiple subdirectories. It is already able to save to the Files directory in addition to the Tiddlers Directory, but with the specific use case you are talking about, you now need to interface user names to the filesystem. I don't know enough about what usernames are limited to in TW, but the FS definitely has limitations. Older FS default to other than UTF8 encoding, and EXT can support different ones depending on the server. I don't know what the situation is on Windows. Node will probably handle that, but there will be cases where the User Name and the Directory don't match. Also, if the Username is a First Name and Last Name, should TW make one folder, two folders First Name first or two folders Last Name first? I tend to favor the last, which can reduce the number of folders in the top level directory.

I don't think I am asking for "Add ability for TW to choose to save to multiple subdirectories", I only ask that with permissions available we can save/download a subset of tiddlers to another file (on the server or locally), just as we currently write to tiddlywiki.html. We can ensure usernames are unique, and even store a hashed filename to save with. eg; in the user name tiddler <<qualify filename>> The folders I suggested were only "virtual" folders, that appear to be there just as mydomean.com/tiddlername appears to be a static tiddler.

 

1. Add the ability to have r/w permissions on the Folders. This is the simplest method, and give nobody the right to edit the main Tiddler top level. This gives you all the benefits, without having to have a canonical source for all wisdom, the top level tiddlers.

In this case perhaps we could use the server to allow/disallow saving the tiddlers to the logged in user name space, but not necessary with trusted users.

1. Additionally, add r/w permissions to tiddlers. This should not be too difficult, in terms of implementation. Each tiddler would get a hidden field with predetermined values. In the wiki model, we could take a page from Facebook and include group permissions, which would allow for, say, adding receipts that are viewable from anyone in the accounting group, or if writing articles, anyone in the editor group. That makes it a set of fields, however.

Interesting but I my suggested solution unnecessary.

 

1. The ability to version control the tiddlers.

Some solutions already exist and should operate independently. 

1. The ability to diff the tiddlers.

Easy with cross namespace tiddlers.

1. A UI for those functions.

Yes

 

1. Extrapolating: The ability to modify username on import. Possibly the ability to change multiple usernames at import.

Yes, but not always necessary if the import is into a namespace, or change control copies an approved tiddler over a master tiddler. 

1. Create namespacing for tiddlers. I have no idea if this is in TW currently. If this were done, I can foresee a lot of performance use cases in large wikis.
1. You could could namespace each connection to the server, and only serve the tiddlers necessary for the current view. This is almost certainly needed if you have 100 people using the same wiki. Other wise, your bandwidth cost will grow extremely quickly out of control. This would force most searches be done server side.

I am not sure about performance issues here. Only the total size of wiki will be an issue and 100,000s of tiddlers are already known to work well. If only one name space is loaded at a time this will give a performance gain.

 

Alright, see if this is close to the idea you are expressing? Correct anything that is wrong with it, or add to it.

Done, Thanks for your contribution.

Tony

[PMario]

Hi Tony,

What you describe, is basically the concept of TiddlyWeb, with different names. ... TiddlyWeb had more security related options.

The "namespace" you describe is named "bags" in TWeb. A bag is a tiddler field, so it doesn't interfere with the title namespace. ... User read / write access was configured "per bag". 

In TiddlySpace (TS) several filtered bags could be combined into a "space", which is similar to your "filtered namespaces".

... BUT TiddlySpace is dead. ...

It suffered from the "centralisation problem" that we have in the net of today. The users data doesn't belong to the user any-more, because it's stored in data-silos, that are controlled by 3rd parties. If a centralized server is shut down the whole "community" is destroyed. ... That's exactly, what happened to TS.

At the moment I'm having a closer look at the DAT-project [1] and the beaker-browser [2], which I think is a very promising peer-to-peer (p2p) platform. Also for multi user configurations. ... I think TW is a perfect match here.

have fun!

mario

[1] https://dat.foundation/

[2] https://beakerbrowser.com/

[TonyM]

Mario, 

Thanks for your comments.

Because I never understood TiddlyWeb I can't say weather my proposal is like it or not. However in some ways what I propose it is quite simple. In fact I now believe I can make a working prototype.

• What I propose here is not about a cloud based server, or service online, it is about a multi-user configuration.
• It is Just like the local storage plugin saves tiddlers to local storage and on reload it applies them on top of what was loaded from the server. 
• This idea would save the users modified tiddler under a prefix or compacted into a json tiddler. The save to server process will be configured to only save the current users "prefixed" tiddlers back to the server.
• I am simply suggesting a similar mechanism that rather than save to local storage, renames the modified tiddlers with a username based prefixes and saves them in the wiki.  At reload it determines the users prefix and "restores" the tiddlers over those loaded from the server. 

However the reason I raise it here is such a mechanism provided in a plugin, a saver or parts in the core would allow the normal save filters and methods to work. Not to mention may be needed to facilitate it for single file wikis.

Single File Wikis

For Single file wikis a similar mechanism would save the users modified tiddlers to a seperate file of tiddlers. When the single file wiki is loaded, it will at the very last step load the users modified tiddlers over the ones loaded from the file. In this model the only thing that changes is the content of the users tiddlers which are saved separately.

Advanced security

In the "single file Wiki" configuration that saves users modified tiddlers into a separate file, implementations such as on PHP with tw-receiver, we will gain the ability to read or write the users modified tiddlers file, and these files could have user and password security applied. Unless I am logged in as fred I can not save to wikiname-fred.json or read from it.

Regards

Tony

[TonyM]

I must add something I have not clarified

Once we have multiple users enabled with their tiddlers stored independently but available we can manage these tiddlers such as saving tiddlers into the server if there is no conflicts, and providing version or change control to manage conflicts in the less common case. Some of these further conflicts can be minimised such as by saving $:/temp into local storage.

Regards

Tony

[William Armstrong]

You could also use other data to namespace for conflict resolution. As an example, tags could be prefixed to avoid ambiguity. Even in the case of two tiddlers with the exact same tag, you could simply switch the order to differentiate them. If they both only had one tag, you could special case a postfix option. TW would then need to remove any of the tags as pre or postfix.

[TonyM]

William.

For me the key is to permit multiple users to to use the same wiki but once this is possible, the designer needs to choose how and when they interact. Keep in mind if you are in the wiki my tiddlers are not loaded over the main wiki, yours are. If they are in a json or external files they will not influence your tags or macros. However we could build solutions to view selected content from other users.

Regards
Tony

[PMario]

On Tuesday, June 25, 2019 at 11:24:08 AM UTC+2, TonyM wrote:

...

For me the key is to permit multiple users to to use the same wiki but once this is possible, the designer needs to choose how and when they interact.

I think we don't need to allow multiple users to write to 1 wiki store. We can "stack" existing wikies. ... For those wikies, where I do have write access I can change them. ... For wikis, where I don't have write access, I either need to "fork" the whole wiki and make it mine, or I need to overwrite single tiddlers and save them into my own "space".

As soon as you start to store username, passwords and write access information on a server you are doomed. ... I think the underlying system needs to deal with authentication and authorisation. 

 

Keep in mind if you are in the wiki my tiddlers are not loaded over the main wiki, yours are.

I think that's the key here. ... Everyone is the owner of his/her own content. That's important!

 

If they are in a json or external files they will not influence your tags or macros. However we could build solutions to view selected content from other users.

Right. TW already has a mechanism to resolve these problems. ... shadow tiddlers ... Everything, that is loaded from a wiki, where I don't have write access is a shadow tiddler. ... Which in turn I can overwrite and save into my "namespace"

[TonyM]

Mario,

Thanks for your comments.

This idea of stacking wikis is very interesting and should be pursued. There are cases though where such a ridged separation of user content is not as desirable and I think we can permit more than one user writing to a single wiki store, especially when it's only permitted to part of the wiki.

I agree authentication and authorisation is critical but once someone is entitled to update content there is a lot we can do in trusted environments. Solutions should cater for both open and locked down cases.

I think everyone can be the owner of his/her own content. Yet there are many cases where people are happy to share there content with other users and even have it become part of the shared wiki. An authorised curator should be able to commit content to the main wiki contributed by any user. It could work a little like github.

I like your suggestion about using the shadow tiddler mechanisium. Sometimes I do not want to overide someone's content with my own by accident as I want them to maintain the source of truth. A way to denigh shadow overwrites is also needed

Regards
Tony

[PMario]

On Tuesday, June 25, 2019 at 2:07:28 PM UTC+2, TonyM wrote:

....

I like your suggestion about using the shadow tiddler mechanisium. Sometimes I do not want to overide someone's content with my own by accident as I want them to maintainthe source of truth. A way to denigh shadow overwrites is also needed

Yea, shadow tiddlers are not really overwritten. They are replaced with "user tiddlers" in the users "space". If the user tiddler is removed, the shadow tiddler takes over again. ...

What we need here is, is a possibility to make it more visible to the reader, that there may be a different version of the tiddler. ...

-m

[PMario]

On Tuesday, June 25, 2019 at 2:07:28 PM UTC+2, TonyM wrote:

...

I agree authentication and authorisation is critical but once someone is entitled to update content there is a lot we can do in trusted environments. Solutions should cater for both open and locked down cases.

That's why I do like the dat-project [1] a lot. DAT is a protocol, to share data between computers. DAT is also a storage mechanism, where you have write access, if you own the "private key".

You do have read access, if you know the "public key". .. The public key is unique and part of the URL address.

eg: dat://32d225818f3928d4f17ed4893108f630d59023ccbbda196262ecd936e4033421/

Which is also reachable as: dat://beakerbrowser.com with the beaker browser or https://beakerbrowser.com/ with every other browser.

If we define the dat-address as a "namespace" in your terms, or a "bag" in my terms, we do have the ability to define multi user writeable "spaces", with a simple configuration. In this scenario everyone has the right to write to their own namespace.

As you can see, the wiki is now scattered all over the network. ... That's a problem, if you don't have a connection. ... but ...

DAT is a distributed system [2], which relies on the fact that other users also store the content. If you host someone elses content, the mechanism is called "seeding".

For seeded content, you only have read access, since you don't know the write key. So as an author you can be sure, that others only seed / host your content, but they can't modify it. The advantage now is, the more often some content is seeded, the more accessible it is. Everyone that seeds some content, has offline access, which makes it a perfect fit for "low connectivity".

If you want to modify my content, you need to "make a copy". The copy gets a new and unique dat-address. Now you own it. ... A different version of it.

If you only copy single tiddlers from my namespace to yours, it's the same mechanism as a overwritten shadow tiddler in TW terms. ... So every bag, where I have read access contains shadow tiddlers. .... The only thing we need to do is, make it more visible.

DAT storage is an "append only log". That means, there is automatic version history built into the system. The storage can be mapped into the real OS file system. So we still have our beloved files, we can deal with.

If we modify the local files, we can create diffs between the dat-storage and the actual version in the file system. ... If the local version is ready to be published, we can write it to the store and it is redistributed to everyone, which seeds the content [3].

... That's just the tip of the iceberg. If you follow the links, you'll get more info.

The "problem" with DAT is, that it is relatively new [6]. The storage back-end seems to be stable, but the rest of the ecosystem is still in flux. Especially the beaker browser, which changes it's face with every major version. ... :/

TiddlyWiki already includes a dat-saver [4] since V5.1.15, which can handle single file wikis. The saver still works with 5.1.19 and the latest beaker browser.  ...

I do have an outdated dat-adaptor prototype [5], to work with single tiddlers. The videos are about 1 year old and I did stop development, since the API used in the video was experimental and deprecated at that time already. ... I would have to have a new look.

have fun!

mario

[1] https://docs.datproject.org/docs/intro

[2] https://docs.datproject.org/docs/terms#distributed-web

[3] https://docs.datproject.org/docs/concepts#distributed-network

[4] https://www.youtube.com/watch?v=YZxWZ0LAHWI

[5] https://www.youtube.com/watch?v=G9fDICizGIM

[6] https://en.wikipedia.org/wiki/Dat_(software)

[Hans Wobbe]

Thanks for sharing this DAT information, Mario. 

In some ways, it seems like a lite-weight version of a blockchain. 

I may be able to structure an evaluation exercise for this fall.  If so, I'll make sure the opinions are made available here.

Regards,

Hans

[TonyM]

Mario,

I applaud your suggestion looking at data tiddlers and have experimented in that space as well. 

I do want however to ensure the the Original Thread is pursued. Ideally the same mechanisms will be used in the various alternative scenarios.

In fact I have learned there are a large number of factors that influence, not just the multi-user approach but also the choice of single, file, server and hosting.

• As a result I plan to collect all these factors and see if I can develop a decision tree to deal with this complexity. 
• From there it will be possible to get a strategic approach to developing components and methods across all the possibilities of TiddlyWiki.
• Here is a quick list of possibilities that highlight what may influence choices, by its preliminary nature you can see it is not separating the solution from the user requirements.
• I often find someone is asking for multiple users but they only have one editor and an indefinite number of readers which is the out of the box behaviour anyway

Note: All of the below may exist in trusted, untrusted access environments. How secure should the wiki (restorable) or totally tracked or locked down

1. Remote Read only tiddly reference on your computer (sourced elsewhere)
2. Local Read only tiddly reference on your computer (sourced elsewhere)
1. With or without install and file management access
2. With or without application install access
3. With or without browser choice
3. Read update wiki just for one on their computer
1. With or without install and file management access
2. With or without application install access
3. With or without browser choice
4. Read update wiki just for one on their computer and their Intranet
1. With or without install and file management access
2. With or without application install access
3. With or without browser choice
5. Read update wiki just for one on their computer and via other devices when out of home
1. Transfer updates on return or before departure
2. Remote access to Wiki Host
3. Cloud storage
4. Cloud database
6. A wiki authored on a desktop LAN or Intranet and published on the intranet (read multiuser)
1. for read only LAN public
2. for read only LAN user to those with a secure password
3. read only to LAN users those with a secure Encryption key
7. A wiki authored on a desktop and published on the internet (read multiuser)
1. for read only internet public
2. for read only internet to those with a secure password
3. read only internet  to those with a secure Encryption key
8. A wiki authored on a desktop and published on a shared cloud drive (read multiuser)
1. for read only public
2. for read only to those with a secure password
3. read only to those with a secure Encryption key
9. A wiki authored/modified on the internet or Intranet by one editor and published on the internet for public read only
1. but including local storage for their customisations and view, most recently viewed etc..
2. but including the ability to export comments and other tiddlers, for import by the one editor
10. Any combination of file or hosted tiddlywikis with serial editors, one at a time, check in check out
11. Multi-user editors (Desktop, Server or Internet hosted variations)
1. Serial editor method, 
1. Whole of wiki checkout
2. Tiddler based checkout perhaps with commits from local storage once checked out
2. Multiple users with their own sandboxed wiki apparently updated only by themself
1. Just so the core and initial wiki seeds the user wiki, after that they are different apparent wikis.
2. User tiddlers saved to the wiki, in their own namespace and visible to all
3. User tiddlers saved to the wiki, their own namespace, and visible to all ONLY if "Imported" to the master
4. User tiddlers saved to the wiki, their own namespace, and visible to other if published using the static tiddler method (Host or export)
5. Users are using a subWiki which is federated to a master wiki automatically
6. Users only see the master wiki until they make changes and get a master + Overlay
3. Multiple users without sandboxing
1. Based on Bob perhaps
1. Using a consolidation view
2. Mechanisms in tiddlywiki to act as a multi-user CMS (at least server versions)

I am sure there are more combinations and perhaps as many again of ways to meet those requirements. 

Where possible a single solution should suit a whole subset of requirements, reducing the choices that must be made by the user/developer.

[PMario]

Hallo Hans,

On Thursday, June 27, 2019 at 11:32:12 PM UTC+2, Hans Wobbe wrote:

Thanks for sharing this DAT information, Mario. 

I'm following the project for quite some time now and I think it's a perfect fit for TW and vice versa.

 

In some ways, it seems like a lite-weight version of a blockchain. 

There's a detailed info at: https://datprotocol.github.io/how-dat-works/ ... It also contains more links on the first page.

I may be able to structure an evaluation exercise for this fall.  If so, I'll make sure the opinions are made available here.

That's interesting.

have fun!

mario

[Hans Wobbe]

Mario:

Thanks again for the insight and the links.

I'll talk to the technical managers early next month to see if they can plan to take this in as a small project for the co-op students we will have on September.

It's nice to be able to touch base once again after being distracted by business growth issue.