Running TW5 on TiddlyWeb

[Robert Kreuzer]

Hi,

I would like to run TW5 via TiddlyWeb for a multi-user wiki. I have a running TiddlyWeb instance, but I couldn't find any documentation how to integrate TW5 with it.
Is there anybody who has done this recently and could point me in the right direction?

Thanks,
Robert

[Chris Dent]

On Monday, March 14, 2016 at 2:26:48 PM UTC, Robert Kreuzer wrote:

I would like to run TW5 via TiddlyWeb for a multi-user wiki. I have a running TiddlyWeb instance, but I couldn't find any documentation how to integrate TW5 with it.
Is there anybody who has done this recently and could point me in the right direction?

Thanks for posting here Robert. As I said I would in the issue[1] I've started experimenting with this and it is has been more complicated that I initially expected, so it is good the thread is started here so we can get some input from people more familiar with the tw5 side of things. I'm gathering my notes here so people can chime in and we can figure out the right solution. Eventually we'll have one.

We didn't talk at the time how you wanted to set things up, there are a few options:

1. Run a web server that hosts a tw5 html file that it serves to users, alongside the tiddlyweb service. This is a bit like what Tank[2] can do and is essentially what tw5 itself does, in server mode.
2. Have editions of tiddlywiki5 which are built to talk to a server where the content lives, but are otherwise empty. This mode could allow the same bare tiddlywiki5 to talk to lots of different server, based on configuration settings.

The editions/tw5tiddlyweb that is in the TiddlyWiki5 code repo is built specifically for talking to TiddlySpace, requiring a few features that are (custom /status response, custom authentication challenger). Since TiddlySpace is dying, this isn't a great solution, so I worked on option 2 to see what issues are present.

There are a few.

On the tiddlywiki5 side:

* tiddlywebadaptor.js requires /status to provide recipe information. This is only done on tiddlyspace, and is extractable from the user's request based on the domain they are accessing the service with. On a plain TiddlyWeb server you can only know the recipe if it is in the URL. So this won't work. In my experiment I modified tiddlywebadaptor.js that will use a $:/config/tiddlyweb/recipe tiddler during the build process. It contains the name of the recipe (in my case "foo") in which tiddlers are to be stored.
* tiddlywebadaptor.js assumes that the wiki will be loaded from http or https. If it is not, the adaptor doesn't run at all.

If I create appropriate $:/config/tiddlyweb/host and $:/config/tiddlyweb/recipe tiddlers, adjust tiddlywebadaptor.js to use, and adjust the adaptor to export itself without checking document.location.protocol, then I can produce an edition which will communicate appropriately with the tiddlyweb server and recipe described by the host and recipe tiddlers.

On the tiddlyweb side:

* Even though it doesn't produce the desired information a /status url still needs to return a json object such that tiddlywebadaptor.js will call it and then call the subsequent callback (listing the tiddlers on the server). This can be done by adding the tiddlywebplugins.status plugin to the tiddlywebconfig.py's system_plugins list.
* In order for the browser to communicate to the server via a file: URL CORS is required on the server. tiddlywebplugins.cors can provide that.
* The recipe defined $:/config/tiddlyweb/recipe needs to exist before the TiddlyWiki looks for it.
* For the time being I've ignored authentication.

Once all that's done, at least in Firefox it all works quite gracefully. Load the tiddlywiki, it gets the list of tiddlers from the server, when you create a new one it is saved to the server.

So that's at least a starting point. That suggests we need to:

* Effectively document the tiddlyweb-side configuration and running pieces. It's not hugely complicated, but it does require installing tiddlyweb plus a few plugins and then configuring the instance to use them. I can make a cookbook for that and then I reckon someone could probably make a container that does it cleanly.

* Either create a new edition for tiddlywiki5 which could be described as "standalone access to tiddlyweb", or make the existing tw5tiddlyweb edition support that use case as well.
* Such as edition would need to account for the changes to tiddlywebadaptor.js described above. Probably by effectively branching in the right places based on certain conditions. This is hard for me to do because I don't know the available contextual variables.

What do people think?

[1] https://github.com/tiddlyweb/tiddlyweb/issues/105
[2] https://tank.peermore.com/tanks/docs/TiddlyWiki%20Five

[chris...@gmail.com]

On Mon, 14 Mar 2016, Chris Dent wrote:

> * Either create a new edition for tiddlywiki5 which could be described as
> "standalone access to tiddlyweb", or make the existing tw5tiddlyweb edition
> support that use case as well.
> * Such as edition would need to account for the changes to
> tiddlywebadaptor.js described above. Probably by effectively branching in
> the right places based on certain conditions. This is hard for me to do
> because I don't know the available contextual variables.

There's a very rough diff of my local tw5 changes here:
https://github.com/Jermolene/TiddlyWiki5/compare/master...cdent:cd/tiddlywebforrealz

--
Chris Dent http://burningchrome.com/
[...]

[Robert Kreuzer]

Thanks for looking into this, Chris. I will try to replicate what you have done with both TW5 and TiddlyWeb and report back with my findings or issues (although I'm an avid TiddlyWiki user I'm not really familiar with the internals of either TiddlyWiki or TiddlyWeb, so it might take me a while).

[Robert Kreuzer]

Hi Chris,

I have followed your instructions as far as I understood them, but I unfortunately know too little about tiddlyweb to make sense of everything. Here is what I have done:

1. created a test instance in tiddlyweb with `twinstance test`
2. Added the tiddlywebplugins.status and tiddlywebplugins.cors plugins to test/tiddlywebconfig.py ('system_plugins': ['tiddlywebwiki', 'tiddlywebplugins.status', 'tiddlywebplugins.cors'])
3. Create a new recipe called 'somerecipe' using: curl -X PUT -H 'Content-Type: application/vnd.tiddlyweb+json' -d '{"recipe":[["system",""],["common",""]]}' http://0.0.0.0:8080/recipes/somerecipe

Does this make sense so far? If I understand you correctly, I should now have a $:/config/tiddlyweb/recipe that points to 'somerecipe'.
But I'm not sure where the TW5 files should go in my 'test' instance. And do I just clone the TW5 repo and copy it in, or is there some other way to do this?

Thanks,
Robert

On Monday, March 14, 2016 at 11:08:09 PM UTC+1, Chris Dent wrote:

[chris...@gmail.com]

On Fri, 25 Mar 2016, Robert Kreuzer wrote:

> I have followed your instructions as far as I understood them, but I
> unfortunately know too little about tiddlyweb to make sense of everything.

Apologies for not keeping track of this as much as I initially
promised. Without further input from the TW5 experts it can only get
so far and so far we've not really heard from them...

> 1. created a test instance in tiddlyweb with `twinstance test`
> 2. Added the tiddlywebplugins.status and tiddlywebplugins.cors plugins to
> test/tiddlywebconfig.py ('system_plugins': ['tiddlywebwiki',
> 'tiddlywebplugins.status', 'tiddlywebplugins.cors'])

You'll also need these settings:

‘cors.enable_non_simple’: True,
‘cors.allow_creds’: True,
‘cors.match_origin’: True,

> 3. Create a new recipe called 'somerecipe' using: curl -X PUT -H
> 'Content-Type: application/vnd.tiddlyweb+json' -d
> '{"recipe":[["system",""],["common",""]]}'
> http://0.0.0.0:8080/recipes/somerecipe

This is right.

> Does this make sense so far? If I understand you correctly, I should now
> have a $:/config/tiddlyweb/recipe that points to 'somerecipe'.
> But I'm not sure where the TW5 files should go in my 'test' instance. And
> do I just clone the TW5 repo and copy it in, or is there some other way to
> do this?

This is the tricky tw5 part. There are changes necessary to code
within the tw5 repo to make things go. Those changes are shown in
this diff:

https://github.com/Jermolene/TiddlyWiki5/compare/master...cdent:cd/tiddlywebforrealz

Where I've put 'foo' in 'tiddlyweb-recipe.tid' you would put
'somerecipe'.

With those changes in place you can then build a TW5 following the
build instructions available with tidldywiki5.

I'm hoping someone will step up from the tw5 side to provide some
detailed instructions on how to do that and maybe how to most
effectively get the tiddlywebadaptor.js code working properly with
tiddlyweb.

The end goal here is making it possible for a local tiddlywiki html
file to talk to a tiddlyweb server effectively. I've made it work,
quite well, but it requires a lot of hacking into the TiddlyWiki
code (as shown by the diff above).

[Robert Kreuzer]

Thanks again for your input and apologies for my late response (I only find time on weekends to work on this unfortunately). But following your instructions and some trial and error I have now managed to make it all work together.
I now built the modified edition of TW5 and it is talking to the TiddlyWeb server, which is very cool. These were the necessary steps (in case somebody else wants to try this):

1. git clone g...@github.com:Jermolene/TiddlyWiki5.git tiddlywiki5
2. cd tiddlywiki5
3. Adapt all the files as per the diff above
   
4. ./tiddlywiki.js wiki –init tw5tank
5. cd wiki
6. ./../tiddlywiki.js ../wiki –rendertiddler $:/core/save/all index.html text/plain
7. firefox output/index.html
8. Run the modified tiddlyweb server with "twanager server"
   

While working on this, I realized that TiddlyWiki itself now also comes with a simple server-side component using NodeJS.
I was wondering what the differences/advantages of using it are rather than using TiddlyWeb as the server-side?

It looks to me as if TiddlyWeb has a lot more features but is currently also quite hard to integrate with TW5.
Are there some fundamental differences that make one more suitable than the other for a multi-user TW5?

[chris...@gmail.com]

On Sun, 10 Apr 2016, Robert Kreuzer wrote:

> While working on this, I realized that TiddlyWiki itself now also comes
> with a simple server-side component using NodeJS.
> I was wondering what the differences/advantages of using it are rather than
> using TiddlyWeb as the server-side?
>
> It looks to me as if TiddlyWeb has a lot more features but is currently
> also quite hard to integrate with TW5.
> Are there some fundamental differences that make one more suitable than the
> other for a multi-user TW5?

As far as I recall TW5's server doesn't support users (or at least
not multiple users) at all. Nor does it support TiddlyWeb's concept
of policies[1], recipes[2] and filters[3] in any significant way.

Recipes are effectively the reason TiddlyWeb was created in the
first place. Since then things like serializers[5] and plugins[6]
have made it very flexible and capable.

[1] https://tank.peermore.com/tanks/tiddlyweb/policy
[2] https://tank.peermore.com/tanks/tiddlyweb/recipe
[3] http://tweb-filters.tiddlyspace.com/ [4]
[4] The filters content will be moving off tiddlyspace soon.
[5] https://tank.peermore.com/tanks/tiddlyweb/serializer
[6] https://tank.peermore.com/tanks/tiddlyweb/plugins

[Devin Weaver]

3. Adapt all the files as per the diff above

This can be done easily by downloading this file: https://github.com/Jermolene/TiddlyWiki5/compare/master...cdent:cd/tiddlywebforrealz.patch and running git am path_to_downloaded_file
 This can easily be done in one line with curl:

curl -L https://github.com/Jermolene/TiddlyWiki5/compare/master...cdent:cd/tiddlywebforrealz.patch | git am

[Jeremy Ruston]

Hi Chris

Apologies for the late response. I’ve created a pull request of your changes:

https://github.com/Jermolene/TiddlyWiki5/pull/2371

I’ll post further questions over there,

Many thanks,

Jeremy.

> --
> You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
> To post to this group, send email to tiddl...@googlegroups.com.
> Visit this group at https://groups.google.com/group/tiddlywiki.
> To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/alpine.OSX.2.20.1603301243590.37861%40shine.local.
> For more options, visit https://groups.google.com/d/optout.

[Jeremy Ruston]

Hi Robert

The built-in server of TiddlyWiki5 is designed for personal use; TiddlyWeb is intended to be the official multi-user back end that’s tough enough to be exposed on the public Internet.

Best wishes

Jeremy.

[Robert Kreuzer]

Hi Jeremy,

thanks for the explanation, that makes sense to me.

Right now it was still quite hard to get TiddlyWeb and TW5 working together for someone with no prior knowledge like me (and would have been impossible without Chris' help).
So I thought maybe I could contribute some docs to describe how to set everything up. But I'm not sure where the best place to do so would be.
I see that Chris' pull request merges the TW5-specific changes into the 'tw5tank' edition. But then there is also a 'tw5tiddlyweb' edition, which would maybe be the better place?

Best,
Robert

[David Andel]

Hi Robert,

Have you placed your docs somewhere in the meantime?
I would be very interested to learn how to do this.

Thanks and best regards,
David