New Plugin tw-receiver saves wiki to PHP

[sendwheel]

Created a new plugin to handle saving the wiki to a PHP based server.

This can replace legacy store.php usage. The plugin is a bit more streamlined, updated code base, and adds some security enhancements to the process.

It also has a fairly thorough debug test that can be run during setup to help squash environment/server problems

Project page here:

https://github.com/sendwheel/tw-receiver

Comments and contributions welcome.

Security wise offers:

 - Challenge Digest Authentication (enhanced security)

 -- This simple mechanism avoids passing the password in plain text. Instead the server is queried for a challenge token and that token is then combined with the password to form a new string that is both unique and temporary.

 - Data Integrity Signing (enhanced security)

 -- This practice creates a unique signature of the wiki text with the secret key. Checking the validity of this signature ensures the integrity of the wiki data and helps prevent tampering in transit.

(also posted to TiddlyWikiDev)

Please let me know your thoughts. 

[Sylvain Naudin]

That sounds great !!

[David Gifford]

I added this to the toolmap (https://dynalist.io/d/zUP-nIWu2FFoXH-oM7L7d9DM#z=3wGygvpg5j7GS-bK4HIkF_DT) under the section "Publishing and hosting TiddlyWikis on the web"

Blessings.

[sendwheel]

Super. Thanks. Hopefully this plugin will get some adoption.

[TonyM]

sendwheel,

Thanks so much for this, I will test it in haste as I have a large online hosting environment based on PHP which has being unsuitable for hosting my "private" wikis. I could never get prior php solutions working.

I will eventually use https and even a CDN for speed.

I am also committed to integrating TiddlyWiki with other platforms like WordPress and this is a good method.

If I may extend the conversation a little, have you done any work with executing php from within TiddlyWiki - basically calling php installed on the host for this purpose? It would allow feedback and comments to read only tiddlywiki users.

Regards

Tony

[sendwheel]

Hi TonyM, let me know if you need any help with the setup. The Environment Test is pretty good for identifying issues...

I haven't done anything with executing php from within TiddlyWiki. I'd gather making a http request from javascript to a php script may be more suitable. Probably want to avoid letting the wiki push any kind of php code payload.

[TonyM]

Thanks Sendwheel,

I possibly have enough knowledge to be dangerous, but not to get this working. Perhaps you could extend the instructions a little to help me

I do not yet have it working, I create a new tiddler then select the save icon which say its started saving the returns Error while saving:

Error:

Server Error: Authentication Failure

I uploaded a Wiki to the host without the TW Receiver enabled otherwise I could not save it locally.

I then go to the served wiki in the browser and enable it, then try and save it with the above result.

     The server URL is by default tw-receiver-server.php

     I tried https://psat.com.au/tiddlywiki/tw-receiver-server.php with no effect

So I discovered the instructions make no mention of placing the following "components" anywhere 

• tw-receiver-config.ini
  
• tw-receiver-handler-comp.js
  
• tw-receiver-ui-comp.tw 
  

So I uploaded tw-receiver-config.ini  to my home/root directory

• However I am not so sure how to provide the secret key, is it encoded? I put it in as entered in the wiki
• Then set $extSecKeyPath = ../../tw-receiver-config.ini;  (Public_html then "root" folder, ie up 2 levels)
• Then set the password (I have not being prompted for this anywhere)
• The wiki is not saving but I have not received the Error: Server Error: Authentication Failure again
• The Debug was not working and I found the following error
  

  [06-Sep-2018 12:24:06 Australia/Melbourne] PHP Parse error:  syntax error, unexpected '.' in /home2/psatadmin/public_html/tiddlywiki/tw-receiver-server.php on line 19
  I changed this to $extSecKeyPath = "/tw-receiver-config.ini;" unlike the // Example: $extSecKeyPath = "../privatedir/tw-receiver-config.ini"

The state of play

• I am getting Error while saving: Error: Server Error: Authentication Failure

  with external key reachable

  NO

  I expect its my path to the tw-receiver-config.ini
  Or not placing the tw-receiver-handler-comp.js and tw-receiver-ui-comp.tw files where they belong.

Where if anywhere, should I place ?

• tw-receiver-handler-comp.js
  
• tw-receiver-ui-comp.tw 

How do I specifiy the path to tw-receiver-config.ini ?

I have some other trivial Questions, I may ask later.

Thanks

Tony

Other notes

I was getting an error in the folders error_log as follows

[06-Sep-2018 01:23:27 UTC] PHP Fatal error:  Call to undefined function random_int() in /home2/psatadmin/public_html/tiddlywiki/tw-receiver-server.php on line 135

However I switched the php version to 7.2 and it no longer seems to be occuring

I created a "backups" folder, which did not work until I read the tw-receiver-server.php and saw it is twbackups The Debug now reports

backups directory exists	OK
backups directory is writable	OK
this directory is writable	OK

[sendwheel]

The only file you need is tw-receiver-server.php on the server. The other files are just for the project and aren't actually needed to use the plugin.

I would recommend you get things working with a simple password first, and save using the $extSecKeyPath mode for later. Keep it at false for now.

password needed in 2 spots:

So just set the same secret password in Control Panel > Saving > TW Receiver
And modify the tw-receiver-server.php (line 20) file and make $userpassword = the same password you're using

-- That should solve your Authentication Error

Maybe start with a fresh copy of tw-receiver-server.php to make sure you don't have any bad edits.

and thank you for this feedback, I'll make some adjustments to the readme and look at minimum php version requirements.

[TonyM]

Sendwheel,

Thanks that is working and working well. It is very exciting for me and I may soon publish a number of tools online based on TiddlyWiki. Thanks so much.

For others reading this I have it working over https

In a spirit of progress a few notes

• So in summary the Wiki is read only and throws a message on the creation of a tiddler unless one goes to
  Control Panel > Saving > TW Receiver and provides the "password/Secret Key"
  
• Once you do this, saving will work in that browser session indefinatly?
  
• The whole folder could be placed behind a password to stop public access even read only.
  

Some feedback

Read only mode

Would it be possible not to throw a message when people have read only access and on every change they make, and leave it to when and If they try and save the wiki? Basically do not try and save if it does not have permission to do so.

Even simple state tiddlers could cause this failed save to occur, I want to build wikis that may be read only (to most) but the user can enter details and export/email their "input" the error message makes this somewhat unworkable. My work around is to turn off autosave however this exposes me to loss if I forget to save. Perhaps turning off Autosave if the tiddlywiki cant save to the server would be a practical solution.

Update Contention

I imagine that if more than one person had the password they could overwrite each others changes. Could we use php to drive a checkout process, basically allow a request to checkout and if not already (because its flagged) save a state on the server including the current user and or contact info (If provided) - thus no one else could check it out until the named user checks it back in (or the state deleted)?

Password "address"

I have noticed a Password Vault LastPass offering to save the password for me, however it only does so for the domain, and does not seem to recognise the password is meant for the site in its folder ie not for https://psat.com.au but for https://psat.com.au/tiddlywiki/PHPWiki.html or https://psat.com.au/tiddlywiki I think something in the authentication is causing this, perhaps there is away to ensure the password is recognised as belonging to the subsite not the domain.

Thanks again for a great leap forward (for me as I could not get others working) Perhaps because of the PHP Version - who knows.

Your contribution opens TiddlyWiki to further adoption.

Regards

Tony

 

On Thursday, September 6, 2018 at 1:06:22 PM UTC+10, sendwheel wrote:

The only file you need is tw-receiver-server.php on the server. The other files are just for the project and aren't actually needed to use the plugin.

I would recommend you get things working with a simple password first, and save using the $extSecKeyPath mode for later.

[sendwheel]

That's great. I'll look into that password saving w/ LastPass and others. See if I can't straighten it out.

[TonyM]

Sendwheel,

I have it running nicely under https: however it does lode over http which means if I do that there is a chance "they" will see me past the "password". Do you know what to do to force https only? I think it needs a rewrite or something. I ask in part because perhaps we should include that in the instructions.

I am just investigating but if I rename/clone the whole wiki to a new name, such as index.html or add additional wikis they should all work if they have the same credentials, is that correct? My Tests suggests so.

Using a separate folder with an independent tw-receiver-server.php and .htaccess file would be needed to have a different secret key is that correct?

Since the first day of install, the automatic backups seem not to be working. Can you suggest how to troubleshoot or fix?

$backupdir = 'twbackups'; // folder exists
// number of backups to keep
$backupcount = 10;

I have attached my own .htaccess file should anyone need it.

It is an excellent solution Thanks

Tony

[sendwheel]

Hi Tony, you should number your questions for easier reply 

1. The plugin doesn't use plain passwords, and instead uses hashed one-time use tokens that are based on the secret. So the secret is never passed out in the open.

That said it should still use HTTPS, but the plugin has no control over whether your server is using HTTP or HTTPS. MAke sure your path to tw-reveiver-server.php is a full path like https//:example.com/tw-receiver-server.php

2. Cloning/renaming is fine

3. yes one secret per wiki/folder

4. If you are having any backup issues you most likely have a permissions issue. To troubleshoot check the environment debug test by visiting tw-receiver-server.php directly. MAke sure you have writability to the backups directory and to the wiki directory. If you have server logs, perhaps check those for errors (on your webserver). Pretty basic system for backups, so the only real trouble is getting those permissions just right to start.

[sendwheel]

Added a new feature to the plugin:

Stale Instance Overwrite Protection

- This ensures the wiki you're working on isn't out of date with the server before saving changes. 

- It avoids a scenario where changes made earlier in another window were not loaded into the current instance of the wiki and would be lost by overwrite.

[TonyM]

Sendwheel,

That sounds great but how does it handle contention, such as when the same tiddler has two different contents?

Tony

[sendwheel]

Hey Tony,

The feature is more a warning system and leaves it up to the human to figure it out.

I wrote it to really only solve one scenario: 

I'm working on a tiddler for a while and do not save (call it Tiddler A). Then in another window or on a different computer I open the wiki and make some changes and save (call these changes B). 

Now maybe a day later I see my half edited Tiddler A, I finish it up and hit save...

In the old system I would have lost my changes (changes B). With the Overwrite Protection at least the save will fail and give me a warning. And in that case I can see about fixing it, maybe copying the content and refreshing.

Not revolutionary, but something that's certainly needed in place of a file lock.

[Scott Kingery]

I was just checking this out. The Read Me makes reference to importing plugin_sendwheel_tw-receiver.json into your wiki. Where can I find plugin_sendwheel_tw-receiver.json? I didn't see it on the github page unless Im missing something.

Thanks

[sendwheel]

Hey Scott, you're not wrong.

I just restored it, so you should be able to find it on the github page now.

Let me know if you run into any troubles.

[Scott Kingery]

Thanks. Got the file and set everything up. At least I think I did. Running into authentication errors when I try to save. Going to try again on a fresh empty wiki just to be sure I don't have anything hidden in there.

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/1pKkov12baI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/178c3dd7-1f1f-4621-a46b-5c90a82eca68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sendwheel]

Great.

Authentication should be easy fix, just make sure you put the password in the wiki and in the PHP file on the server.

[Scott Kingery]

Something still isn't right. Do I need to set anything on the Tiddlyspot Saver tab or only the TW Receiver tab? Debug looks ok.

Debug Tests

ini setting: file_uploads	OK
ini setting: upload_max_filesize	64M
ini setting: post_max_size	64M
backups enabled	YES
backups max count	10

backups directory exists	OK
backups directory is writable	OK
this directory is writable	OK

external key enabled	NO
external key reachable	NO
secure connection (https)	YES
challenge digest auth mode	YES
check data integrity signature	YES
check stale overwrite protection	NO

Notes: 
- Your upload_max_filesize and post_max_size must be at least larger than your wiki filesize 
- On NGINX client_max_body_size is another parameter worth looking at if uploads fail with 413 Request Entity Too Large  

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/69be41a7-ba53-4366-a24b-6cf29967c217%40googlegroups.com.

[sendwheel]

Ok let's figure this out.

You only need to work with the Saving>TW Receiver tab. It needs the secret key that you put into the tw-receiver-server.php on line 4

The Filename should also match whatever your wiki file is name (but that should not cause an auth issue)

What exact error msg is TiddlyWiki giving you? May help to debug. (Assuming you have triple checked your password matches in both places :)

[sendwheel]

I wanted to add that the Challenge Digest Authentication relies on Sessions and cookies. I don't mention it in the readme, but if for some reason your system had  cookies disabled, it could cause an auth problem.

If that could be the case, the feature can be disabled in the wiki tab and on line 37 of the php file. (although challenge auth is recommended, since you're using https it would be fine)

[dva...@gmail.com]

Thank you so much for this wonderful plugin. I was having a hell of a time getting tw5 to work on nginx with either node or the old php saver. This plugin WORKS and is finally allowing me to junk my old apache server for a new nginx setup.

Thanks again,

Deepak

[Mario Public]

When importing the plugin the error:

JSON error: SyntaxError: JSON.parse: unexpected character at line 7 column 1 of the JSON data

is reported

Mario

[HC Haase]

lørdag den 7. marts 2020 kl. 16.51.23 UTC+1 skrev Mario Public:

When importing the plugin the error:

JSON error: SyntaxError: JSON.parse: unexpected character at line 7 column 1 of the JSON data

is reported

Mario

if you copied the file form github, are you sure you have the json code, and not the html code for the link to the file? ( I did that) check the file in a text editor and see if it match what is on github

[Mario Public]

Thank You!

You were right!

Testing now.

Mario

[sendwheel]

Still maintained and still used. Glad to see new faces using the plugin.

[TonyM]

Sendwheel,

A most useful and working method, on a very common platform.

All I dream of now is being able to save other files to the same server platform, eg a users comments, or proposed changes.

Regards

Tony

[Anjar]

Hi!

I was experimenting a bit, and is the stale overwrite protection feature still working? When I tested with the tw open in two tabs, A and B, then saved A first with one change, and added another change to B, then only the latest saved change was kept (i.e. A was overwritten)

Best,

Anders

[Xekima Leo]

Hi, everybody!

I added Zip compress / decompress on send

https://github.com/daidalvi/tw-receiver

• JS zip creates via zip.js: https://github.com/gildas-lormeau/zip.js
• In backend unzip via ZipArchive: https://www.php.net/manual/en/class.ziparchive.php

If the size of your wiki exceeds post_max_size value on the server, you can check option "Create zip archive" in plugin settings to compress the document in the zip archive on the client side and send it to server where it decompress

I hope somebody needs this. Have a nice day!

пятница, 16 октября 2020 г. в 16:37:48 UTC+3, Anjar: