[TW5] Bob and BobEXE version 1.1.0 Actually Garlic Bagels prerelease

[Jed Carty]

I ended up doing some things on the back-end of Bob that would break backwards compatibility if anyone modified the server components of Bob. I am not aware of anyone who has but this gets a minor version bump anyway. I never got through testing version 1.0.7 before getting distracted so those changes got rolled into this.

This is a prerelease for now because there have been a lot of changes and while I am not aware of any new bugs from them, there may be some I have missed. As always, back up your wikis!! The easiest way is to just make an archive of your Wikis folder and the IndexWiki.

The big changes from the perspective of using Bob are:

- you can now build html wikis that take tiddlers from any of the wikis served by Bob instead of just building an html version of an existing wiki.
- You can select which plugins from the server should be used in the current wiki
- Custom tiddler paths using $:/config/FileSystemPaths should work now

The plugin version of Bob is on GitHub here: https://github.com/OokTech/TW5-Bob

The newest version of BobEXE is available here: https://github.com/OokTech/TW5-BobEXE/releases

If you want to support the development OokTech has a patreon page here

https://www.patreon.com/OokTech

or if you prefer there  is a link for PayPal here

https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=ZG94CTLHTKYRE

Here is the current changelog for version 1.1.0

!! Version 1.1.0 (ongoing)

- Add `savePluginFolder` web socket message to save packaged plugins as folders on a server. These plugins can then be available to other served wikis and in the plugin library, if it is enabled.
- Add the `import` browser message handler to make fetching nicer
- Add options for how to handle conflicts for `internalFetch`
- `internalFetch` now displays a list of imported tiddlers
- Added a way to change which plugins a wiki uses from inside the wiki
   - In the Bob Settings tab there is a Manage Plugins tab
- Added an interface to save plugins to the server and to select which plugins from the server should be used in the current wiki
- Change the internals to make one $tw.Wiki object for each wiki instead of having them all in $tw.wiki
   - There are lots of internal changes to make this work
   - $tw.boot.files still uses the prefixed name
- Streamlined and simplified the file system monitor
   - It should now watch new subfolders inside the tiddlers folder
- Environment variables are now set from the plugin, themes, and languages paths given in the settings.
   - This is needed by the core to properly list available plugins
   - This means that you can set the location of one folder listing plugins, themes and languages in the settings instead of using environment variables. At the moment this is limited to one folder.
- The `unloadWiki` message now ensures that the connection to any browsers with that wiki open are closed correctly.
- Updates to the `buildHTMLWiki` message
   - You can now pull tiddlers in from any served wiki while making an html file. This means you can combine wikis in the output.
   - There is a default filter for excluding plugins like Bob and components that are only useful on a server
   - You can add an additional filter to exclude tiddlers from the exported html wiki
   - You can disable the default exclude list for more control
- The `$:/config/FileSystemPaths` settings should work now

[Matthew Lauber]

One issue I noticed.  The UI for saving plugins doesn't allow you to save updated versions of plugins.  I've a few plugins I regularly update, but after I added them to the Bob library once, I couldn't update them again.  Is there a reason why adding a plugin twice would result in problems, or is it just for the UI convience that things already added are removed from the list?  Would it be possible to compare version numbers and allow updating plugins based on differing numbers?

~Matt

[Jed Carty]

I think I need more information because I am not sure what you are referring to here.

If you are trying to save a plugin to the server as part of the plugin server should check the plugin version number and if the plugin you are trying to save exists on the server with a newer version than it won't save it, but never plugin versions should save fine.

I suspect that you imported the plugin into your wiki using drag and drop and then didn't delete the plugin tiddler after saving it to the server. Plugin tiddlers in your wiki always override plugins listed in tiddlywiki.info.

If that is not the case than I need more information before I can say anything useful.

[Matthew Lauber]

Ok, so how I'm using Bob is that in the root wiki I'm making all my changes to my plugin via tinka, and then I'm using Bob's plugin saving feature to share it with all my other wikis.  So, steps to reproduce.

1) Import a plugin into the root wiki for the first time.

2) Save it using the "Manage Plugins" section of Bob Settings.  (It shows up in the drop down next to Save Plugin as Folder)

3) Import it to other wikis as needed via the "Manage Plugins" section of Bob Settings in the other wikis.  

4) Update the plugin in the Root wiki, and repackage via tinka with a different/higher version number.  

5) ??? The plugin does not show up in the drop down box of "Manage Plugins" and my other wikis are not showing the changes I made to the plugin.  

Other information that may be useful. I imported the plugin in the root wiki, but I'm not selecting it in the "Save Plugin Selection" of the root wiki.

If there's anything else I can add to help, let me know.  

Thanks, Matt

[Jed Carty]

My first thought is that my UI isn't very good. If I remember correctly the list in the dropdown only shows plugins that are in the wiki but aren't listed as being on the server but doesn't check the versions. So I believe that your initial assessment is correct. I made it so that the part that sends the plugins to the server checks version numbers but I probably didn't put that on the UI component.

So it is an oversight on my part. Until I get around to finding a way to fix that you could manually send the updated plugin to the server by making a copy of the button that has the plugin explicitly listed.

The button code would be something like this (changed the plugin name to match):

<$button> Save Plugin to Server
<$action-websocketmessage $type='savePluginFolder' pluginName='$:/plugins/Your/Plugin'/>
</$button>

there may be another bug in how the plugins are saved, if you remove a file from the plugin it may not be properly removed on the server. So back up your data!

I need to find out how to solve that in a way that doesn't make it easy to accidentally delete plugins from the server.

[Mark S.]

Trying it out in Linux, Ubuntu 18.04. FF 62.0.3. Bob Plugin 1.1.0.

When attempting to attach to existing TW, I get a series of 404 errors:

syncer-browser-tiddlyweb - 15:31:55 23 10 2018

XMLHttpRequest error code: 404

syncer-browser-tiddlyweb - 15:31:56 23 10 2018

Sync error while processing '$:/StoryList':
XMLHttpRequest error code: 404

syncer-browser-tiddlyweb - 15:31:56 23 10 2018

Sync error while processing '$:/plugins/OokTech/Bob/Unsent':
XMLHttpRequest error code: 404

Here's the messages in the browser console:

Error: WebExtension context not found! ExtensionParent.jsm:1014:13
we are insidehttp://127.0.0.1:8081/Recipes logsimple.js:15:3
Error: Could not establish connection. Receiving end does not exist. Browser.js:3:1937
moz-extension://3175b0c3-efc9-4241-8dff-066790403f8d/background.html: chrome tabs query did not return a result while changing window focus

Log.js:3:3828
TiddlyWebAdaptor: Getting status $:/core/modules/utils/logger.js:33:10
Logging system initialised at Tue Oct 23 2018 15:31:55 GMT-0700 (PDT) common.js:206:17
syncer-browser-tiddlyweb: XMLHttpRequest error code: 404 $:/core/modules/utils/logger.js:33:10
Opened socket $:/plugins/OokTech/Bob/BrowserWebSocketsSetup.js:77:7
[Show/hide message details.] tempTab is undefined background.js:662
XML Parsing Error: no root element found
Location: http://127.0.0.1:8081/status
Line Number 1, Column 1: status:1:1
XML Parsing Error: no root element found
Location: http://127.0.0.1:8081/Recipes
Line Number 1, Column 1: Recipes:1:1
syncer-browser-tiddlyweb: Dispatching 'save' task: $:/StoryList $:/core/modules/utils/logger.js:33:10
syncer-browser-tiddlyweb: Sync error while processing '$:/StoryList':
XMLHttpRequest error code: 404 $:/core/modules/utils/logger.js:33:10
syncer-browser-tiddlyweb: Dispatching 'save' task: $:/plugins/OokTech/Bob/Unsent $:/core/modules/utils/logger.js:33:10
XML Parsing Error: no root element found
Location: http://127.0.0.1:8081/recipes/undefined/tiddlers/%24%3A%2FStoryList
Line Number 1, Column 1: $:/StoryList:1:1
syncer-browser-tiddlyweb: Sync error while processing '$:/plugins/OokTech/Bob/Unsent':
XMLHttpRequest error code: 404 $:/core/modules/utils/logger.js:33:10
XML Parsing Error: no root element found
Location: http://127.0.0.1:8081/recipes/undefined/tiddlers/%24%3A%2Fplugins%2FOokTech%2FBob%2FUnsent
Line Number 1, Column 1: $:/plugins/OokTech/Bob/Unsent:1:1

The file actually serves -- but with constant stream of yellow popup error messages.

Got the same errors when using TW's made by TiddlyDesktop.

Other comments.

Couldn't find a "filepathroot" setting, so assume we have to add it manually under "manual" ??

Thanks!

-- Mark

[Jed Carty]

Mark,

It looks like I somehow broke the part that removes the filesystem and tiddlyweb plugins from served wikis. It looks like all of the errors you are getting are caused by that. Until I get that fixed one solution is to remove the tiddlyweb and filesystem plugins from the tiddlywiki.info file. I will try and fix that later today.

The configuration.md file shows where each configuration property goes and has a brief description of each one. https://github.com/OokTech/TW5-Bob/blob/master/Configuration.md

[Mark S.]

Additional note. It's possible for the local network address listed at the top to be different from the actual port number used by the wsserver at the bottom of the configuration (mine is 8081 at the top and 8082 in wsserver).

-- Mark

[Mark S.]

So far no luck getting images to serve. But there wasn't an existing filePathRoot entry, so perhaps I am setting it up wrong?

-- Mark

[Peter Merel]

My head's been out of TW for several months so all this Bob stuff is new to me. Well done Jed and and everyone involved!

But I have a question about a statement on the Multi-User TW5 plugin github site  where you say "a git plugin is currently being developed”. I see this as critical to multi-use TW on the open web and very keen to hear about its status. 

I should explain why.

20 years ago I spent a lot of time on the original c2 wiki. I ran one of its sister sites, GreenCheese, edited about 2000 pages of the original WardsWiki, and built the first open source clone of it, “CvWiki". So I saw most of the heat-death of c2 up close and personal. I was one of the dozen folk Ward picked as stewards to try to defend it form the trolls … unsuccessfully …

Right now I'm running the “free and open source agile” community at http://xscalealliance.org . We have a TW at http://xscale.wiki but most of our communications are going via slack. I see slack as very inferior to the wiki way and I’m keen to open up xscale.wiki to multiple users. But given the broad potential community I think I need git integration to do it.

I had been thinking that we’d just let practitioners use TW's single user node.js server to edit, and then submit pull requests in the usual git way. I probably don’t need to tell you that’s clunky …

What I’m hoping for is a multi-user mode where users are able to make a number of local edits on their local copy of the wiki, then when they press "save" we would have a pull-request generated automagically along with a little form for them to fill in the purpose of their edits. After that, normal github-flow should be sufficient for the community to police its own development. 

Though I think we might also want the “Recents” tab to be improved to change the river to reflect the recently changed tiddlers. RecentChanges was the beating pulse of c2 … to the point that back in the day many of us made it our browser’s homepage. We called ourselves "RecentChangesJunkies" ...

No worries if you meant something completely different by this but I thought it might save us a lot of duplicated effort if you’re going down this path and I'm enormously impressed with the progress you've made on Bob to date!

Cheers,

Pete

[@TiddlyTweeter]

Ciao Mark

Yeah, you need (1) have the "filePathRoot" set in Manual Settings. Also (2) optionally "fileURLPrefix". (3) Ensure that the "mimeMap" is active (I found, 2 Bob versions back, I had to set it manually despite docs saying they were on by default. I haven't checked if I can remove those manual settings on the latest).

Here is an example of my settings ...






I wrote a brief note on my broad take on it for managing files for export: https://groups.google.com/forum/#!msg/tiddlywiki/XsVzEJBDd9Q/6svo1i_BAwAJ

Not being familiar with the node style of serving external files I found the "virtual"address of "files" initially somewhat confusing. Now I'm a bit clearer.

Once up and running the serving is robust.

HTH
J

[Jed Carty]

Mark,

I am not sure what you mean by 'at the top' and 'at the bottom', but there is a known bug with the port numbers that can happen when you allow it to auto-increment ports https://github.com/OokTech/TW5-Bob/issues/24

To make images serve you have to set the filePathRoot which is the root path served by the static file server. The default mime is listed in the serving files documentation https://github.com/OokTech/TW5-Bob/blob/master/Documentation/Serving%20Files.tid but that was just updated to fix some errors, it had an incomplete listing for the mimeMap and it said the default url prefix was file instead of files.

And if there isn't a filePathRoot entry than serving files doesn't work, it has to be enabled by setting a filePathRoot.

I have no idea what 'virtual addresses' are supposed to be, the files are served in the normal way by a webserver.

Here are the steps that I just did to make the file server work:

1) Download BobEXE for your system (or set up the plugin version, it makes no difference but this is faster for testing)

2) Start it and open up the root wiki

3) Go to the $:/ControlPanel

4) Go to the Bob Settings tab in the control panel

5) Go to the Manual Settings tab in the Bob Settings tab

6) At the bottom of where the settings are listed click the last + button, name the key (the smaller text entry field) filePathRoot and give it the value (the longer text entry field) that is where you have the files you want to serve (in my case /home/inmysocks/pictures)

7) Click the check button under the text entry fields

8) Click the Update Settings button

9) Under the Server Actions tab click Shutdown Wiki

10) Restart the wiki

11) Make a new tiddler and put in [img[./files/01.jpg]] (assuming you have a picture called 01.jpg in the folder listed in the filePathRoot)

it does look like there is a bug here where if you do something that saves the manual settings again it may stop serving files until the server is restarted again.

[Jed Carty]

It looks like the bug isn't caused by any changes to the manual settings, just to adding a new wiki. There is probably something about how the new wiki updates the server paths that doesn't play well with the file server component. This is why I have a separate server component for the online version.

[@TiddlyTweeter]

Jed wrote ....

I have no idea what 'virtual addresses' are supposed to be ...

My settings are ...



By "Virtual" I meant that an image is addressed in a Bob TW tiddler like this ...

[img[./f/bm/img/Funny-Sheep-Facts.jpg]]

The on disk address of the file is actually ...

C:\bag\PortableApps\twBOB\ext\bm\img\Funny-Sheep-Facts.jpg

The "virtual" bit is the "f" //that doesn't actually exist on disk// we need in the tiddler's reference to the image in Bob for it to work.

That is my current understanding and works.

Best wishes
Josiah

[aged...@yandex.com]

This is exactly what I was looking for yesterday! 

Thank you for this precise step-by-step ... I had read the documentation but didn't "get it" ... this worked perfectly for me!

[Mark S.]

I had to add the fileURLPrefix (undocumented) before it would work. 

Having to serve images from one location is a bit limiting, when you may have Wikis in multiple locations.

Thanks

-- Mark

[aged...@yandex.com]

Mark,

I don't know if this would work for you, but I just created sub-folders inside the folder I identified as my filePath so that I can keep the images for a given wiki separate from the others.  You just need to make sure you add the subfolder to the image's address in the tiddler you're putting it in.

I hope that made sense, and was helpful.

[@TiddlyTweeter]

Mark S. wrote:

I had to add the fileURLPrefix (undocumented) before it would work. 

It's semi-documented. It was documented enough for me to know it was important.
 

Having to serve images from one location is a bit limiting, when you may have Wikis in multiple locations.

Its easy to create subordinate levels within the "files" folder so that files can be segregated as needed.

Its just slightly problematic on export methodology to create single-file wiki that exports their specific files to them. This is something I commented on before but I don't want to lumber Jed's thread with my personal interests too much.

Josiah

[@TiddlyTweeter]

Dear aged ... yes!

I just created sub-folders inside the folder I identified as my filePath so that I can keep the images for a given wiki separate from the others. 

FWIW I suggested elsewhere that it might be a good idea that when you create a new wiki you can also assign a "files ID" (just two letters that indicate the sub-directory). I reasoned like this ...

What I am wondering is whether when you create a new Wiki you can be prompted to define a "shorthand ID" for it to ensure you never forget the base address you need for files so you don't get in a mess? This comes from concern some of my wiki have many images associated with them.

 Why is setting a "shorthand ID" useful? You can use it as a variable that is invoked in a Bob script passed to the OS that tells its where to copy the files from and where to. Also a problem arises if you don't because, if you change the name of the wiki, it becomes unclear where to find them. Basically I (personally) think a set shorthand ID could be useful.

Josiah

[aged...@yandex.com]

@Josiah

You just caused me to see that what I did may not be the 'best' way! I'm still very new to TW5 and do not yetl always appreciate the future proofing actions many of you have come to understand. 

I did name the sub-folder the current wiki's name ... now I see that should I ever change the wiki's name in the future, all those links will end up being broken.  NOT A GOOD SCENARIO! 

I like your suggestion of using some sort of ID or prefix code to name the sub-folders.  Also, an ID or prefix-code potentially keeps the links shorter as well, as some of my wiki's names [which is what I used for the subfolder names] are a little longish at the moment.

AgedLace

Thank you for pointing this out - much easier to fix now than later on down the line!

[Mark S.]

On Thursday, October 25, 2018 at 8:03:01 AM UTC-7, @TiddlyTweeter wrote:

Mark S. wrote:

I had to add the fileURLPrefix (undocumented) before it would work. 

It's semi-documented. It was documented enough for me to know it was important.

Where exactly is it "semi-documented" ?

Having to serve images from one location is a bit limiting, when you may have Wikis in multiple locations.

Its easy to create subordinate levels within the "files" folder so that files can be segregated as needed.

But having to have all your images in one location, no matter how you divvy that location up, is what's limiting. With plain TW, or TW with tiddlyserver, your files can be somewhere in respect to your TW location, no matter where it is. That way you can zip up the contents of a directory and have all the information that's required to make a complete TW-eco system.

-- Mark

[@TiddlyTweeter]

@TiddlyTweeter wrote:
It's semi-documented. It was documented enough for me to know it was important.

 

Mark S. asked ...

Where exactly is it "semi-documented" ?

When someone asks "exactly" I know I need to come to attention :-). 

I tend to use https://github.com/OokTech/TW5-Bob/blob/master/Configuration.md which does include it. Its a kind of Bible of Bob.

But having to have all your images in one location, no matter how you divvy that location up, is what's limiting.

I agree. My take is its about making the best of that. I think its a limit on node derivation?

With plain TW, or TW with tiddlyserver, your files can be somewhere in respect to your TW location, no matter where it is. That way you can zip up the contents of a directory and have all the information that's required to make a complete TW-eco system.

I agree. The files "in the other place" is awkward on export from Bob. It requires clear planning in advance or you can get in a mess. You can't just press a button and do it auto.

On the other hand Bob scripts can ameliorate the issue somewhat (like quite easy "copy" via OS from and to). But the "virtual" addressing in the wiki itself still needs fixing (i.e. "./f/img/fred.jpg" -> "./img/fred.jpg") when you export to singular.

Josiah

[Jed Carty]

The fileURLPrefix is not undocumented, it is listed with everything else https://github.com/OokTech/TW5-Bob/blob/master/Configuration.md

I have not run into any troubles where I had to set the fireURLPrefix before it works, if following the steps as I listed them does not work for you than there is some other problem that we should find and fix.

[Mark S.]

On Thursday, October 25, 2018 at 9:40:39 AM UTC-7, @TiddlyTweeter wrote:

On the other hand Bob scripts can ameliorate the issue somewhat (like quite easy "copy" via OS from and to). But the "virtual" addressing in the wiki itself still needs fixing (i.e. "./f/img/fred.jpg" -> "./img/fred.jpg") when you export to singular.

I  don't understand why it needs  fileURLPrefix . It means automatically that any existing media tiddlers you have will have to have their path names changed.

-- Mark

[@TiddlyTweeter]

Jed

https://github.com/OokTech/TW5-Bob/blob/master/Configuration.md

.. is very good. But I can't find it in BobEXE on Windows? I go to GitHub for it. Its a kind of Bible Of Bob.

[@TiddlyTweeter]

Mark S. wrote ...
 

I  don't understand why it needs fileURLPrefix . It means automatically that any existing media tiddlers you have will have to have their path names changed.

I have no idea why. All paths to files served right need a path prefix--that would be redundant for stand alone TW. I guess its something to do with its node history? The naming is a bit strange. And on export to single file its kinda the sand in the ointment of an otherwise brilliant system.

Just thoughts
Josiah

[Mark S.]

Oddly enough, I assumed that "Bob documentation" was the actual documentation.

Bob wouldn't serve images until I added fileURLPrefix. At first I thought it might be the missing mime type that was the problem, but after fileURLPrefix I could remove the manual mime types.

Thanks

-- Mark

[Jed Carty]

The reasons are the same as why until 5.1.18 comes out the node version didn't have a static file server and that the static file server with the server in 5.1.18 does the same thing:

1) engineers that make things have an obligation to make sure that the things that they make are safe to use

2) it is a webserver, not a local file. The paths aren't virtual paths or anything like that, they are urls. A url doesn't necessarily have any relation to the file system location of what it points to, but it does have meaning to the server.

The server Bob uses a web server that can serve multiple wikis, when the server gets a request it has to determine what the url is supposed to point to, if you don't have a prefix for files than requests to the server can be ambiguous about if they are supposed to be a served file or a wiki and you get inconsistent behaviour. When you have inconsistent behaviour is almost always a security risk.

To prevent Bob from exposing everything on your hard drive to anyone who wants it there are a number of safeguards. When serving a wiki the wiki has to be listed in the settings and the tiddlywiki.info file has to exist, because of how tiddlywiki is structured there is relatively little risk involved there. There is far more risk when serving arbitrary files off of your computer. If you are not very careful with it than there are ways to ask for a file located somewhere like ../../../../.ssh and then get all the private keys or passwords on a computer.

And furthermore, Because Bob lets you have wikis hosted on paths like localhost:8080/Wiki1 and localhost:8080/Wiki1/Wiki2 it becomes ambiguous how to handle something like localhost:8080/Wiki1/foo.jpg, it could be a wiki called foo.jpg or it could be the file foo.jpg. Exploiting edge cases like that is one method of gaining unauthorised access to systems.

Putting all the static file serving on localhost:8080/files and using a defined mimeMap to filter requests helps with those problems.

[Mark S.]

On Thursday, October 25, 2018 at 10:36:26 AM UTC-7, Jed Carty wrote:

The server Bob uses a web server that can serve multiple wikis, when the server gets a request it has to determine what the url is supposed to point to, if you don't have a prefix for files than requests to the server can be ambiguous about if they are supposed to be a served file or a wiki and you get inconsistent behaviour. When you have inconsistent behaviour is almost always a security risk.

I don't get what you mean by "a served file or a wiki". Isn't there a world of difference between the two?

And why couldn't it be the other way around -- have the "file/" prefix associated with the Wikis, while the served files can have any prefix (but only served from below the file root path). Having the prefix on the Wiki name would not cause the consternation that having it on the image path name.

As it is, the image paths for Bob will be totally incompatible with standalone, TiddlyDesktop, and tiddlyserver file paths.

Thanks,

-- Mark

[@TiddlyTweeter]

Jed Carty wrote:

The reasons are the same as why until 5.1.18 comes out the node version didn't have a static file server and that the static file server with the server in 5.1.18 does the same thing:

Right. I don't have any problem with it per se now I better understand it. And I'm grateful you keep malicious hackers away. You have that skill & its good.

The singular issue I have is that node/Bob serving of files involves (1) an url path that has a "folder" that does not exist on the hard drive so:  (2) the wiki end up with "hard-coded" addresses that don't exist on disk. Which means (3) you have to manually edit them to adjust if you export to single file. That is the central issue for me to solve for the kind of wiki I make, involving quite a lot of images.

Regarding me using the word "virtual". I don't really mind what word is used actually. But the point was to help users, not familiar with node/Bob that pathing to external local files do use a "fictive entity" in the path (needed in the world of Bob/node, but a bit of a drag in any other). *I* found that hard to grasp. And I think others would too, at first.

Best wishes
Josiah

[Mark S.]

The solution would be if Bob used a special path name (virtual path) for the Wikis, and not for images. That way the two path types could be distinguished without making Bob incompatible with other uses of TW.

-- Mark