[TW5] Bob and BobEXE version 1.4.0-beta, Garden Gnomes, lazy loading and file stuff

[Jed Carty]

Due to the changes in this update it has a higher than normal chance of having weird bugs. Back up your data and please make bug reports. That is why it is labelled beta.

This update changes a lot on the back end. On the front end the changes are mainly updates to the configuration UI so there is almost no reason to use the manual settings tab.

The highlights of this update are:

- Lazy loading
- The ServerImages plugin is now a part of Bob
- Bob uses the core syncer for saving and loading
- The BobSaver part only accepts '.html', '.htm' and '.hta' files (a security thing, it shouldn't affect anyone)
- Added an option to disable the file system monitor, this can help with network drives and if you are using an external file syncer. Using external file syncing applications may still cause trouble.
- UI improvements for the file server configuration and other configuration options.

The plugin version of Bob is on GitHub here: https://github.com/OokTech/TW5-Bob
The newest version of BobEXE is available here: https://github.com/OokTech/TW5-BobEXE/releases 

If you want to support the development OokTech has a patreon page here https://www.patreon.com/OokTech
or if you prefer there is a link for PayPal here https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=ZG94CTLHTKYRE

The full changelog is here:

!! Version 1.4.0-beta Garden Gnomes

''Note:'' this version changes some parts of how tiddlers are saved to the file system. There shouldn't be any problems but there is always a risk of data loss when making changes like this. Make sure to backup your data.

• Implemented lazy loading
  ** You can enable/disable lazy loading in the Bob Settings tab of the $:/ControlPanel
  ** Documentation is under the Bob Settings tab in the control panel with the other documentation
• Setup saving and loading in normal syncadaptors to work better with the core
  ** The dirty indicator is now handled by the core, so it should behave better now
  *** Saving now takes longer, but 'longer' is about 1 second instead of about 1/10th of a second.
  **** This is because of a delay built into the core, it isn't a bug. In future versions of tiddlywiki this will be configurable.
  ** The 'there are unsaved changes' message when you close a tab is also now handled by the core, so it should behave better as well.
• finished moving all of the functionality of the ServerImages plugin into Bob.
  ** This means that the ServerImages plugin is no longer necessary.
  ** Information about the new setup is available in the Bob Documentation called Server Media
  ** The server media part is enabled by default if you turn on the file server. It can be disable in the $:/ControlPanel under Bob Settings -> Server -> File Server
  *** The server media component only works if the file server is enabled, if it isn't enabled than the server media component has no effect.
• Changed WebsocketAdaptor.js to MultiWikiAdaptor.js because it has nothing to do with websockets, it handles saving and loading when you have multiple wikis.
• Removed some unused template files
• Switch to using more asynchronous functions
  ** changing multiple functions in the file system monitor to use asynchronous versions
  *** While this prevents some race conditions and reduces the places where writing tiddler changes can be delayed, it introduces another race condition that arises when you rename a tiddler or a file on the file system so that the tiddler file name does not follow the rules used by the wiki. The file system monitor deletes the incorrectly named file and creates a new file with the expected name. The file watcher starts acting before the file is completely deleted so it acts as though the file still exists which leads to an error being thrown. The error is handled but there may be unexpected behaviour is situations where tiddler titles or file names are changed in the file system. Back up your data.
• Modified the BobSaver so the server only accepts files with the extensions .html, .htm and .hta
  ** Changing the host for the BobSaver requires that you agree to not ask for tech support and understand that none will be given by filling out the acceptance part of the configuration
• Added a checkbox to the server tab to enable or disable saving media on the server and creating a tiddler with a _canonical_uri field when importing media into the wiki
• ''(BobEXE only)'' Added a checkbox to enable or disable automatically opening up a browser when BobEXE starts. This has no effect if you are using the plugin version.
• Checkboxes that change settings no longer require you to click a button to save the change
  ** Text entry boxes still require you to click a button to save the updated settings.
• Added/updated documentation about the saver server and the media server (the replacement for the ServerImages plugin.)
• ''(BobEXE only)'' Added command line arguments to set the location of the index wiki and the wikis folder.
• Added an option to disable the file system watcher
  ** This may help reduce memory usage further if you don't ever edit tiddlers on the file system
  ** ''Using syncing software with your wiki folders is not recommended, it can cause dulpicate files and data loss, Bob and the syncing software fight each other when changing files.'' This may help reduce conflicts if you are using syncing software for your wiki folders.
  ** This helps prevent bugs that arise from not being able to properly monitor network drives.
  *** This is a limitation of networked drives, it is not specific to Bob. I may write an alternate filesystem monitor that can function with network drives but it will necessarily be slower and have the potential to miss changes made on the file system. This may be unavoidable on networked drives.

[TonyM]

Jed,

Your are very prolific, I will be sending you a patreon donation today and Arlen as well and making something of it to encourage others in this season. Unfortunately the Australian dollar is not so strong. 

Thanks for the developments in this release I look forward to exploring the opportunities it opens up.

Most savers or servers current support the .tw extencion so if it possible you could include this in the allowable file extensions, I would appreciate it, no hurry just when convenient to you. No need to even document it. This relates to using tiddlywiki as smart documents and being able to attach them to emails as well as choose the application or server that opens them from a desktop perspective.On Windows this allows double click to open rather than "Open with and select application".

Regards

Tony

[Dave]

Looking forward to the lazy loading :)

I tried it after backup, and couldn't find any tiddlers, so I went back to the previous Bobexe.  I'll check again in a while...

PS, I wonder if this is related to me using syncthing to sync between work and home - my created and modified dates seem to perpetually disappear, so that might be related (also the order of dates in the "recent" tab are often out of order) <- but that was happening before I switched to BobExe from the normal Bob

[Stobot]

Thanks Jed!

When trying the [] Disable File System Monitor I'm getting an error "Failed to update settings with error: SyntaxError: Unexpected token o in JSON at position 1"

Using BOBEXE Windows 64-bit downloaded a couple of hours ago.

[Stobot]

It also looks like I cannot import images at all - either with our without Server Images enabled. In both circumstances, it looks like it will work, and then nothing happens. I've attached gifs in case I'm not explaining it well.

Attachments I had don't seem to animate, here's hosted on Imgur: https://imgur.com/a/AjIp3Z0

[Jed Carty]

Stobot,

It looks like there is some problem with the server routes that I missed, I will try and get a fix for that tomorrow.

Dave,

Using an external sync program with wikis that Bob is serving will cause trouble.

I need more information about the lazy loading part, I haven't had any trouble with lazy loading on anything I have tried.

[Jed Carty]

oh, and the disable file system monitor part is set up incorrectly, I have a fix for that too, hopefully it will be up tomorrow as well.

[Stobot]

Great! Thanks Jed as always.

[Dave]

Re the lazy loading, what is the expectation for normal operation?

I imagine a fast load of the default tiddllers and menu tiddlers and system/macro tiddlers, and then all the "normal" tiddlers load silently in the background over the next 10-30 seconds or so?

I have approx 4000 tiddlers in mine.  With previous editions its not so bad loading in a PC browser, but loading on mobile is painful.

[Jed Carty]

For the future I would greatly appreciate it if people would please follow some guidelines about making bug reports. https://ooktech-public.gitlab.io/guidelines/

I am working on collecting examples of effective and ineffective bug reports and feature requests I have received to add to it.

Dave,

Lazy loading sends everything but the text field of tiddlers that haven't been used yet. For more detail than that you would have to ask one of the devs that has more knowledge of it than me, I think that Jeremy, Mario and Arlen know more about it. I am sure there are others as well.

[Dave]

Oh, ok, sorry - I thought it was entirely a Jed brainchild.  I'll start searching this forum for info.

[Jed Carty]

I pushed out a new version, 1.4.0 beta 4

It should fix problems with the file server and media scanning parts. The checkbox to turn off the file system monitor should actually turn off the file system monitor.

Note that when you use the media scanning part it can take a long time to save all of the tiddlers if there are a lot of media files. It doesn't copy or affect the media files themselves but it creates a _canonical_uri tiddler for each media file in a folder and if you have hundreds or thousands of them it can take a while to get all of them.

Remember that you have to restart Bob before changes to the file server fully take effect. Some parts may work without restarting, but it is inconsistent and you shouldn't trust that it will work without restarting Bob first.

When there is an error you should get an alert telling you to restart Bob, so you will get a reminder if you forget to restart.

[Stobot]

Both of my previously mentioned issues have been fixed in 1.4.0 beta 4 as far as I can tell (64bit BOBEXE Windows)!

• [] Disable File System Monitor
• Now changes without error message
• Media server
• Now images importing normally

Thanks for the support!

[TonyM]

Jed,

I also had a fault, I did not have the chance to report, where imports would fail to create the imported tiddlers, but this is no longer occurring on 1.4.0 beta 4 as far as I can tell (64bit BOBEXE Windows)!

Thanks