Announcing the new "Browser-Storage" plugin

[Jeremy Ruston]

Further to Mat’s discovery of the new mathematics operators in v5.1.20, there’s another newly added feature that has also been discussed for quite a long time: a plugin that lets TiddlyWiki save and load tiddlers from browser’s built in “local storage”.

At startup, the plugin reads tiddlers from local storage. Any tiddlers that are identical to those built into the file are deleted from local storage. Once the wiki is up and running, any tiddler changes are written straight to local storage.

You can install it using the plugin library in a v5.1.20-prelease wiki, or there’s an example here that has it pre-installed:

https://tiddlywiki.com/prerelease/editions/full/#%24%3A%2Fplugins%2Ftiddlywiki%2Fbrowser-storage

You’ll see that even though the wiki warns you when you refresh in the usual way, once you reload it any changes you’ve made will reappear.

Browser local storage is not a panacea for TiddlyWiki:

• Browsers limit the amount of local storage available to a page, typically to 5 or 10MB

• Keeping personal data in browser local storage can lead to unexpected privacy violations

• Browsers reserve the right to without warning delete data stored in local storage at any time

• Browsers tie local storage to a URL which can lead to problems if you move a wiki to a URL previously occupied by a different wiki

The plugin isn’t quite finished; I want to add a more visible indicator for when data is stored in local storage, and a way to delete any stored data and disable it for the future.

As usual, questions and comments welcome,

Best wishes

Jeremy.

[Mark S.]

Hi Jeremy,

That sounds pretty neat!

When you say "5 to 10 mb" per page, how does that translate into the tiddler world? Is the entire TW one page, or each tiddler? If the latter, then it's almost no limit at all.

Hopefully there will be an export mechanism?

If you run encryption, will the internal tiddlers also be encrypted?

What I found with selfnote was that it didn't cleanly delete itself. That is, after installation, and then deletion, you couldn't re-install, presumably because there were still bits of the last install floating around.

Thanks!

-- Mark

[@TiddlyTweeter]

With all the caveats (I agree with) would this method obviate the need for the user to install a saving system? I.e. click wiki and its saved auto into browser without any further steps?

[Jeremy Ruston]

With all the caveats (I agree with) would this method obviate the need for the user to install a saving system? I.e. click wiki and its saved auto into browser without any further steps?

It is categorically not suitable as a way for most users to save changes to their own wikis. The fundamental problem is that browsers are free to "evict" local storage whenever they want, and we get no warning of it.

Browsers added this feature to give web developers a way to improve performance through locally caching data pulled from the server. The assumption that the data is volatile data and can be retrieved from the server if it is lost, runs through the implementation.

Having said that, local storage can still be very useful, including things like experimentally loading plugins without having to save before reloading.

I plan to improve it a bit further, in particular I want to get it working well in the client server configuration so that local storage can be used as a temporary holding area in case the connection to the server is lost.

Best wishes

Jeremy 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/58dfcf33-f02c-4c15-ace4-16eaea3c3616%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[TonyM]

Jeremy,

That has a great deal of utility. I was working on a method to export comments and re-import them into to master copy of a wiki to then apply them to the master. The same can be done for any new or modified tiddlers using local storage. 

Local storage raises the following for me

1. Can we differentiate tiddlers modified via local storage and those not,  including across multiple sessions - NoteSelf seems to keep the haschanged flag for such tiddlers (along with versions)
   
2. With a Wiki + Local storage changes, we should be able to provide easy access to a set of "Big Green Save Buttons"
   

• Save Empty based on this Version
• Save As Served without tiddlers from local Storage (When someone wants to make an Edition out of the served Wiki).
• Save As Served Plus tiddlers from local Storage (allows save in current state) and to leave local storage method
• Save only new or changed tiddlers stored in Local Storage 
• Export changes and could be re-imported into another browser/session
• Can send to master wiki owner for selective import.
• Could include a filtered export eg export only comments.

With the introduction of a local storage feature I believe all of the above are relatively trivial to achieve however they will add great utility to tiddlywiki and users. They could be included in the local storage plugin but in reality the permit emergency save, export changes and more that would help in most use cases. Perhaps a set of "Big Green Buttons"  can be placed in ControlPanel > Saving > General

Regards

Tony

[TonyM]

Jeremy,

Another Question in relation to this what about leveraging pouchDB as noteself has, which can I believe remove the volatile nature of the local storage you describe? this is without needing to provide the external couchDB component thus no configuration settings.

Regards

Tony

On Friday, February 8, 2019 at 9:11:47 AM UTC+11, Jeremy Ruston wrote:

[Jeremy Ruston]

Hi Tony

Another Question in relation to this what about leveraging pouchDB as noteself has, which can I believe remove the volatile nature of the local storage you describe? this is without needing to provide the external couchDB component thus no configuration settings.

I'm afraid PouchDB uses the same local storage mechanism, and is subject to exactly the same restrictions if it's not connected to a server.

Best wishes

Jeremy.

Regards

Tony

On Friday, February 8, 2019 at 9:11:47 AM UTC+11, Jeremy Ruston wrote:

Further to Mat’s discovery of the new mathematics operators in v5.1.20, there’s another newly added feature that has also been discussed for quite a long time: a plugin that lets TiddlyWiki save and load tiddlers from browser’s built in “local storage”.

At startup, the plugin reads tiddlers from local storage. Any tiddlers that are identical to those built into the file are deleted from local storage. Once the wiki is up and running, any tiddler changes are written straight to local storage.

You can install it using the plugin library in a v5.1.20-prelease wiki, or there’s an example here that has it pre-installed:

https://tiddlywiki.com/prerelease/editions/full/#%24%3A%2Fplugins%2Ftiddlywiki%2Fbrowser-storage

You’ll see that even though the wiki warns you when you refresh in the usual way, once you reload it any changes you’ve made will reappear.

Browser local storage is not a panacea for TiddlyWiki:

• Browsers limit the amount of local storage available to a page, typically to 5 or 10MB

• Keeping personal data in browser local storage can lead to unexpected privacy violations

• Browsers reserve the right to without warning delete data stored in local storage at any time

• Browsers tie local storage to a URL which can lead to problems if you move a wiki to a URL previously occupied by a different wiki

The plugin isn’t quite finished; I want to add a more visible indicator for when data is stored in local storage, and a way to delete any stored data and disable it for the future.

As usual, questions and comments welcome,

Best wishes

Jeremy.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/e28a6f28-0a73-4174-a070-e7fde47ba8cd%40googlegroups.com.

[Jeremy Ruston]

Hi Tony

1. Can we differentiate tiddlers modified via local storage and those not,  including across multiple sessions - NoteSelf seems to keep the haschanged flag for such tiddlers (along with versions)
   

The list of tiddlers that was loaded from local storage at startup is available in a config tiddler (see the settings tab).

2. With a Wiki + Local storage changes, we should be able to provide easy access to a set of "Big Green Save Buttons"
   

• Save Empty based on this Version

We have that.

• Save As Served without tiddlers from local Storage (When someone wants to make an Edition out of the served Wiki).

Not so easy. The original values that were overwritten by local storage are no longer available.

• Save As Served Plus tiddlers from local Storage (allows save in current state) and to leave local storage method

We have that. We don't clear local storage at the time of saving, but at the time of reloading.

• Save only new or changed tiddlers stored in Local Storage 

That's the same as saving tiddlers that have been changed since the last reload.

• Export changes and could be re-imported into another browser/session
• Can send to master wiki owner for selective import.
• Could include a filtered export eg export only comments.

With the introduction of a local storage feature I believe all of the above are relatively trivial to achieve however they will add great utility to tiddlywiki and users. They could be included in the local storage plugin but in reality the permit emergency save, export changes and more that would help in most use cases. Perhaps a set of "Big Green Buttons"  can be placed in ControlPanel > Saving > General

Perhaps the "save empty" button is worth putting in control panel, I'm not sure about the other examples you mention.

Best wishes

Jeremy

Regards

Tony

On Friday, February 8, 2019 at 9:11:47 AM UTC+11, Jeremy Ruston wrote:

Further to Mat’s discovery of the new mathematics operators in v5.1.20, there’s another newly added feature that has also been discussed for quite a long time: a plugin that lets TiddlyWiki save and load tiddlers from browser’s built in “local storage”.

At startup, the plugin reads tiddlers from local storage. Any tiddlers that are identical to those built into the file are deleted from local storage. Once the wiki is up and running, any tiddler changes are written straight to local storage.

You can install it using the plugin library in a v5.1.20-prelease wiki, or there’s an example here that has it pre-installed:

https://tiddlywiki.com/prerelease/editions/full/#%24%3A%2Fplugins%2Ftiddlywiki%2Fbrowser-storage

You’ll see that even though the wiki warns you when you refresh in the usual way, once you reload it any changes you’ve made will reappear.

Browser local storage is not a panacea for TiddlyWiki:

• Browsers limit the amount of local storage available to a page, typically to 5 or 10MB

• Keeping personal data in browser local storage can lead to unexpected privacy violations

• Browsers reserve the right to without warning delete data stored in local storage at any time

• Browsers tie local storage to a URL which can lead to problems if you move a wiki to a URL previously occupied by a different wiki

The plugin isn’t quite finished; I want to add a more visible indicator for when data is stored in local storage, and a way to delete any stored data and disable it for the future.

As usual, questions and comments welcome,

Best wishes

Jeremy.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/76286393-aadf-4d03-a450-40ffd6cbc45a%40googlegroups.com.

[TonyM]

Jeremy,

Interesting, I have had no reason to consider PouchDB storage to be volatile in anyway, as is suggested on the NoteSelf site. Nor have I lost the data this way. My experience suggests I can trust local storage, your outline suggests I should not. 

I'm afraid PouchDB uses the same local storage mechanism, and is subject to exactly the same restrictions if it's not connected to a server.

If local storage is volatile and undependable it undermines the trust we need to use such a feature.

Regards

Tony 

[Jeremy Ruston]

Hi Tony

Interesting, I have had no reason to consider PouchDB storage to be volatile in anyway, as is suggested on the NoteSelf site. Nor have I lost the data this way. My experience suggests I can trust local storage, your outline suggests I should not. 

I'm afraid PouchDB uses the same local storage mechanism, and is subject to exactly the same restrictions if it's not connected to a server.

If local storage is volatile and undependable it undermines the trust we need to use such a feature.

The trouble with local storage is that it works fine day after day until suddenly it doesn't. It might be because of something inscrutable that the browser has done when it detects a shortage of disc or memory space, or it might be yourself clearing history without remembering that doing so also nukes local storage...

There's nothing wrong with PouchDB, it uses local storage in entirely the most appropriate way: as a local cache to add a layer of protection in case of network issues or crashes.

Best wishes

Jeremy.

Regards

Tony 

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/600d6cda-62c1-4167-84d5-460f793db539%40googlegroups.com.

[TonyM]

Jeremy,

In relation to

• Save As Served without tiddlers from local Storage (When someone wants to make an Edition out of the served Wiki).

Not so easy. The original values that were overwritten by local storage are no longer available.

Could we use the innerwiki plugin or iframe to spawn a window containing the fresh tiddly wiki loaded from disk "before" local storage is used, and present a save button? 

 

And inrelation to

Perhaps the "save empty" button is worth putting in control panel, I'm not sure about the other examples you mention.

To me this "save empty" is the least helpful "Big Green Button", but with the others you say will work, where are they?, do users have access to them or must they "develop" them? I am keen to have them clearly available, even if only in a plugin or documentation.

Thanks for your feedback

tony

[TonyM]

Jeremy,

The trouble with local storage is that it works fine day after day until suddenly it doesn't. It might be because of something inscrutable that the browser has done when it detects a shortage of disc or memory space, or it might be yourself clearing history without remembering that doing so also nukes local storage...

So we can say local storage is fairly reliable, at least in the current browser session, so perhaps providing the export changes button that can be re-imported, can offer a form of changes only backup, which can use the standard download mechanisium, and since the file may never be accessed the download mechanism is sufficient.

As long as a Backup/restore changes and full wiki option, is clear and in front of users, I do not think this volatility will be an issue.

Regards

Tony

[Brian Theado]

Jeremy,

On Thu, Feb 7, 2019 at 5:56 PM Jeremy Ruston <jeremy...@gmail.com> wrote:

[...] 

I plan to improve it a bit further, in particular I want to get it working well in the client server configuration so that local storage can be used as a temporary holding area in case the connection to the server is lost.

Will it also handle the case of deleting a tiddler (one which originated from the server and isn't in localstorage) while disconnected? I would think you would need some sort of "whiteout" tiddler in localstorage in order to handle that case.

I ask the question because of this workflow I thought of:

1. file:/// wiki with this plugin enabled
2. The plugin automatically saves changes to localstorage as they are made
3. Every once in a while (several times per day), the user clicks the save button which downloads a backup of the full wiki
   
4. Less frequently, at the OS level, the user copies the latest downloaded version over top file:/// they started with and refreshes browser
5. This refresh will cause the localstorage items to be deleted, starting over fresh

But there will be cases with this workflow and your current implementation where deleted tiddlers would reappear after refresh.  Deletes of any tiddler which exists in the wiki after step 4 would persist only until the next refresh.

Brian

[Jeremy Ruston]

Hi Tony

Could we use the innerwiki plugin or iframe to spawn a window containing the fresh tiddly wiki loaded from disk "before" local storage is used, and present a save button? 

The original tiddlers are available in the store area in the DOM, and so it would be possible to retrieve them again.

(An iframe loading an entire TiddlyWiki is highly resource intensive, and so we wouldn’t want to do that in the core).

And inrelation to

Perhaps the "save empty" button is worth putting in control panel, I'm not sure about the other examples you mention.

To me this "save empty" is the least helpful "Big Green Button", but with the others you say will work, where are they?, do users have access to them or must they "develop" them? I am keen to have them clearly available, even if only in a plugin or documentation.

“Save as served” is tricky: it’s hard to guarantee that we’ll be saving exactly what was served. For situations where users want that it would be better to just download the file from the original location. We can do that with a simple download link.

“Save only new or changed” doesn’t seem that useful, or at least I don’t see why it’s connected with the browser-storage plugin. But in any event for things like it makes sense to reuse the advanced search export function: we can provide a dropdown for “new or changed tiddlers” and then the user can export the results.

Best wishes

Jeremy.

Thanks for your feedback

tony

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/711c7e75-c71a-42be-b3f7-007bf40824f6%40googlegroups.com.

[Jeremy Ruston]

Hi Tony

So we can say local storage is fairly reliable, at least in the current browser session,

I do not share the view that browser local storage is reliable :)

so perhaps providing the export changes button that can be re-imported, can offer a form of changes only backup, which can use the standard download mechanisium, and since the file may never be accessed the download mechanism is sufficient.

But why wouldn’t you just use the existing save changes button to save a fully functional copy of the wiki? What use is a file that just contains some of the tiddlers you were working on? That seems like an edge case.

As long as a Backup/restore changes and full wiki option, is clear and in front of users, I do not think this volatility will be an issue.

Which users are we talking about? I’m arguing that local storage is not a universal solution, and that it should only be recommended in specialised situations where there is a low likelihood of data loss.

Best wishes

Jeremy.

Regards

Tony

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/5d121f20-6a12-4a90-aac5-8d17ac168e9d%40googlegroups.com.

[Jeremy Ruston]

Hi Brian

Will it also handle the case of deleting a tiddler (one which originated from the server and isn't in localstorage) while disconnected? I would think you would need some sort of "whiteout" tiddler in localstorage in order to handle that case.

That is listed in the readme as a problem with the current code, and indeed the solution is to keep a delete marker in local storage.

I ask the question because of this workflow I thought of:

1. file:/// wiki with this plugin enabled
2. The plugin automatically saves changes to localstorage as they are made
3. Every once in a while (several times per day), the user clicks the save button which downloads a backup of the full wiki
   
4. Less frequently, at the OS level, the user copies the latest downloaded version over top file:/// they started with and refreshes browser
5. This refresh will cause the localstorage items to be deleted, starting over fresh

But there will be cases with this workflow and your current implementation where deleted tiddlers would reappear after refresh.  Deletes of any tiddler which exists in the wiki after step 4 would persist only until the next refresh.

That is indeed exactly the intended workflow.

Best wishes

Jeremy

Brian

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/CAO5X8Cwu6TVD47BunWE9sY9wbf7t_s%2B2FmOtOqOjbrKoJ9wtyg%40mail.gmail.com.

[BurningTreeC]

Hi Jeremy, I think this is a nice addition to the core plugins and I can see cases where it could be very useful to me

I often open tiddlywiki.com or its prerelease to try out something and end up in a list of tiddlers that contain code I'd like to copy over to a personal wiki

It would be cool if the browser plugin could provide a tools button to export its tiddlers to a (json?) file for easy import in another wiki

all the best,

BTC

[Jeremy Ruston]

Hi Mark

On 7 Feb 2019, at 22:34, 'Mark S.' via TiddlyWiki <tiddl...@googlegroups.com> wrote:

If you run encryption, will the internal tiddlers also be encrypted?

No, the browser-storage plugin doesn’t handle encryption at the moment; I’ll add a warning to the readme.

Best wishes

Jeremy.

[Jeremy Ruston]

Hi BurningTreeC

Hi Jeremy, I think this is a nice addition to the core plugins and I can see cases where it could be very useful to me

Thanks! I’m also finding it very useful while I’m working on TiddlyWiki coding, makes it super easy to get a wiki back into the same state for testing.

I often open tiddlywiki.com or its prerelease to try out something and end up in a list of tiddlers that contain code I'd like to copy over to a personal wiki

I’m torn as to whether we should enable it on tiddlywiki.com; I think not, because of the dangers we’ve been discussing, and yet I’d love to have an easily accessible up to date wiki with it. For the moment that is:

https://tiddlywiki.com/prerelease/editions/full/#%24%3A%2Fplugins%2Ftiddlywiki%2Fbrowser-storage

It would be cool if the browser plugin could provide a tools button to export its tiddlers to a (json?) file for easy import in another wiki

I think that’s useful independently of browser storage. I’ve added a new canned filter for advanced search that yields the tiddlers modified in the current session:

https://github.com/Jermolene/TiddlyWiki5/commit/0e6855eba8945e023cc5ef19809cfb6d6025627b

Best wishes

Jeremy.

all the best,

BTC

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/e783465e-ad1a-48e0-9f0d-398ebdf493e1%40googlegroups.com.

[Brian Theado]

Thanks, Jeremy. I missed that bullet point.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/3CB84210-3C76-4E88-A514-8E0189801630%40gmail.com.

[Xavier Cazin]

Hi Jeremy,

I’m torn as to whether we should enable it on tiddlywiki.com; I think not, because of the dangers we’ve been discussing, and yet I’d love to have an easily accessible up to date wiki with it. For the moment that is:

https://tiddlywiki.com/prerelease/editions/full/#%24%3A%2Fplugins%2Ftiddlywiki%2Fbrowser-storage

It works superbly!

Maybe even a bit too well: when we add new tiddlers then reload the wiki despite the warning, it let us retrieve the newly added content as miraculously expected, but then the wiki is not marked as tainted anymore, which seems misleading. I think the red alert should probably remain until the wiki is actually saved.

Cheers,

Xavier.

[PMario]

Hi,

Some more info about the storage limits. If you take your time an read this stuff, you'll see that the behaviour is completely unpredictable. 

And in my opinion way to fragile for our usecases. ... read on, I'll show you why!

 - open: https://tiddlywiki.com/prerelease/editions/full

 - F12 open the browser debug tools

 - select the Storage tab in the top menu

 - in the left menu, open the Local Storage tab

 - Select http://tiddlywiki.com

 - You'll see the different tiddlers stored there! ... in plain text :/ ... that's what we expected.

But what follows is a complete mess :/

 - open http://tiddlywiki.com

 - Have a look at the Local Storage

 - All the stuff from prerelease/editions/full is still visible -> WTF

 - even worse!!!!

 - go to https://tiddlywiki.com/prerelease/editions/ ... which returns a 404 page

 - have a look at the local storage -> you can still read the stuff

 - So the only thing you need to know is tiddlywiki.com and you can read the local storage from all TWs stored in the subdirectories.

From my point of view, this mechanism is way to fragile atm, to be used properly.

just some thoughts

have fun!

mario

[@TiddlyTweeter]

Jeremy Ruston wrote:

I wrote: With all the caveats (I agree with) would this method obviate the need for the user to install a saving system? 

It is categorically not suitable as a way for most users to save changes to their own wikis. The fundamental problem is that browsers are free to "evict" local storage whenever they want, and we get no warning of it.

I agree. An additional issue is backup. Its totally not intuitive figuring out what a browser is storing where. External backup is needed, which brings it back to the basic saving issue. A major downside is easy portability goes.

Where I can see a clear use case is, say, a student who just needs fill in a form in the wiki that, on completion, is emailed to me. Its a short term task that may be done in more than one sitting ... needs storage ... but not for long. Chances of meltdown minimal.

Just thoughts

Josiah

[@TiddlyTweeter]

PMario wrote:

Some more info about the storage limits. If you take your time an read this stuff, you'll see that the behaviour is completely unpredictable. 

And in my opinion way to fragile for our usecases. ... 

I agree.

I think there are a limited set of use cases where it could work usefully. But for longer-term saving of a wiki it is not a good strategy  IMO.

FYI, when NoteSelf first appeared I looked into it and found understanding browser storage very difficult. NoteSelf is a really great tool, but I'd never use it without a CouchDB remote database its constantly syncing with.

My two cents

Josiah

[Jeremy Ruston]

Hi Mario

On 8 Feb 2019, at 13:53, PMario <pmar...@gmail.com> wrote:

 - So the only thing you need to know is tiddlywiki.com and you can read the local storage from all TWs stored in the subdirectories. 

That’s a bug, fixed now:

https://github.com/Jermolene/TiddlyWiki5/commit/2710fae71d276c8a82e78d4f13b0d2735e0b71f5

Best wishes

Jeremy.

[Jeremy Ruston]

Hi Xavier

On 8 Feb 2019, at 12:12, Xavier Cazin <caz...@gmail.com> wrote:

Maybe even a bit too well: when we add new tiddlers then reload the wiki despite the warning, it let us retrieve the newly added content as miraculously expected, but then the wiki is not marked as tainted anymore, which seems misleading. I think the red alert should probably remain until the wiki is actually saved.

Ah, interesting problem! Hopefully now fixed here:

https://github.com/Jermolene/TiddlyWiki5/commit/6ff7a7d2616efc0bae7ac73dec9dc92132b4c129

Best wishes

Jeremy.

[TonyM]

Jeremy et al,

This subject is very interesting and the afformentioned plugin may have its place. The utility of saving in the browser reliably is very powerful, as a result I hope the conversation continues to this end. Looking at pouchdb that uses indexdb seems particularly strong especialy in firefox which will prompt if you exceed the basic limits on top of your local disk availability.

I read the following;

In Firefox, PouchDB uses IndexedDB. Though Firefox has no upper limit besides disk space, if your application wishes to store more than 50MB locally, Firefox will ask the user using a non-modal dialog to confirm that this is okay.

It is true that other local save mechanisms and browsers are much less reliable because they treat local storage more like a cache, its disposable in some circumstances.

Given my expierience with variouse cloud services that have local and offline services I am confident there are reliable solutions available, we just need to find them, options that have a low or zero complexity to the user.

Regards
Tony

[Jeremy Ruston]

Hi Tony

This subject is very interesting and the afformentioned plugin may have its place. The utility of saving in the browser reliably is very powerful, as a result I hope the conversation continues to this end. Looking at pouchdb that uses indexdb seems particularly strong especialy in firefox which will prompt if you exceed the basic limits on top of your local disk availability.

It appears that Mozilla plan to do the same prompting for localstorage quotas:

https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria

For a long time, the problem with IndexedDB is that it wasn't supported on all browsers, in particular Safari took a long time to implement it and then for years their implementation was so bug ridden that nobody could use it. That's why PouchDB for years shipped with two storage adaptors: IndexedDB and the weird and incompatible database supported by old Safaris.

Now the situation is a little different and it appears that we do have reasonable cross-browser support for IndexedDB:

https://caniuse.com/#feat=indexeddb

But I don't think it's worth switching from LocalStorage for us. The main advantages of IndexedDB over LocalStorage is that it is a full database with querying capabilities, but in our application those features aren't used.

I read the following;

In Firefox, PouchDB uses IndexedDB. Though Firefox has no upper limit besides disk space, if your application wishes to store more than 50MB locally, Firefox will ask the user using a non-modal dialog to confirm that this is okay.

It is true that other local save mechanisms and browsers are much less reliable because they treat local storage more like a cache, its disposable in some circumstances.

I don't think the size limit is the sticking point with browser storage: much more serious constraints are the risk of eviction and the privacy implications.

Given my expierience with variouse cloud services that have local and offline services I am confident there are reliable solutions available, we just need to find them, options that have a low or zero complexity to the user.

These browser storage technologies are pretty well understood, and the roles it can play are clear: caching material that has been retrieved from a server, caching stuff that will be sent to a server, and remembering settings.

For us, it's useful in some situations, but it's not a universal panacea to the challenges of saving changes.

Best wishes

Jeremy.

Regards
Tony

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/cdba024d-1b0a-44a7-9f5e-6d406a86ba33%40googlegroups.com.

[PMario]

That's OK. ... BUT ... My biggest concern is privacy. ... Now I get even more info about the underlying structure of a webpage.

eg: If users with different privileges share a PC, a user which has access to the main origin can see, the subdirectory structure, plus "some" content in plain text.

Some setups may use "un-guessable" subdirectory names to store data. This "un-guessable" directory is plaintext, as soon as someone (with the right privileges) ever opened the subdirectory. ... I know, that "security by obscurity" is used, because it's simple to implement. ...

I just want to make sure, that our users know about these problems!

have fun!

mario

[Ste Wilson]

Sorry if this has already been said, but doesn't this make things a little more noteself?

[@TiddlyTweeter]

Its Noteself-ish. But without external CouchDB.

The point being that Noteself, though it can work just in browser, it is also, importantly, a syncing system. In Sync mode with an external CouchDB its much more robust.

J.

[Mark S.]

Actually I'm puzzled what the use case might be. If you can't trust your data in it for long, and if there is no synchronization, then what are the use cases? Mostly I could see it in an environment where every other save technique is disallowed and where you don't mind if the data could be exposed. Perhaps a system where your only connection to the outside world is via email. Perhaps a mechanism for using TW for collating and processing information, that then gets copied and pasted elsewhere. What else .... although your data isn't secure, if you've ever rifled through your FF profile for information (like I did with NoteSelf), it's definitely obscure. Obscure may be good enough when all you want is to not leave a trail of files behind.

Other ideas?

-- Mark

There's another product, Laverna, that uses local and/or indexed storage as well. They tout that they have an emphasis on privacy, because your data isn't on the web.

[TonyM]

Mark,

I am experimenting with local storage over here. Anyone can use this as a platform to test Local storage and play with the pre-release as at 2019-02-20 so have a look.

I have no doubt it will have a number of great use cases. 

• One is simply permitting trivial search and settings the visitor uses to be saved for your return
• In this case it is trivial if the settings are lost on return
• Local storage saved Pallets, themes, toolbar settings etc... make the experience much better.
• Remember it only saves the changes from the loaded version so it needs only the space required to store these changes
• Imagine if I had a "Home Loan Calculator", the user details are small in volume.
• The Result of the computation could be printed so the result is captured by the user, 
• If the local storage goes (by my calculation very unlikely) they can enter it again (if you do not promise they will not expect it)
• At any time the user can save the wiki in totality - they are in charge of their own risk aversion.

Issues Encountered

• If I export an updated wiki to file, then upload and replace the online copy with those changes, when I return to the URL it is still showing the local storage items which are now redundant
• I need to find a way to purge these
• I Found a Way to purge this 
• It is all good showing there are unsaved changes, but this would be unnecessarily confusing to casual users. I have thus hidden this in the DOcument Info tiddler.

Regards

Tony

[TonyM]

Issue?

On importing a tiddler, it is not being saved in Local Storage and thus is not available on reload?

Regards

Tony

[TonyM]

Another use case;

We can request a username be given, and saved in the browser session. That username will be there on return.

Local storage permits the saving of new tiddlers such as comments. If we get the user to provide their user name all their created tiddlers can be exported easily and such a package can be forwarded as a submission to the website and imported by or applied by the wiki owner with update access.

Regards

Tony

On Wednesday, 20 February 2019 13:22:58 UTC+11, TonyM wrote:

[TonyM]

Further feedback

Perhaps a setting in the local storage plugin, or a seperate session storage plugin to use session only storage

Session storage is per-origin-per-window-or-tab and is limited to the lifetime of the window. Session storage is intended to allow separate instances of the same web application to run in different windows without interfering with each other, a use case that's not well supported by cookies

• I am struggling a little with the workflow to configure and deploy Wikis with Local storage since once activated it starts using local storage.

Regards

Tony

[Jeremy Ruston]

Hi Tony

On 26 Feb 2019, at 05:01, TonyM <anthony...@gmail.com> wrote:

Further feedback

Perhaps a setting in the local storage plugin, or a seperate session storage plugin to use session only storage

Session storage is per-origin-per-window-or-tab and is limited to the lifetime of the window. Session storage is intended to allow separate instances of the same web application to run in different windows without interfering with each other, a use case that's not well supported by cookies

I’m not keen to support session storage because it makes things more complicated without addressing any new use cases.

• I am struggling a little with the workflow to configure and deploy Wikis with Local storage since once activated it starts using local storage.

You might find it helpful to set up a filter for which tiddlers are saved to local storage. A useful variant is to set $:/config/BrowserStorage/SaveFilter to [prefix[$:/state/]] and thus to only save state tiddlers.

Best wishes

Jeremy.

-- 
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/af265d40-69fe-461d-8fee-2b358bf4c85b%40googlegroups.com.

[TonyM]

FYI:

The developer update in FireFox 65 for developers may impact the proposed local storage and the Innerwiki plugins for good or bad

https://developer.mozilla.org/en-US/docs/Web/API/Storage_Access_API

Regards

Tony

On Friday, February 8, 2019 at 9:11:47 AM UTC+11, Jeremy Ruston wrote:

Further to Mat’s discovery of the new mathematics operators in v5.1.20, there’s another newly added feature that has also been discussed for quite a long time: a plugin that lets TiddlyWiki save and load tiddlers from browser’s built in “local storage”.

At startup, the plugin reads tiddlers from local storage. Any tiddlers that are identical to those built into the file are deleted from local storage. Once the wiki is up and running, any tiddler changes are written straight to local storage.

You can install it using the plugin library in a v5.1.20-prelease wiki, or there’s an example here that has it pre-installed:

https://tiddlywiki.com/prerelease/editions/full/#%24%3A%2Fplugins%2Ftiddlywiki%2Fbrowser-storage

You’ll see that even though the wiki warns you when you refresh in the usual way, once you reload it any changes you’ve made will reappear.

Browser local storage is not a panacea for TiddlyWiki:

• Browsers limit the amount of local storage available to a page, typically to 5 or 10MB

• Keeping personal data in browser local storage can lead to unexpected privacy violations

• Browsers reserve the right to without warning delete data stored in local storage at any time

• Browsers tie local storage to a URL which can lead to problems if you move a wiki to a URL previously occupied by a different wiki

The plugin isn’t quite finished; I want to add a more visible indicator for when data is stored in local storage, and a way to delete any stored data and disable it for the future.

As usual, questions and comments welcome,

Best wishes

Jeremy.

[Mark S.]

Is there a mechanism for deleting tiddlers from local storage, when you're sure you're done with them?

Thanks!

-- Mark

[Jeremy Ruston]

Hi Mark

Is there a mechanism for deleting tiddlers from local storage, when you're sure you're done with them?

I’ve just added a button to clear browser storage, and a checkbox to enable/disable its use. The new build should be available at https://tiddlywiki.com/prerelease in 6-7 minutes,

Best wishes

Jeremy

Thanks!

-- Mark

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/78ffbb14-b7f9-4182-86f7-047a6d8520f7%40googlegroups.com.

[TonyM]

Jeremy,

I imagine users may be a little reluctant to use this if they think they will loose tiddlers they have not saved to file. 

If this is the case perhaps 

Click this button to clear browser storage and disable its use:

all tiddlers loaded from local storage remain in this wiki until save or reload

If the above is in fact true or a variation there of.

Thanks

Tony

[TonyM]

Jeremy,

Understood

I’m not keen to support session storage because it makes things more complicated without addressing any new use cases.

• I am struggling a little with the workflow to configure and deploy Wikis with Local storage since once activated it starts using local storage.

You might find it helpful to set up a filter for which tiddlers are saved to local storage. A useful variant is to set $:/config/BrowserStorage/SaveFilter to [prefix[$:/state/]] and thus to only save state tiddlers.

Your addition of disable and clear, along with the filter resolves this in effect, thanks.

One could say that "local storage can emulate session storage" by simple saving only selected tiddlers relating to the user interface such as the user ID, settings and state tiddlers.

Here is how I understand it and may provide documentation: First draft

Does this work flow sound correct?

• Build your desired wiki offline where you can save all changes, customise and add plugins
• Install the Local storage plugin but disable browser storage on the plugins settings page
• You may want to hide the Save wiki button/indicator as this may confuse casual visitors
• You should perhaps turn off autosave Control Panel > Saving "Do not save changes automatically" so that read only/local storage only users, do not get save errors.
• If you are hosting the wiki on a read only or read/write host 
• Activate the plugin and save the whole wiki via the default save mechanism
• Upload to your hosting location (Note: a change in URL or file name will not have access to a prior use of local storage)
• The Visitors/Wiki will have access to the Local storage when they visit your "site" and can save changes to local storage
• If you are hosting the wiki on a site where you have read write access
• Follow the above instructions to install with or without activating the plugin "use browser local storage"
• Then as the wiki owner to make Permanent changes online
1. Visit the site, clear Local storage and disable "use browser local storage" (if necessary)
• Steps 1 and two are interchangable
2. Make your various "permanent" changes
3. You may want to include a message to subsequent online users that a change in the permanent wiki has occurred.
• In case they have modified existing tiddlers that have being saved in local storage.
4. Enable "use browser local storage".
   
5. Save the wiki using the save mechanism (Tick in a circle)
• Use whatever method is necessary to save over the existing file 
6. Reload the wiki
7. All uses of this Wiki will have access to the Local storage when they visit your "site" and can save changes to local storage

Users saving the wiki "offline"

• It is possible to build custom save buttons that will allow online users using local storage to export/save empty, full wikis (including local storage) and the permanent wiki (excluding local storage). 
• Similarly it is possible to export only tiddlers in local storage or that have changed since the last save or reload.

Regards

Tony

Thanks

 

[Mark S.]

You've mentioned wanting a back-up before doing batch changes. The problem with backups is that the data has to be stored somewhere, and a single operation could conceivably almost double the number of tiddlers in your collection.  I'm wondering if the storage plugin could provide a solution. The backup tiddlers would be directed into the browser storage area instead of the actual TW. Eventually the browser would reclaim the space, but by that time you would presumably have committed to the changes, so that would work out. Something to think about.

-- Mark

[TonyM]

Mark

To whom are you replying?

Regards
Tony

[Mark S.]

Sorry. The quote below was from TT, so aimed @Josiah.