TWc encryption / pwd protection

[Wolfgang]

Hi,

I wonder, if it would be feasible, to implement a password protection and encryption for TWc just like in TW5...

Maybe, there is already a plugin or sth for that...

Many thanks,

Wolfgang

[Okido]

Hello Wolfgang,

There is an encryption plugin for TWC, but it is old, no idea if it still usable.

The TWC can be found here: visualTW

Have a nice day, Okido

Op donderdag 25 mei 2023 om 09:57:53 UTC+2 schreef Wolfgang:

[Yakov]

Hi Wolfgang,

I have good news for you: I have seriously refactored and improved the plugin that Okido has mentioned. Since it's of interest to you, I'll prioritize releasing it once I finish releasing the version 2.9.4 and make other announcements.

Best regards,

Yakov.

четверг, 25 мая 2023 г. в 19:36:41 UTC+3, Okido:

[Yakov]

PS in fact, you can already give it a try: http://yakovlitvin.pro/TW/pre-releases/NewEncryptedVaultPlugin.txt

It's somewhat documented already, so just let me know if something's not clear.

It uses a contemporary Stanford Javascript Crypto Library, but still I suggest you not to rely on this solution as a strong security measure. It requires some updates and ideally an audit.

But in terms of usability it's considerably better: for instance, it uses a "password" input (although no "confirm password" input yet), allows to autofocus the input to decrypt (typical solution is to put it into the MainMenu or the SidebarOptions), it works correctly with different savers etc.

There's still a number of changes to be made, that's why it's a pre-release.

пятница, 26 мая 2023 г. в 13:51:39 UTC+3, Yakov:

[Wolfgang]

Hi Yakov,

I tried your refactored version of the EncryptedVaultPlugin. It is perfect, works fine, just a little bit "slow" with 18.000 tiddlers. I will see, if I can "live" with that...

Best regards,

Wolfgang

[Yakov]

Hi Wolfgang,

I see, haven't tested performance for such cases yet. Do you really want to encrypt all the 18K tiddlers? You can use ListUnencrypted and filters to specify a smaller subset of tiddlers that you'd like not to compromise – this may improve the performance considerably.

Best regards,

Yakov.

воскресенье, 28 мая 2023 г. в 09:18:17 UTC+3, Wolfgang:

[Wolfgang]

Hi Yakov,

I tried the approach you mentioned, made a Filter "[tag[Home]]" in "ListUnencrypted" (90% of my tiddlers are tagged with "Home"), but my TW is frozen then!

For me, a "ListEncrypted" (the other way round) would ev. be more feasible...

Maybe, you can find some performance improvements. Actually I'd like to have everything encrypted...

Best regards,

Wolfgang

[Yakov]

Right, let's tinker this a bit. Do I get it right: you have autosave enabled, and saving causes TW to hang? Can you estimate for how long?

понедельник, 29 мая 2023 г. в 07:36:02 UTC+3, Wolfgang:

[Wolfgang]

No autosave.

Loading the locked TW is fast. Unlocking takes about 10s. Saving takes about 6s.

I use a relative old laptop with Windows 10 64bit on i5-4200U CPU @ 2.30 GHz and 8MB RAM.

Add [tag[Home]] to ListUnencrypted and press save: could not wait for finishing. After 5min I killed the process.

The TW is still usable, it did not save anything, I suppose...

If I provide ListUnencrypted with a tag with just 1 tiddler, it's the same problem...

Strange: I edited the TW in a text-editor and added [tag[Home]] to ListUnencrypted: unlock was successful, all tiddles with "Home" were there, saving was the same problem than before...

[Wolfgang]

8GB RAM, sorry...

[Yakov]

> Unlocking takes about 10s. Saving takes about 6s.

I guess, slow saving is more problematic than loading, isn't it?

> Add [tag[Home]] to ListUnencrypted and press save: could not wait for finishing. After 5min I killed the process.

> If I provide ListUnencrypted with a tag with just 1 tiddler, it's the same problem...

Oh wow, this definitely sounds like a bug. Could you tell me

• what saver do you use?
  
• what's the average size of a tiddler (in characters)? I need to reproduce the issue first, so I have to generate something "similar" to what you have.
  • in fact, it would be nice if you can create a copy of your TW, install the plugin below in it, reload, remove the plugin, save again, check whether the bug is reproducible, check if anything private is left, and send it to me either here or in a private message, it'll save me time reproducing the bug.
    

Here's the plugin (let me know if you have private stuff in fields or tags, and I'll update the plugin; note that the plugin run can take time for 18K tiddlers):

/***
|Description|Substitutes all tiddlers' content and titles (but not tags and fields) with random stuff – except those with {{{systemConfig}}} tag and those substituting shadows|
***/
//{{{
const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'
const charactersLength = characters.length
// switches content to random letters
const substitute = s => s.replace(/\S/g, () => characters.charAt(Math.floor(Math.random() * charactersLength)))

store.forEachTiddler((title, tiddler) => {
if(tiddler.tags.includes('systemConfig') || config.shadowTiddlers[title]) return

tiddler.text = substitute(tiddler.text)
store.saveTiddler(tiddler.title, substitute(tiddler.title))
})
//}}}

понедельник, 29 мая 2023 г. в 16:14:11 UTC+3, Wolfgang:

[Wolfgang]

Hi Yakov,

I once generated a TWc with 16K tiddler, that I can send you. You can reproduce the problem with that. Unlock with "test"...

https://eh.nurzurinfo.at/index.php/s/N55zx4sD9tBZAj7

Best regards,

Wolfgang

[Yakov]

Right, I've done some research and the news are not very good.

In my testing, I've found that allEncryptedTiddlersAsHtml takes ~4 seconds and encrypt takes 7-9 seconds. While it may be possible to optimize the first part, the second one is hardly optimizable (well, may be updating sjcl to 1.0.8 will improve this, but I doubt that) – it's kinda normal that encrypting that much characters requires that much time.

So the only real solution I can think of is to try to create a web worker which will do the heavy encryption work, and save once it reports back. The problem is, the current core updateOriginal isn't supposed to be async. I'll think about adding async support in the next core version, but that's not trivial in terms of backward compatibility, plus I'm not sure if moving all the necessary bits of code to a worker will create an overhead which affects performance as well.

So I can't really promise that I'll fix this soon (I'll see what I can do in the next core release, though). In fact, it seems that for such case it's more practical to do encryption on back-end, which requires another solution.

суббота, 3 июня 2023 г. в 09:11:08 UTC+3, Wolfgang:

[Wolfgang]

Hi Yakov,

if you find time to optimize encryption, it would be fine, but it is not urgent in any way.

Maybe, you could first find out, why filtering in ListUnencrypted (e.g. [tag[Home]] in my case) hangs. Since 80% of my tiddlers are tagged with "Home", encryption should be much faster using this filter...

Best regards,

Wolfgang

[Yakov]

Hi Wolfgang,

yeah, sounds like this should be debugged separately indeed. I'll let you know once I take a closer look. In the example TW that you've shared, is there a tag with similar "coverage" as Home in your case (80%), or should I try to add it myself?

Best regards,

Yakov.

воскресенье, 4 июня 2023 г. в 17:49:51 UTC+3, Wolfgang:

[Wolfgang]

Yes, TestTiddler1...

[Yakov]

Hi Wolfgang,

great news: there was indeed an algorithmic problem that caused quadratic execution time, I've fixed it by introducing some caching and now tag filter works ([tag[TestTiddler1]] in the case of the TW you've sent); but not only that, this also dramatically improved the overall saving time, even the time of encrypting (don't know how comes, presumably I misinterpret some profiling results). In my tests, saving that big TW takes no more than a second (profiling tells, that those operations which took 4 and 7-9 seconds now take 0.017 and 0.002 seconds, respectively). Saving with [tag[TestTiddler1]] in ListUnencrypted takes more encrypt is reported to take ~4.5 seconds (??).

So, the profiling results are not consistent, and I suggest you to test different approaches; keep in mind that there seemingly may be differences between the first saving (after changing ListUnencrypted) and the next ones. It looks like "encrypt everything" approach in fact works fine. Get the updated version (1.7.0) from http://yakovlitvin.pro/TW/pre-releases/NewEncryptedVaultPlugin.txt and let me know how it goes.

Best regards,

Yakov.

воскресенье, 4 июня 2023 г. в 20:15:31 UTC+3, Wolfgang:

[Wolfgang]

Great! It's much quicker now!!

But I'm a bit confused:

• when I put [tag[Home]] into ListUnencrypted, exactly those tiddler tagged with Home get encrypted?!
• when I put any filter into ListUnencrypted, the rest of the enumerated tiddler in ListUnencrypted also get encrypted?!

That's the opposite behavior I'd expect from a ListUnencrypted...

Or, am I wrong?

Many thanks,

Wolfgang

[Yakov]

Oops, sorry, that's a bug, removed the extra ! in 1.7.1

понедельник, 5 июня 2023 г. в 13:39:42 UTC+3, Wolfgang:

[Wolfgang]

Hi Yakov,

very good! Your adaption of EncryptedVaultPlugin is now working perfect!

One tiny issue: the messages for passwordSet and passwordUnset contain special characters ("ðŸ").

Maybe, you could explain the strength of security of EncryptedVaultPlugin a bit. As far as I understand from debugging:

• the pwd is hold in ceVault object during the session
• neither pwd nor hash are persistently stored somewhere
• when entering a pwd to unlock, callSjcl tries to decryt, getting a "crypto error", if pwd is wrong
• callSjcl stays a bit of magic, unfortunately...
  
• guessing tiddlers and adding them "offline" (editor) to ListUnencrypted does not decrypt the tiddlers (I've tried that)
• what else may be relevant?

Many thanks,

Wolfgang

[Yakov]

Hi Wolfgang,

a quick reply to your issue is – this is an encoding problem (presumably because the plugin is currently hosted as a txt and not on GitHub yet); you can temporarily fix it by copying the emoji that I've used originally:

• 🔒 Successfully set password
• 🔓 Successfully unset password  /to be honest, I'm not sure this is a good choice as it's barely different from the "locked" emoji (at least in Windows) while meaning "unlocked"; an ugly but recognizable alternative is ꗃ

As for explaining security aspects, I'll reply on this later as docs are, in fact, one of the two reasons of the plugin not being fully released yet.

Best regards,

Yakov.

вторник, 6 июня 2023 г. в 11:01:48 UTC+3, Wolfgang: