NoteSelf uses local browser storage seamlessly with a remote synching server

[@TiddlyTweeter]

Danielo's NoteSelf is a pretty amazing innovation that combines in one TiddlyWiki saving to a local in-browser database (i.e. no old style saving problems, it works out of the box via PouchDB) WITH ability to save your TiddlyWiki to a remote CouchDB based server that will auto-sync your work.

Here is an informative, independent, blog post about it ...

https://thejeshgn.com/2018/02/08/noteself-tiddywiki-couchdb-backend/

@TiddlyTweeter

[Mark S.]

Just received a notice today that Cloudant shared service is being retired. I haven't seen anyone explain how to use the new Blue Cloud (or whatever they're calling it) service.

A killer feature would be if tiddlers would be encrypted transparently before being sent to the server. This would give it an edge over SimpleNote and Evernote, neither of which promise to encrypt your data.

-- Mark

[Ste Wilson]

Yeah..
I tried to contact ibm help on how to transfer over easily and they were leash than help full.

[Mark S.]

I don't think IBM is targeting individual users. They don't really like things "personal". They prefer "Corporate", "Colossal", "Immense". Helping small time users with their free accounts isn't really their thing.

It also appears that all the free couchdb services dried up. Could find one service claiming $3/month, at which point why not just get Evernote?

-- Mark 

[TonyM]

Mark,

Depending on the limits of a $3 a month couchdb it may allow a virtualy unlimited number of tiddlywikis and users. Usage dependant of course.

I do not yet know how but we can also host our own couchdb lan or internet.

Regards
Tony

[Danielo Rodríguez]

Hello everyone,

Sorry for all the inconveniences. Yes, it is true, cloudant as an independent platform is being closed. However, you can sign-up for a bluemix account and continue to use the free tier of cloudant inside the bluemix platform. The problem is that sign up is not as easy as before, the username is not as easy to remember as before and the password is automatically generated, it can be changed but it is very inconvenient. 

I don't have much time to spend on NoteSelf lately, but my idea is to build a service on top of bluemix dedicated to NoteSelf, so you can just sign up and use the service normally, much more like evernote, and we take care of the rest under the hood.

This kind of functionality will require a server, something I have to find where to host, and an update to the online edition to offer better support for this.

In any case, someone has added some written instructions about how to create a bluemix account and how to use it with NoteSelf on the tutorial video about how to create a cloudant account.

A killer feature would be if tiddlers would be encrypted transparently before being sent to the server. This would give it an edge over SimpleNote and Evernote, neither of which promise to encrypt your data. 

That is an Idea I have been wanting to implement for years, but I just delayed it. 

Regards

[Arlen Beiler]

Considering that this is tied to the browser, which is tied to the computer, I have now switched to using Dropbox synced data folders, which work about the same. There are three or four different ways to serve them: NodeJS TiddlyWiki, TiddlyServer, an electron-based shim which loads the filesystem directly into the browser, and maybe TiddlyDesktop (not sure about this one). I also have a copy of electron stored in my Dropbox, but that is optional.

The only downside is that there is not yet a way to access a data folder from the web, except in raw text, but I am hoping my exploration in TiddlyWiki will one day lead to that. 

Just my two cents.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/d412ed1f-3e74-4697-a82a-d8e4792f2c98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[@TiddlyTweeter]

Ciao Arlen

I think the kind of broad understanding you have is very good.

IMO, lack of near any resources discussing and showing, comparatively,  the various routes and their outcomes in different cases is negatively impacting TiddlyWiki uptake. How can you adopt something you don't understand?

I sometimes wonder if we (not you personally) could get better at addressing this in a way that will benefit end-users to get a better OVERVIEW of options?

Josiah

[ljay...@gmail.com]

Danielo's solution made Tiddlywiki "complete" for me personally. I've been running it since December and it syncs way better than Evernote or Onenote. Which isn't saying much, it's being more than gracious to call those two pieces of software malicious garbage. Noteself syncs REALLY well. The closest thing I've come to this is using syncthing to sync a zim notebook. Which works pretty well, except you can't edit it on a phone. And it's a complicated mess.

I learned about TW years ago, but I require syncing so it wasn't useful for my use-case until Danielo added noteself. I was really impressed with TW back then actually, it's just that syncing is a critical part of the system for me, so I ended up trying every other wiki-like system I could get my hands on. There are some really cool systems, but nothing compares to TW + noteself. Thank you Jeremy and Danielo and everyone who's efforts have helped make my dreams possible! (do you have wallets I might be able to contribute to or some way to show appreciation?)

My own setup is using couchdb instances on a couple debian devices and using one to sync and one as a manual periodic backup. It's public facing with https/SSL and I'm syncing great with Android (Have not been able to get IOS to work) and also current Firefox on several debian workstations. I load the files locally since I wasn't able to get Noteself working via DAV over https with simple authentication.

Hope this is helpful to anyone looking at Noteself and Tiddlywiki, I personally feel like this is some of the most important and helpful software development going on currently and would like to encourage everyone to check it out!

[Mat]

Something always glitches when attempt at getting NoteSelf to work properly. Maybe with Bluemix it will work so could someone please give some guidance? 

I successfully set up a new Bluemix account (via here) and created a new db named foobar. This can be seen on my db page, i.e:

https://<range-of-characters>-bluemix.cloudant.com/dashboard.html#/_all_dbs

There are no other db's on that page.

On https://noteself.github.io/online/ (i.e the online noteself version) I have some earlier "leftover attempts" listed in the Notebook name dropdown, so I type in foobar.

...and the CouchDB URL, as you see, ends in "...bluemix.cloudant.com" (no subfolder or whatever)

...I click "Save Config" and reload.

Now for some reason, in Danielos instruction video, the "online cloud symbol" is seen in the TW page toolbar. (How could it be "connected" to the account without any password given?)

For me there is instead an [Offline] button. Clicking this gives a mini-popup with:

[Username]

[Password]

...I naively attempt with my Bluemix username (i.e my email address) and then the db name, i.e "foobar", and the password to my Bluemix account... and get this popup as reply:

In earlier attempts, I once got things to work only after having created a "_replicator" db or a "_user" db (as seen in he video here) but, if I recall, Danielo said this should not be necessary. Regardless, that doesn't solve it this time around. I only get that yellow "Login failed" popup.

What to do? 

Thank you in advance!

<:-)

[Mat]

BTW, here's a positive review from someone.

<:-)

[willw]

Mat

I had similar problems with failed logins on moving from cloudant to IBM bluemix. It's working now, with Danielo's help.

The cause? It was CORS. Of course!

My instructions for configuring CORS are posted March 18th on this github issue (with links to IBM documentation):

https://github.com/NoteSelf/NoteSelf.github.io/issues/52

I couldn't find a way to configure CORS through the dashboard GUI menus so ended up using a few lines of Python code to do the set up via their API.

The forced migration from Cloudant's friendly service was a shame, especially to lose the simple named account. IBM's bluemix with it's cryptic username and confusing setup is off-putting, but I'd encourage you to persist. 

I find Noteself to be a fantastic way to sync up tiddly notes and have everything backed up, including past revisions, on a cloud db.

Will

[Mat]

Thanks willw

The cause? It was CORS. Of course!

OK, I find your instrux on github... but I believe I already had CORS set up. In the bluemix-cloudant dashboard:

...or would you say this is not correct? Are there more CORS settings?

Thank you!

<:-)

[Ste Wilson]

I'm still using cloudant.., its not stopping I don't think but the shared plan is no longer available.

On a tangential note if Danielo is around.. NoteSelf doesn't seem to be displaying my templates.

Ste

[Diego Mesa]

Hey all,

I have some questions about noteself maybe someone can help me answer! I understand it uses your browsers local storage. I am used to using nodejs, and keep my tiddlywiki folder in my dropbox. That means I have a folder with many .tid files I can see and touch. I also frequently "clear all" on my browsers, erasing everything. 

1. When using noteself, how can I "export" the local database noteself is using so that I can keep my own backups?
2. I understand noteself can by synced with an online database provider. I understand many people used cloudant, which closed, and are now using IBM Cloud (used to be called bluemix). When I signed up yesterday for the lite plan I noticed it said this: "Lite plan services are deleted after 30 days of inactivity.". Is everyone aware that if you dont sync for 30 days your db is gone?! Is there another free service that is trustworthy and does not have this?
• If I set up a good answer to 1., this one wont really bother me.
  

Thank you!

Diego

[TonyM]

Diego,

I belive if you tap save icon twice or more rapidly you will be prompted to download the whole wiki with you content included.

You could also go to advanced search > filter > select all tiddlers except system tiddlers and down load only regular tiddlers to a json file and drop import this back on your noteself to restore them.

Regards
Tony

[TonyM]

If I could build a cloudant server on a cpanel host php/apatche I would offer some free accounts

But I do not know if or how.

Perhaps someone could build a mysql or maria db version. I think that would realy take off because there are many ways to host those dbs.

Regards
Tony

[Diego Mesa]

Hey Tony,

Thanks for your reply. As this storage model is based on a database that stores revisions inside it, my backup strategy would be to keep one or two database copies around at a time, as opposed to an .html file. Is this possible?

[Corey Woodworth]

Would it be possible to provide something like a docker image for a backend so people could host their own noteself db on their own server or a raspberry pi?

[Mark S.]

Good point Tony!

I suppose couch/pouch is a better match for the way TW stores data, where any tiddler could have any number of fields.

But, what if you limited the number of fields exposed? Just title, caption, text, created, modified, tags and a couple extra (field1, field2). Then maybe it could be a match for a SQL database.

As you say, SQL databases are way more common, thanks to Cpanel.

-- Mark

[Diego Mesa]

As I am terrified of losing data, I would just like to emphasize this (this isn't a criticism of noteself, just an FYI for those using IBM Cloud):

IBM Cloud Lite plans are deleted after 30 days of inactivity.

[@TiddlyTweeter]

Diego Mesa wrote:

As I am terrified of losing data ...

A few notes on this in relation to NoteSelf. Understandable IF the server clocks-out AND the browser clocks-out and you didn't do any additional backups.

I do think (manual, redundant) backup-methodology for NoteSelf could be made a bit clearer.

But if you think about it its a pretty robust system already. Actually really robust. Especially if you connect to it through more than one browser.

To reduce paranoia :-) maybe additional backup could be good. FWIW I just backup the "local storage" of the browser via a prog that runs every 30 minutes or so.

Best wishes
Josiah

[Diego Mesa]

Josiah, 

For me, I frequently erase everything in my browser, and since IBM Cloud resets after 30 days of inactivity, that also isnt a fair backup option for me. 

I guess to use noteself Ill have to stop erasing my browser? 

[@TiddlyTweeter]

Ciao Diego

Look at your browser settings. What you want is to NOT delete "Local Storage" / "Offline ... Data". Not sure if all browsers allow you to pick and choose. My FF does ... 

Josiah

[TonyM]

Mark et al..

I would of thought the tiddler could be saved as a "blob", that is the fields are just more details in the tiddler, all just one big text file. They only become meaningful when they are loaded into the Tiddly-wiki which knows about fields. The question is can you have variable length fields in MySQL? I think you must because WordPress stores posts this way.

With any database including Cloudant there would be methods to extract all content and save it in a file, and the schema. That is you back up at the database administrator level without any knowledge of tiddlywiki.

Regards

Tony

[TonyM]

Diego,

That is a fairly brutal practice, why do you do it? If you can do that regularly enough, why not also backup your DB at the same time (or simple export tiddlers).

If you do not do it surely the reliability for your NoteSelf wiki goes up for you/

Regards

Tony

[Jed Carty]

I don't have any real experience using NoteSelf but I do know a bit about databases. You can have variable length columns in MySQL, each column has a type which determines the allowed content and some types have limits on the size but text columns can have variable lengths.

SQL is optimised around having a known number of columns with a known type in each column which isn't a good match for tiddlywiki because tiddlers can have any number of custom fields they would all be stored as a single text blob anyway.

But it should be possible to host your own CouchDB server on a raspberry pi or whatever hardware you have available so I don't think that there is a reason to use MySQL, if I understand correctly you could set up a CouchDB server on your local network and it would sync with remote changes or you could just use it on your local network.

I know very little about docker and the few times I tried to use it I gave up and just installed everything manually. I am often wrong about the relative levels of expertise needed for things like this but I think that if you are comfortable setting up a raspberry pi setting up the CouchDB server shouldn't be much of a stretch, there are instructions here https://andyfelong.com/2017/09/couchdb-2-1-on-raspberry-pi-raspbian-stretch/

[Mark S.]

On Tuesday, April 24, 2018 at 1:07:39 AM UTC-7, Jed Carty wrote:

SQL is optimised around having a known number of columns with a known type in each column which isn't a good match for tiddlywiki because tiddlers can have any number of custom fields they would all be stored as a single text blob anyway.

You would only need to put a handful of fields into actual SQL fields and roll the rest into some sort of blob. The advantage is that there are a large number of hosts that offer MySQL thanks to cpanel. But couch is not offered at all. With skinny loading, you could have a working TW with as many tiddlers as the SQL system could handle.

The last time I tried Noteself, I found that I could not cleanly delete and restore an instance. This suggests that there is data being left behind. This would worry me in any situation where security is a concern.

-- Mark

[TonyM]

Mark et all

Yesterday I installed CouchDB on My Windows 10 64bit, I have not got a noteself to connect to it yet. There seems to be a gap in the instructions on how to configure a tiddlywiki DB. This would allow me to look at the database structure that NoteSelf is using and thus determine if MySQL would be a practical re-invention.

As You (mark says) One way to look at this tiddler in DB is that the title is the key and rest is a blob this is in keeping with the concept of a skinny tiddler. Loads as needed.

On Couch DB there is a docker image which I intend to try on my Synology NAS but if you have a host that supports Docker you could build an internet facing CouchDB.

Unfortunately I only have a highly asymetric Internet connection with a small speed up to the internet so I am not in a position to host much from inside my LAN out to the internet, thus a desire to get a DB back end for TiddlyWiki hosted on the internet. I have a wholesaler/re-seller hosting service on cpanel

Regards

Tony

[Danielo Rodríguez]

Hello all.

Glad to see that more people is getting interested by NoteSelf.

I'm sorry to read people is having some trouble. To be honest, the new Cloudant policies have hit the NS community hard. However, I have plans to mitigate this problem (see below).

I want to make some clarifications about NoteSelf. It is not just a plugin, neither just a database for your tiddlers, it is an entire product and it is not fair to call it "just a thing to use a remote database". It has many features that are not only saving tiddlers automatically, and if you try to port all of those to "just a database" you will lost many of them. For example:

- Revision history. You can see the different old revisions of your tiddlers, and restore them if you want on a more or less nice interface.

- Offline support. You can use NS while you are offline, no network connection required, and changes will be synchronized automatically when you come back online.

- No need of backend. Yes, syncing requires a backend, but you don't need any remote database if you just want to use NS. You can even use the normal TW workflow (downloading files) and use NS as an intermediary cache-step for automatic saving.

- An hybrid android app. It may not be spectacular, but works fine and it is better than using any browser.

- A system that allows you to install plugins without saving a new HTML file

I am happy to see that other people is taking inspiration from NS to build their own solutions, that's fantastic, but have in mind that NS is more than just a database.

Also I want to remind you that what makes an open source project strong is it's community. If each user tries to create their own solution, then we will end with dozen half-baked solutions. If you think that NS is lacking some small detail, don't hesitate to file a pull request to add that feature you are missing.

For all those users using or interested on NoteSelf:

I am working on a project to bring easy login flow to NoteSelf. It will use a passwordless approach, so you will not need to remember a password neither registration.

You will input your email and hit login, then we will generate a random pin with a small lifetime (say 30 minutes), then you input that pin on NoteSelf and we will provide you a long-term authentication token that will be automatically managed by NoteSelf.

At first the service should be considered beta, will be free and only one database per user will be provided.

I would like to know your opinions on this approach, if you like it or not and if you will use it.

There will be a section for novice users, for being able to use this easy-automatic workflow, and I'll keep a section for advanced users that want their own couch servers.

Regards

[Lost Admin]

I agree, it is pretty amazing. I had some initial issues getting it to work but I think that was because I tried right around the time IBM bought out Cloudant and the "set-up" process changed. Instead I started messing around with other things. I always meant to get back to it. Yesterday, I did just that...

I've set-up my own Apache CouchDB back-end and Apache Web server on a US$2.50 per month VM with Vultr (not to say other cloud hosting wouldn't work just as well). It works very nicely with a downloaded copy of Note Self served from a simple static web server. I set-up SSL with Let's Encrypt, so it is even somewhat secure.

In my case, I use TiddlyWiki as a sort of simple content creation tool. Note Self is more of an online notebook for my own use.

On Wednesday, April 4, 2018 at 12:17:04 PM UTC-4, @TiddlyTweeter wrote:

Danielo's NoteSelf is a pretty amazing innovation that combines in one TiddlyWiki saving to a local in-browser database (i.e. no old style saving problems, it works out of the box via PouchDB) WITH ability to save your TiddlyWiki to a remote CouchDB based server that will auto-sync your work.

Here is an informative, independent, blog post about it ...

https://thejeshgn.com/2018/02/08/noteself-tiddywiki-couchdb-backend/

@TiddlyTweeter