Like Mixing Evernote and Tiddliwiki: NoteSelf

[Danielo Rodríguez‏]

Hello tiddliwikiers,

Recently I have been thinking about tiddliwiki and mi personal projects. I have a couple of TW-plugins/projects almost ready, and I didn't touched them for a while (close to a year or two).

It's been a hard chapter on my life, but today I decided to take the dust from one of them, make a small extra effort and release it. And here it is: NoteSelf.

We all love TiddlyWiki. It is powerful, free, highly customizable and open source. But don't you miss something? Yes exactly! Sync capabilities like Evernote has. In the other hand, probably you love Evernote's sync ability, but don't you miss something? Yes exactly! Customization.

NoteStorm tries to fill that gap, bringing to TiddlyWiki a sync mechanism that allows you to work offline and sync wen you come back online.

I'm not saying that it is going to work automagically and you will have an Evernote-like experience out of the box, but it is quite close.

It uses Tiddlypouch, a plugin that brings all the power of PouchDB into tiddlywiki which will allow you to sync to any couchdb server.

PLEASE note that this software is still in beta, and has not been deeply tested. So please, don't just throw all your tiddlers into it and make it your main tiddlywiki!! Start downloading the empty edition, check that it saves, reload the page a couple of times, try to configure a remote sync server, check the remote database.... do the same stuff on your mobile phone...

I have been using it for a month or so, and I never experienced issues, but that does not guarantee that you will not!!!

Thanks to all of you taking the time to check it out. Please share your doubts and experiences. 

Check it out at:

http://NoteSelf.github.io

Regrads

[ParisWiki‏]

Hi Danielo,

I must say that I am totally lost !

First you speak about Notestorm. Is it the same as David Gifford's ? Apparently not, but what are the differences? Do you mean in fact NoteSelf ?

Second you say "NoteSelf does not require any installation". Fine, but what about Couch or Pouch DB? Downloading your file (TiddlyWiki), did not upload (P)(C)ouchDB on my PC.


Then you say "Everything is stored on the company's servers!". Which company? Any link to access those servers ?

Could you provide some working examples with such a working database ?

And maybe the steps needed for this mix to work "out of  the box " ?

Thank you

JCG

Le dimanche 31 juillet 2016 03:13:20 UTC+2, Danielo Rodríguez a écrit :

Hello tiddliwikiers,

Recently I have been thinking about tiddliwiki and mi personal projects. I have a couple of TW-plugins/projects almost ready, and I didn't touched them for a while (close to a year or two).

It's been a hard chapter on my life, but today I decided to take the dust from one of them, make a small extra effort and release it. And here it is: NoteStorm.

[RichardWilliamSmith‏]

Hi JCG,

It's not my place to reply on Danielo's behalf but when he says "everything is stored on the company's servers" he means Evernote, the problem is with imperfect (though very good!) English.

When he writes;

"You already know and love Evernote, we know it. It is comfortable, it syncs, has search capabilities... Wow, it's almost perfect, but... don't you miss?:
Privacy - Everything is stored on the company's servers!
Customization - If you don't like the interface, there's nothing you can do"

Instead of "but... don't you miss?" he should probably write something like "but... what about?"

He also mis-spelled "pouchDb" as "pouchDd", further down.

It won't install couchDB on your computer, but I don't think it actually needs the back-end to work (I only just got to playing with it) and it probably loads pouchDB from a CDN rather than bundling it (?), but it doesn't need to be 'installed', as such, for it to work (though obviously the functionality is limited if you don't have your own DB - it's actually very easy to install your own couch if you are using mac/win/linux)

I'm super-excited to play with this to see just how robust it is, especially to see if I can work on TWs with my phone/ipad offline and sync them back to my laptop when it's on, without worrying about losing data.

Danielo - congratulations! and thank you for this. I look forward to being one of your first testers over the next couple of weeks. Excellent work, this is very interesting and exciting stuff.

Regards,

Richard

[Danielo Rodríguez‏]

El domingo, 31 de julio de 2016, 10:53:09 (UTC+2), ParisWiki escribió:

Hi Danielo,

I must say that I am totally lost !

Hello ParisWiki, thank you very much for your feedback, I really need several points of view because when I write instructions I'm too excited, and sometimes I'm not clear about what I want to say.

 

First you speak about Notestorm. Is it the same as David Gifford's ? Apparently not, but what are the differences? Do you mean in fact NoteSelf ?

It's a misspeling. I was reading about Notestorm before writting about NoteSelf :P

 

Second you say "NoteSelf does not require any installation". Fine, but what about Couch or Pouch DB? Downloading your file (TiddlyWiki), did not upload (P)(C)ouchDB on my PC.

And it does not require any installation. Pouch DB library it's embedded on the wiki itself. That is it's magic.

 

Then you say "Everything is stored on the company's servers!". Which company? Any link to access those servers ?

Sorry, that is also a misunderstanding. I mean that problem is one of the problems of Evernote, not NoteSelf ( I was about to write NoteStorm again! I have to admit that the name is far better).

 

Could you provide some working examples with such a working database ?

Do you mean how to configure a couch db server? I have plans to record a video about that for one of the recommended couchdb providers, Cloudant.

 

And maybe the steps needed for this mix to work "out of  the box " ?

The steps are on the installation section: 

1. Write a name on the input field close to the Download button.
2.  Click the download button.
3. That downloaded file works out of the box.
   

Server side synchronisation requires some additional steps, but for basic functionality that is all you need. 

Thanks. Let me know if you are still unclear about it.

[Danielo Rodríguez‏]

El domingo, 31 de julio de 2016, 12:27:22 (UTC+2), RichardWilliamSmith escribió:

Hi JCG,

It's not my place to reply on Danielo's behalf but when he says "everything is stored on the company's servers" he means Evernote, the problem is with imperfect (though very good!) English.

When he writes;

[....]

Instead of "but... don't you miss?" he should probably write something like "but... what about?"

Thank you for your suggestion Richard. You are totally right, your sentence is much more clear and straightforward. I have corrected the welcome page to use it, thank you very much.

 

He also mis-spelled "pouchDb" as "pouchDd", further down.

Also corrected! Thanks again.

 

It won't install couchDB on your computer, but I don't think it actually needs the back-end to work

Exactly! Backend is not mandatory. Maybe I should be more clear about that.

 

(I only just got to playing with it) and it probably loads pouchDB from a CDN rather than bundling it (?), but it doesn't need to be 'installed', as such, for it to work

It is indeed bundled into the wiki. No CDN because that would totally kill the offline mode.

 

 (though obviously the functionality is limited if you don't have your own DB - it's actually very easy to install your own couch if you are using mac/win/linux)

Exactly, functionality is limited, but it is functional. But you don't need your own server to do that, there are several free providers of couchdb databases as services. My favourite one is cloudant.com

 

I'm super-excited to play with this to see just how robust it is, especially to see if I can work on TWs with my phone/ipad offline and sync them back to my laptop when it's on, without worrying about losing data.

I have been using that setup, and I never lost a tiddler, but as you say, it has to be tested. Please, keep me posted about your usage and the problems you could find.

 

Danielo - congratulations! and thank you for this. I look forward to being one of your first testers over the next couple of weeks. Excellent work, this is very interesting and exciting stuff.

Thank you very much Richard !!!  Your words are very encouraging, I'm very happy to see that you are going to use it.

[Danielo Rodríguez‏]

Did someone read the FAQ section ?

I'm very interested about how clear the information is, and about how much of it should be on the main page rather than the FAQ.

Also I'm very interested about any possible misunderstanding or dificult to understand term.

Regards

[RichardWilliamSmith‏]

Hi Danielo,

I'm currently trying to get syncing working with a local couchDB.

I can set the address http://127.0.0.1:5984 and the name of a db that doesn't exist, then save and reload, and the database gets created in couch - so it can 'see' my couch - but no records get created with new tiddlers and the sidebar says I'm "offline" still.

If I press the "save config" button I get another empty.html, that points to the same place, but now it has sync errors, saying

Sync error while processing '$:/StoryList': {"status":409,"name":"conflict","message":"Document update conflict"}

when the storylist changes.

Any idea what's going on?

Regards,

Richard

[Jed Carty‏]

I am going to have to take some very close looks at this. If I am right we may be able to make this an alternate way to use twederation. It wouldn't be a replacement to what we have now since it requires a server but I think that having something like this could work well with what we have by giving an alternative way to send and fetch new tiddlers.

[RichardWilliamSmith‏]

Hi Jed,

I agree entirely - this is convergent with my previous ramblings about ipfs. I think eventually you and I get equal access to a shared global namespace (of the whole internet) into which we publish everything. Private communication is all by encryption.

RR

[Danielo Rodríguez‏]

El domingo, 31 de julio de 2016, 13:33:57 (UTC+2), RichardWilliamSmith escribió:

Hi Danielo,

I'm currently trying to get syncing working with a local couchDB.

I can set the address http://127.0.0.1:5984 and the name of a db that doesn't exist, then save and reload, and the database gets created in couch - so it can 'see' my couch - but no records get created with new tiddlers and the sidebar says I'm "offline" still.

If I press the "save config" button I get another empty.html, that points to the same place, but now it has sync errors, saying

Sync error while processing '$:/StoryList': {"status":409,"name":"conflict","message":"Document update conflict"}

when the storylist changes.

Any idea what's going on?

Regards,

Richard

Sorry Richard,

For some reason (as I said, this project has been in hold for a long long time) the remote synchronisation was deactivated (code commented).

I already uploaded a new version, this time with the sync active. I have tested it and it works perfectly with cloudant. Hope it works for you too.

Sorry but you have to download again the wiki. I tried updating the plugin but TW complains that version 0.9 is older than 0.8  o_0

Because we are going to be betatesters, I recommend you to activate the debug mode on the control pannel, that way you will be able to check the synchronization logs even on mobile.

Make sure to respect the naming convention of your pouch server when naming the database.

Please let me know if it works for you now.

[Mark S.‏]

Very tiny correction

"which uses creates"

should be

"which creates"

[Mark S.‏]

Hello Danielo,

I have some questions.

Is there any advantage to this system if you don't sync it with a couchDB system?

How close did you come to the $50 mark during your test phase with Cloudant (how much data will be typical?).

If we're using Cloudant, don't we have the same privacy concerns as with Evernote?

Thanks! Your site and technology are really amazing!

-- Mark

On Sunday, July 31, 2016 at 4:13:45 AM UTC-7, Danielo Rodríguez wrote:

[Danielo Rodríguez‏]

El domingo, 31 de julio de 2016, 19:15:32 (UTC+2), Mark S. escribió:

Hello Danielo,

I have some questions.

Cool!

 

Is there any advantage to this system if you don't sync it with a couchDB system?

Not as much as using a remote coucDB but yes, there are some:

1. You don't need any plugin installation to have automatic saving, and you can also use the download method. 2x1!
2. You can have several databases, which means completely different collections of tiddlers with the same file. Probably not very useful, but there it is.
3. Not an advantage at this moment, but PouchDb actually stores revisions of the documents. It's my plan to provide a review and restore system (not diff comparison at first instance)

Not to mention that this can be used as a base to create a tiddlywiki mobile application. I don't have the time at this moment, but who knows. 

How close did you come to the $50 mark during your test phase with Cloudant (how much data will be typical?).

I never hit such limit, not even close. The heaviest month was less than $2

Let's review pricing...

Data volume 

in GBs / month $1.00 per GB / month

Well, my biggest TW is about 1MB, so you can have upt to 10 without even hitting $1 limit."Heavy" API requests * 

PUTs, POSTs, DELETEs $0.015 per 100

100k and you will be using 15 cents... quite cheap

My recomendation is to start using it and see how heavy is your usage after one mont. I'm 100% sure that you will not reach the $50 limit on the first month.

 

If we're using Cloudant, don't we have the same privacy concerns as with Evernote?

In some way yes, you do. But, you can choose to don't rely on any third party provider and host your own server. And if you want to switch to a different one all you have to do is sync to a different server and delete the old one.

Not to mention that at least you were using an open sourced client (tiddlywiki and pouchdb) so less obscurantism than Evernote.

 

Thanks! Your site and technology are really amazing!

Thank you very much Mark! Hope you enjoy using it.

[Mark S.‏]

Thought of another question. Of course.

If I want to transfer to another machine/browser, is there a mechanism for doing that?

Thanks!
Mark

[Danielo Rodríguez‏]

El domingo, 31 de julio de 2016, 20:48:00 (UTC+2), Mark S. escribió:

Thought of another question. Of course.

As many as you want.

 

If I want to transfer to another machine/browser, is there a mechanism for doing that?

I have plans to implement an easier mechanism, but you can of course use the ones that Tiddlywiki provides. For example, if all the tiddlers are already loaded, you can just download the file.

Another way would be going to advanced search, filter tab and use a filter like [!is[system]sort[title]]. You can export it as JSON, CSV, or even tid files.

 

Thanks!

Welcome! 

[Danielo Rodríguez‏]

I didn't mentioned it, and I'm not sure if someone has noticed it but the site it's actually just a tiddlywiky with a custom theme. That demonstrates how powerful tiddlywiki is!

[ParisWiki‏]

Thank you for your explanations.

It is certainly clearer, but I am still missing some points.

I understand that your modified tiddlywiki is embarking the "motor" for a data base. Fine.

For me,  any database contains records, each record with (pre?) defined fields.

I saw that the database name is defined when loading the wiki. But how and where do I declare the type of records and the various fields  for this database ?

What can constitute a record : A tiddler, a specific field of some identifed tiddlers (thru tags or listing) or a new "entity" that needs to be specified ?

And where is kept this database ? Within the TiddlyWiki as other data files already possible, or in a specific file on the computer ?

So I am really missing some exemples, since I think that an external database that could be acessed by one (or more?) tiddywiki would be a real plus for this already fantastic piece of software !

Sincerely,

JCG

[Danielo Rodríguez‏]

El lunes, 1 de agosto de 2016, 16:49:11 (UTC+2), ParisWiki escribió:

Thank you for your explanations.

You are welcome. 

 

For me,  any database contains records, each record with (pre?) defined fields.

I saw that the database name is defined when loading the wiki. But how and where do I declare the type of records and the various fields  for this database ?

What can constitute a record : A tiddler, a specific field of some identifed tiddlers (thru tags or listing) or a new "entity" that needs to be specified ?

You are thinking about a classical or "relational" database. But NoteSelf implements a non relational database. This has many implications, the main one is that no structure is required. Each "row" on the database is a document that can have any arbitrary number of fields and any structure. Tiddlywiki follows the same principle, each tiddler can have as many fields as you want, the only mandatory field is the title, the others are optional.

So each tiddler is saved as a single entry on the database

  

Tiddler => database record

No matter the structure of the tiddler or the size.

 

And where is kept this database ? Within the TiddlyWiki as other data files already possible, or in a specific file on the computer ?

The database is stored inside the browser. Modern browsers allow you to store data within it, like a virtual system running inside your computer. There is where the database lives. 

 

So I am really missing some exemples, since I think that an external database that could be acessed by one (or more?) tiddywiki would be a real plus for this already fantastic piece of software !

The external database is an additional option that you can use to synchronise your database across several browsers. Something like dropbox, you keep a local copy of your files, which replicates to dropbox servers which in turn synchronises to your other computers running your dropbox client. So you have a local copy of your tiddlers inside your browser, than can be synchronised to an external database which can be synchronized to any other client that has access to the same server.

 

If you have more doubts, please ask!

[Mark S.‏]

I notice that the "Pack Database" option doesn't have a button associated with it. At least not on mine. I can include a snapshot if that is helpful.

Also, if there was a "Pack Database" button, would there also be a "Unpack Data Tiddler" somewhere?

Thanks!
Mark

[Danielo Rodríguez‏]

Sorry Mark, I missed this message.

The Pack database button is missing. I have plans to implement it, but I have deferred it. The idea was to pack the database, save it as a system tiddler and download an empty wiki. Then pick it automatically on wiki startup if there is no database. To be honest, I forgot about this idea, and I think it is probably the better option for portability. 

Please feel free to comment on the newest thread