[TW5] How does the revision field work?

[William Shallum]

Hi,

I am trying to write a sync adaptor for CouchDB. CouchDB requires
matching revision (_rev) values when updating / deleting entries for
conflict detection.

Here's how I think the sync adaptor's methods work with revisions, do
let me know if I'm wrong:

getSkinnyTiddlers: load title and revision (revision stored into
tiddler field and tiddlerInfo) from server
saveTiddler: server may change revision, updated revision NOT stored
into tiddler field, only tiddlerInfo
loadTiddler: revision stored in tiddler field and tiddlerInfo
deleteTiddler: tiddlerInfo is passed to the method, can use revision
in tiddlerInfo to perform DELETE

I am wondering why the revision in the tiddler field is not changed
after a successful saveTiddler. I tried to change it within the
saveTiddler method but the fields are read-only. Also, when editing an
existing tiddler, why is the draft's revision not set to null when it
is created? (It copies the original tiddler's revision instead, making
it hard to distinguish between a new draft tiddler and an updated
draft tiddler).

I've seen the previous discussion about the CouchDB adaptor and the
implementation here: https://gist.github.com/judell/9744381 does not
seem to have conflict detection for updates & deletes (it tries to get
the existing document first and immediately uses its revision to
overwrite).

Thanks,
William

[PMario]

Hi William,

The revision is _always_ generated and managed by the server.
It's main usecase is for conflict detection, if you replicate 2 dbs.
see: http://guide.couchdb.org/draft/conflicts.html

It is automatically generated and can just be used to save a tiddler back.
With the revision, you send back to the server, it can detect if there is a conflict.
see: http://guide.couchdb.org/editions/1/en/api.html#revisions

Here's a very important Note:

The terms version and revision might sound familiar (if you are programming without version control, drop this book right now and start learning one of the popular systems). Using new versions for document changes works a lot like version control, but there’s an important difference: CouchDB does not guarantee that older versions are kept around.

-mario

[Danielo Rodríguez]

Sorry for not helping you with any comment but... 

Please keep us updated about this. I'm very interested on this kind of setups.

Regards.

[William]

Hi Mario,

Thank you for the explanation. I am not asking about what the _rev is used for in CouchDB. I am asking about how the TW5 app (syncer and the app in general) handles the revision field, especially:

* why the updated revision returned from saveTiddler is not stored in the tiddler itself

* why a new draft tiddler's revision is set to the original tiddler's revision instead of null

Regards,

William

[PMario]

On Wednesday, September 24, 2014 12:51:07 AM UTC+2, William wrote:

Thank you for the explanation. I am not asking about what the _rev is used for in CouchDB. I am asking about how the TW5 app (syncer and the app in general) handles the revision field, especially:

Are you talking about the mechanisms used in syncer.js and may be tiddlywebadaptor.js?
The revision handling there is in line with the TiddlyWeb backend behaviour but TW5 doesn't know about revisions yet.
TiddlyWeb keeps permanent revisions of every tiddler.
 

* why the updated revision returned from saveTiddler is not stored in the tiddler itself

That's probably a bug. But at the moment, it doesn't matter. Since there are no conventions defined, how to deal with "backend fields" in a tiddler.
IMO what we have here is a _naming conflict_. So I think, what is needed, is a general discussion, how to handle fields, that are needed to deal with a specific backend.
 

* why a new draft tiddler's revision is set to the original tiddler's revision instead of null

 

That's probably a bug. But in the tiddlyweb context the revision sent from the client is always ignored by the server.
The server creates a new revision, when it saves something. Conflict detection is done with the tiddlers etag.

-----------------

The only field every tiddler must have is: title

Common fields are:
- modified, modifier, created, creator, tags, text

Application specific fields can be seen at: tiddlywiki.com:
It has a lot of them: http://tiddlywiki.com/#%24%3A%2Fcore%2Fui%2FControlPanel%2FTiddlerFields

Empty TW:
tiddlywiki.com/empty.html has: http://tiddlywiki.com/empty.html#%24%3A%2Fcore%2Fui%2FControlPanel%2FTiddlerFields

So what we need is some type of namespace, that separates backend fields from "common" and "application" fields.

With TiddlySpace a prefix "server." eg: server.bag, server.content-type, .etag, .permission, .recipe. ...... was introduced,
but these prefixed fields are explicitly removed by the TW5 import mechanism. see issue #238

There are some open issues eg:
 - https://github.com/Jermolene/TiddlyWiki5/issues/689
 - https://github.com/Jermolene/TiddlyWiki5/issues/238

So imo we need to deal with these issues and the namespace discussion first. Otherwise different backends will create a big field naming mess.

@Jeremy,
What do you think?

just my 2 cents
Mario

[William Shallum]

Hi Mario,

Thanks for the explanation. So it seems that the revision field is
treated as an opaque string by the TW5 app. The syncer just checks if
the revision returned from getSkinnyTiddlers is different between the
client and the server and refreshes the existing tiddler on the client
if it is different.

How does tiddlyweb detect conflicts using the tiddler's etag in the
current implementation? In tiddlywebadaptor.js the etag is not sent
with the request and you said that the revision sent from the client
is always ignored.

Regards,
William

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "TiddlyWikiDev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/tiddlywikidev/TSwYsd0ZxFw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> tiddlywikide...@googlegroups.com.
> To post to this group, send email to tiddly...@googlegroups.com.
> Visit this group at http://groups.google.com/group/tiddlywikidev.
> For more options, visit https://groups.google.com/d/optout.

[Jeremy Ruston]

Hi William

> I am trying to write a sync adaptor for CouchDB

Terrific, I'm very happy to hear this.

One CouchDB feature that I'm very interested in is the ability to use views written in JavaScript. I believe that it ought to be possible to package the TW core code as a view so that CouchDB can dynamically build TiddlyWiki's on the server.

In terms of revision handling, one point to note is that by design the core itself is not aware of the revision field; it is entirely implemented within the syncer module and any syncadaptor modules that need it. I wanted to do it this way so that we could develop new syncer modules that use a different basis for tracking revisions.

> * why the updated revision returned from saveTiddler is not stored in the tiddler itself

The problem here is that updating the tiddler with the new revision would constitute a change to the tiddler, triggering it to be saved once more to the server, which in turn would generate a new revision number, repeating the whole cycle.

> * why a new draft tiddler's revision is set to the original tiddler's revision instead of null

The "new tiddler" mechanism doesn't know anything about the revision field, and so just blindly copies all the fields of the original tiddler into the draft.

I'm very keen to support you on this work, please feel free to ask further questions!

Best wishes

Jeremy

You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

--
Jeremy Ruston
mailto:jeremy...@gmail.com

[William Shallum]

Hi Jeremy,

> The problem here is that updating the tiddler with the new revision would constitute a change to the tiddler, triggering it to be saved once more to the server, which in turn would generate a new revision number, repeating the whole cycle.

> The "new tiddler" mechanism doesn't know anything about the revision field, and so just blindly copies all the fields of the original tiddler into the draft.

OK, is there a way of getting the newest revision from the syncer's
tiddlerInfo inside the saveTiddler method? Maybe it could be passed to
saveTiddler like it is passed to deleteTiddler? This would probably
fix both issues since the sync adaptor can store the server revision
inside the tiddlerInfo and distinguish between a draft tiddler it has
not seen before (that has the original tiddler's revision) and a draft
tiddler that already exists on the server.

Regards,
William

[Jeremy Ruston]

Hi William

OK, is there a way of getting the newest revision from the syncer's
tiddlerInfo inside the saveTiddler method? Maybe it could be passed to
saveTiddler like it is passed to deleteTiddler?

Yes that makes sense, I've added it here:

https://github.com/Jermolene/TiddlyWiki5/commit/a159b5baf3ad91d8defc68cbf81c78d01b69c416

 

This would probably
fix both issues since the sync adaptor can store the server revision
inside the tiddlerInfo and distinguish between a draft tiddler it has
not seen before (that has the original tiddler's revision) and a draft
tiddler that already exists on the server.

Great, let me know how you get on,

Best wishes

Jeremy

[William Shallum]

Hi Jeremy,

Thanks for making the changes so quickly. I will let you know when it is done.

Regards,
William

[William Shallum]

Hi Jeremy,

I ran into another problem with the syncer: after a tiddler is
succesfully deleted on the server, the syncer's tiddlerInfo for that
tiddler is not cleared. This caused problems when editing a tiddler
twice. The draft tiddler is deleted on save but the tiddlerInfo for
the draft tiddler is still there, causing the code to send a revision
ID to the server that is not there when editing the tiddler again.

Also, is it possible to add HTTP 201 (Created) to the list of
allowable response codes in core/modules/utils/dom/http.js?

Regards,
William

[Jeremy Ruston]

Hi William

On Wed, Sep 24, 2014 at 2:40 PM, William Shallum <wil...@shallum.net> wrote:

I ran into another problem with the syncer: after a tiddler is
succesfully deleted on the server, the syncer's tiddlerInfo for that
tiddler is not cleared. This caused problems when editing a tiddler
twice. The draft tiddler is deleted on save but the tiddlerInfo for
the draft tiddler is still there, causing the code to send a revision
ID to the server that is not there when editing the tiddler again.

Aha, that makes sense. I've adjusted the code here:

https://github.com/Jermolene/TiddlyWiki5/commit/eacb9e53ebf2a814d61bf005d68f449f7b9e63b5

 

Also, is it possible to add HTTP 201 (Created) to the list of
allowable response codes in core/modules/utils/dom/http.js?

Sure:

https://github.com/Jermolene/TiddlyWiki5/commit/e2046ce4ffb6b8232a4ad5e7f51c431798917787

Best wishes

Jeremy

 

Regards,
William

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[William Shallum]

Hi Jeremy,

The initial version is up on GitHub: https://github.com/wshallum/couchadaptor

I uploaded it to Cloudant so others can replicate it to their own
instance: https://lmz.cloudant.com/tw/ and the TiddlyWiki URL is
https://lmz.cloudant.com/tw/_design/tw/index.html

Next steps will probably cover login, logout, and making the database
name configurable.

Regards,
William

> You received this message because you are subscribed to a topic in the
> Google Groups "TiddlyWikiDev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/tiddlywikidev/TSwYsd0ZxFw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Danielo Rodríguez]

Awesome! It even asks for username and password for creating tiddlers. Can be multiple users ?
Very good work. Thank you for sharing. This looks similar (but with less options) to tiddlyweb

[Jeremy Ruston]

Hi William

Congratulations, that's great. Good to see that the adaptor code itself is pretty short.

I think you may have unnecessarily inherited the special handling of the "fields" hashmap from the tiddlywebadaptor. It's needed because on TiddlyWeb all the "non-standard" fields are hoisted into the "fields" hashmap.

What are your plans for next steps? It would be pretty cool if you were able to draw up some simple instructions for TW users without Couch expertise to get things up and running on Cloudant (or elsewhere).

Best wishes

Jeremy.

[William Shallum]

Hi Jeremy,

> Congratulations, that's great. Good to see that the adaptor code itself is
> pretty short.
>
> I think you may have unnecessarily inherited the special handling of the
> "fields" hashmap from the tiddlywebadaptor. It's needed because on TiddlyWeb
> all the "non-standard" fields are hoisted into the "fields" hashmap.

Thanks for the comments. I am leaving it there just in case any users
want to use a field name starting with "_". This will cause errors in
CouchDB since names starting with an underscore in the root object are
reserved.

>
> What are your plans for next steps? It would be pretty cool if you were able
> to draw up some simple instructions for TW users without Couch expertise to
> get things up and running on Cloudant (or elsewhere).
>

Before I write the instructions, I have questions about where to put
the files for my custom edition. Should I just put it in the
repository?

Also there is a question of how the adaptor can best find the server
and the database name. I can probably edit the code to assume it is
always served from within couchdb and derive the database and the
design document name from the document's URL. However this does make
it harder to serve the app from another web server. What do you think
about forcing the user to edit a system tiddler and rebuild the
TiddlyWiki if they need to point it to another server?

The simplest way to get things running for people with no expertise is
probably to replicate the whole database from an empty instance.:

1. Create new database
2. Replicate from URL
3. Done

Of course, this means I have to maintain an empty instance and
regularly update it. Alternatively the push shell script in the github
repo can be used.

Regards,
William

[Jeremy Ruston]

Hi William

Thanks for the comments. I am leaving it there just in case any users
want to use a field name starting with "_". This will cause errors in
CouchDB since names starting with an underscore in the root object are
reserved.

Aha, that makes sense, I wasn't aware of that restriction in CouchDB. The core "_canonical_uri" field would get caught by it.

 

Before I write the instructions, I have questions about where to put
the files for my custom edition. Should I just put it in the
repository?

To begin with you should publish the edition independently, on your own repo (I'll add a link from tiddlywiki.com; is https://github.com/wshallum/couchadaptor the best URL to use?) . Assimilating the plugin and edition into the main tiddlywiki.com distribution will take a little longer; we'll need to get things to the point where I can easily test it against changes to the core.

 

Also there is a question of how the adaptor can best find the server
and the database name. I can probably edit the code to assume it is
always served from within couchdb and derive the database and the
design document name from the document's URL. However this does make
it harder to serve the app from another web server. What do you think
about forcing the user to edit a system tiddler and rebuild the
TiddlyWiki if they need to point it to another server?

A reasonable strategy might be to default to deriving the database and design document from the URL, but to enable them to be overridden with system tiddlers. The TiddlySpot saver does something similar composing the upload URL from the wikiname, but allowing it to be overridden with the $:/UploadURL system tiddler.

 

The simplest way to get things running for people with no expertise is
probably to replicate the whole database from an empty instance.:

1. Create new database
2. Replicate from URL
3. Done

Of course, this means I have to maintain an empty instance and
regularly update it. Alternatively the push shell script in the github
repo can be used.

Sounds good. For people without prior experience of CouchDB we'd need to explain steps 1 and 2 in a little more detail I guess.

I guess we'd also need to think about how an end user update their wiki to a new version of the core?

Best wishes

Jeremy.

 

Regards,
William

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[William Shallum]

Hi Jeremy,

Thanks for the advice. I am now trying to clean up the build process
and document it.

> A reasonable strategy might be to default to deriving the database and design document from the URL, but to enable them to be overridden with system tiddlers. The TiddlySpot saver does something similar composing the upload URL from the wikiname, but allowing it to be overridden with the $:/UploadURL system tiddler.

Where will this updated system tiddler be stored? Will the users be
building the edition themselves with an updated $:/UploadURL coded
inside the HTML?

Regards,
William

> You received this message because you are subscribed to a topic in the
> Google Groups "TiddlyWikiDev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/tiddlywikidev/TSwYsd0ZxFw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Jeremy Ruston]

Hi William

Thanks for the advice. I am now trying to clean up the build process
and document it.

Great. 

> A reasonable strategy might be to default to deriving the database and design document from the URL, but to enable them to be overridden with system tiddlers. The TiddlySpot saver does something similar composing the upload URL from the wikiname, but allowing it to be overridden with the $:/UploadURL system tiddler.

Where will this updated system tiddler be stored? Will the users be
building the edition themselves with an updated $:/UploadURL coded
inside the HTML?

Yes, I think that's the best we can do at the moment. I think we're catering to two audiences: those who want a turnkey click-and-go solution with no configuration, and more sophisticated users who are experimenting with non-standard configurations. It's only the latter group that would need to modify the configuration tiddlers.

Best wishes

Jeremy

[Danielo Rodríguez]

Don't you think is time to open a new thread to show the world your project? The title of this one is not very meaningful.

Additionally, would you consider to post instructions to get this up and running in android? There are some couchDB implementations.

[Danielo Rodríguez]

Hello,

Since I'm on windows, I tried your upload flow manually.

I'm using iriscouch.com as provider.

1. I build the index file (I don't know why this is needed) using: 
   tiddlywiki edition/ --verbose --output out --rendertiddler $:/core/save/all index.html text/plain
   
   
   
2. I used the node library that you provides, but without specifying the port. the command was something like: 
   node_modules/.bin/couchapp.cmd push couchapp.js myUrl
   
   
   
   
3. Everything worked and the file upload correctly,but when accesing the control panel I can see the couch-db adapter is not there.

What am I doing wrong?

[William Shallum]

Hi Danielo,

If you look at the shell script there is this part:

TIDDLYWIKI_PLUGIN_PATH=../

that tells tiddlywiki to look for plugins in the parent directory. The
tiddlywiki.info file specifies a "couchadaptor" plugin. Since I don't
see you setting this environment variable, it may be that tiddlywiki
fails to find the "couchadaptor" plugin. Try:

set TIDDLYWIKI_PLUGIN_PATH=../
tiddlywiki...

You can check that the adaptor is loaded in the index.html file
without needing to upload it. Just search for the contents of the
couchdbadaptor.js file inside the generated HTML.

Regards,
William

[Danielo Rodríguez]

Hello William,

I missed that part. I tried with your suggestion, and I tried just adding ../couchadaptor in tiddlywiki.info. None of thos things worked, so at the end I moved all the tiddlers to a subfolder of the edition folder plugins/couchadaptor. This worked for local server under node.js. I mean this 

tiddlywiki.cmd .\edition --server

Now the problem is that tirying to build the HTML gives me just a blank file. I don't know why, because this started to happen even before I moved all tiddlers to the mentioned subfolder.

Anyway, I can't see how building the HTML file can help to upload the correct template to couchdb since it just sents a JS file

[Danielo Rodríguez]

Anyway, I can't see how building the HTML file can help to upload the correct template to couchdb since it just sents a JS file

Ok, maybe this makes the magic:

couchapp.loadAttachments(ddoc, "out"); 

[Danielo Rodríguez]

OK,

Now I have it up and running.

The problem of the 0 bytes output was because I was launching the build command from powershell. I don't know why this happens but... building index.html from the normal windows console did the trick.

So the final configuration is:

• Your plugin inside the plugins folder of the edition folder
• manually building the index.html file
• uploading the file using your couchapp.js script

Now everything seems to work except for the recent tab. There are no tiddlers there. Why?

Can I now sync this database to other databases? And... what about an android app with server sync? :P It would be a killing feature, for sure.

[Danielo Rodríguez]

The plugin mechanism seems to not work properly due to lazy loading.

The plugin tiddlers are saved, but they are not loaded unless I open the plugin tiddler, but they does not work.

[William Shallum]

Hi Danielo,

I tested on my Windows install and it seems to work:

C:\Users\me\dev\couchadaptor>set TIDDLYWIKI_PLUGIN_PATH=..

C:\Users\me\dev\couchadaptor>tiddlywiki edition --verbose --output out

--rendertiddler "$:/core/save/all" index.html text/plain

Boot log:
Startup task: load-modules
Startup task: info after: load-modules before: startup
Startup task: startup after: load-modules
Startup task: story after: startup
Startup task: commands platforms: node after: story
Executing command: output out
Executing command: rendertiddler $:/core/save/all index.html text/plain

I really don't have any info on how the plugin mechanism works when
the tiddlers are synced from a remote source as I am new to TiddlyWiki
myself. Maybe they should be baked into the .html file itself.

Regards,
William

[Danielo Rodríguez]

Well, currently no matters how should I have set it up since it is now working. I though the same as you suggested, including the plugin in the HTML file before uploading. I will try it tomorrow. Thanks