null channels when sync gateway is down

[Rajagopal V]

Hi

Im creating a simple document and also updating it via iOS when the Sync Gateway is down. Upon restarting SG, the document (original and update revision) are both pushed to Couchbase server through Sync Gateway. But if I view the document properties, the _sync hash has "null" channels for both these revisions.

But if the SG is running and I make any updates to the same document, the channel is properly populated. Shouldnt the channel be updated even in the case the SG is brought up again and the documents are synchronized to the server.

This is causing me to skip revisions since Im doing a continuous feed on the _changes UI. If I just do a normal feed, I can see "holes" (corresponding to null channels) like

curl -N -X GET 'http://localhost:4984/<DB>/_changes?style=all_docs&feed=normal'

returns

....

,{"seq":"public:21","id":"00E91C69-5296-4A69-A1AC-91FBF4BD3587","changes":[{"rev":"3-73a4481278d6441f1de493b22318e8cf"}]}

,{"seq":"public:24","id":"AA781585-FD0D-47BA-A70D-57114678A72B","changes":[{"rev":"2-c998bc58f846d7ebcb21df839df8fca3"}]}

,{"seq":"public:27","id":"54A6A32D-699E-4833-9DA0-8D9A46945E97","changes":[{"rev":"3-79cb0b2be14b28bca658defca9349513"}]}

,{"seq":"public:28","id":"9E18F7A0-CFC4-4ABD-A3F8-126433A90149","changes":[{"rev":"1-cc70d0d05ca64e8f3a1f08cd046a67cb"}]}

Sequences 22,23,25,26 are creates/updates I did when the Sync Gateway was down.

Im using the simple "public" channel as mentioned in the documentation for the GUEST user. Also, Im building sync gateway from source (Last week's changes) since I wanted to try out "bucket shadowing" (Im not using that feature but wanted to try out , which is why Im not using Beta2 build).

Thanks

Raja

[J. Chris Anderson]

It's hard to tell without more context, but maybe this will help:

There is no guarantee that all updates will show on the changes feed. If you update a document in quick succession (or multiple times will offline), only the last version will be sent to Sync Gateway.

Similarly, if you modify a document multiple times on Sync Gateway, while you are connected to a continuous changes feed, you MIGHT see all the intermediate versions (or you might only see the last one), but if you make a new changes connection, you'll definitely only see the last version of each document.

Chris

[Jens Alfke]

On Feb 19, 2014, at 12:43 PM, Rajagopal V <raja...@gmail.com> wrote:

Im creating a simple document and also updating it via iOS when the Sync Gateway is down. Upon restarting SG, the document (original and update revision) are both pushed to Couchbase server through Sync Gateway. But if I view the document properties, the _sync hash has "null" channels for both these revisions.

But if the SG is running and I make any updates to the same document, the channel is properly populated. Shouldnt the channel be updated even in the case the SG is brought up again and the documents are synchronized to the server.

There shouldn't be any difference between those cases. In both of them, the client sends the document to the gateway and the gateway assigns channels. It doesn't matter when the doc was created in the app.

Can you give a more detailed description of the steps that occur in the first case?

(Also: The fact that you're talking about "the _sync hash" worries me. You shouldn't be looking at the internal form of the documents in the bucket; that's private to the gateway. I mean, you can look if you want, but don't draw conclusions about what's actually going on :) The REST API is the source of truth. If you want to find out what channels docs are in, you can do a request like

curl :4985/db/_all_docs?channels=true

.)

—Jens

[Rajagopal V]

On Thursday, February 20, 2014 2:55:38 AM UTC+5:30, J. Chris Anderson wrote:

It's hard to tell without more context, but maybe this will help:

There is no guarantee that all updates will show on the changes feed. If you update a document in quick succession (or multiple times will offline), only the last version will be sent to Sync Gateway.

Similarly, if you modify a document multiple times on Sync Gateway, while you are connected to a continuous changes feed, you MIGHT see all the intermediate versions (or you might only see the last one), but if you make a new changes connection, you'll definitely only see the last version of each document.

This is exactly what is happening. Im trying to see what will happen if the sync gateway is down while the mobile is making changes. I have a python listener running _changes using continuous feed. When I bring down the Sync Gateway, the listener also shuts down as the connection is terminated. Now I make some changes on the mobile (create and multiple changes to the same document) and then restart the Sync Gateway. When I restart the python listener to listen again for continuous feed changes, I only get the last version of the document rather than the complete audit trail.

The Sync Gateway however pushes all the changes (all revisions) to the couchbase bucket, so I was thinking that the same will happen to my python listener as well. 

Now, I brought down the sync gateway only to simulate the loss of network from a mobile and checking how its coping with the updates, but its probably not the best thing to do. I should figure out a way to shut down network on the iPhone Simulator rather than shutting down Sync Gateway.

Thanks for your response.

Raja

[Rajagopal V]

On Thursday, February 20, 2014 4:03:03 AM UTC+5:30, Jens Alfke wrote:

On Feb 19, 2014, at 12:43 PM, Rajagopal V <raja...@gmail.com> wrote:

Im creating a simple document and also updating it via iOS when the Sync Gateway is down. Upon restarting SG, the document (original and update revision) are both pushed to Couchbase server through Sync Gateway. But if I view the document properties, the _sync hash has "null" channels for both these revisions.

But if the SG is running and I make any updates to the same document, the channel is properly populated. Shouldnt the channel be updated even in the case the SG is brought up again and the documents are synchronized to the server.

There shouldn't be any difference between those cases. In both of them, the client sends the document to the gateway and the gateway assigns channels. It doesn't matter when the doc was created in the app.

Can you give a more detailed description of the steps that occur in the first case?

I am running a Continuous feed listener that waits for changes made to the Sync Gateway and then pushes those changes to the server using our own REST API. Im trying this to make sure that the server application and mobile devices are always in sync. The server is a PHP Mysql based application which exposes a set of REST APIs. Everytime there is a change in the sync gateway from the mobile devices, I connect to the Server's REST API and POST/PUT/DELETE calls into it. Same happens the other way too. If a new object is created/updated/deleted on the server, I have a listener that will push those back to the Sync Gateway, which will then propagate the changes to the mobile devices.

All of this work well if my sync gateway is running but since I wanted to test out how this works in the case network isn't available, I turned off the sync gateway (which was probably a wrong thing to do  -- I should have shut down the network connection on the iPhone simulator rather than shutting down Sync Gateway) and made create/updates to a document on the mobile. When I restarted the Sync Gateway and my python listener, it only brought me back the last change to that document. (From Chris' response above, it seems that only the last change will be sent if a new _changes request is made). 

I looked into the _sync hash and correlated the channels being null with the revisions not being sent. Since I made 3 revisions and 2 of them were in offline mode (which resulted in null channels) and the third one was in online mode (with a proper channel), I assumed that the revisions with "null" channels were not sent. Probably a misunderstanding of how things work, but Im very new to this and still learning :)

Thanks for your response

Raja

[Jens Alfke]

On Feb 19, 2014, at 7:36 PM, Rajagopal V <raja...@gmail.com> wrote:

I only get the last version of the document rather than the complete audit trail.

That is the correct behavior.

The Sync Gateway however pushes all the changes (all revisions) to the couchbase bucket

If revisions are added one at a time to the gateway, it will keep the full info of each one (until you compact the database, which you should do or it will grow monotonically.)

But the replication protocol only transfers the full data of the current revision(s) of a document. Intermediate revisions are only known by a revision ID and parent revision: that's sufficient to construct a revision tree for conflict resolution. The data of obsolete revisions is not transferred by the replicator.

(We didn't make this up; it's part of the CouchDB replication protocol that Couchbase Lite and the Sync Gateway use.)

—Jens