Viability of CBL "poor man's sync" using putExistingRevisionWithProperties?

[Brendan Duddridge]

Hi Jens et al,

I'd like to be able to provide my customers with another way of syncing their devices. Currently I'm providing Cloud sync via IBM Cloudant and Nearby Sync using peer-to-peer syncing in CBL 1.3.

But many of my customers would prefer to use something like iCloud or Dropbox for syncing. In fact, I just released the new version of my app after a year in development switching over to Couchbase Lite and I've been roasted in the app reviews because I dropped support for iCloud and Dropbox syncing. So I"m trying to find a way to get it back yet still be able to have all the advantages of Couchbase Lite, which is a fantastic framework of course.

I already have code in my app to generate an archive file using CBLDocument's putExistingRevisionWithProperties, so I know how to work with that.

But I was wondering what the viability of using that method would be to provide a file based sync solution? What I mean by that is I would take incremental snapshots of the changes made since the last time a sync operation occurred, then I would store the change file in iCloud Drive or Dropbox and then on the other device, read the changes file and process it using putExistingRevisionWithProperties.

It would have to be able to handle deletes, updates, and new documents. 

Does something like that sound viable? I'm thinking that it is as long as I can get a list of documents that have changed recently and if those documents contain the isDeleted flag. It seems like it's a possible solution for me.

Any comments, ideas, or concerns with implementing something like this?

Thanks!

Brendan

[Jens Alfke]

> On Aug 8, 2016, at 10:12 AM, Brendan Duddridge <bren...@gmail.com> wrote:
>
> But many of my customers would prefer to use something like iCloud or Dropbox for syncing. In fact, I just released the new version of my app after a year in development switching over to Couchbase Lite and I've been roasted in the app reviews because I dropped support for iCloud and Dropbox syncing.

You can never please all the users…

>
> But I was wondering what the viability of using that method would be to provide a file based sync solution? What I mean by that is I would take incremental snapshots of the changes made since the last time a sync operation occurred, then I would store the change file in iCloud Drive or Dropbox and then on the other device, read the changes file and process it using putExistingRevisionWithProperties.

Yeah, building some kind of “change archive” like this wouldn’t be hard. Run an all-docs query sorted by sequence starting just past the last sequence you synced, then for each document write its current revisions' JSON and all attachment bodies whose revpos is equal to the current revisions’ generation.

A Zip archive would be a convenient way to store this, where the internal filename is either the docID or “docID/attachmentname”.

But you’d have to write and upload a new snapshot every time you sync. A device coming online has to read each snapshot that’s newer than the latest one it’s read. So it’s not clear how you can ever delete old snapshots.

—Jens

[Brendan Duddridge]

Thanks for the feedback Jens. Yes, that was the problem I had with my older implementation. The change files would just grow and grow and eventually the user could run out of space. I did provide a Reset Sync function to let the user clear them all out.

Now back to my encryption problem for now. I'll deal with syncing later... Perhaps you've seen my panicking in the Github issues.

Thanks,

Brendan

[Brendan Duddridge]

Hi Jens,

Run an all-docs query sorted by sequence starting just past the last sequence you synced

I know how to create an all-docs query, but how do I sort it by sequence number starting just past the last sequence synced?

Is there a way I can query for the last sequence synced? And does the putExistingRevisionWithProperties function update that automatically? Or do I need to keep track of that separately in a local document in the database?

I know that I can query by startKey but can I query on the last synced sequence to get all the changes in the database since I last synced? And for sorting by sequence number, do I set the sortDescriptors on the query? Doesn't that do the sorting in memory after querying the database? And can I sort by sequence number in that case?

If you have some code example that shows me how to query all documents to get a list of changed documents since the last sync that would be super helpful.

Thanks,

Brendan

On Monday, August 8, 2016 at 12:11:52 PM UTC-6, Jens Alfke wrote:

[Brendan Duddridge]

Ok, I figured out how to sort by sequence number. I didn't see the kCBLBySequence option for that:

CBLQuery *docsQuery = [database createAllDocumentsQuery];

docsQuery.allDocsMode = kCBLIncludeDeleted | kCBLBySequence;

docsQuery.mapOnly = YES;

CBLQueryEnumerator *docsEnumerator = [docsQuery run: &error];

So the next challenge is how to query by sequence number. Can I specify the startKey and endKey and pass in the sequence number? I wouldn't think so because those are to query the view's key for a range of results. Not sure yet how to fetch all documents after a specific sequence number.

Thanks,

Brendan

[Jens Alfke]

On Aug 21, 2016, at 11:25 PM, Brendan Duddridge <bren...@gmail.com> wrote:

docsQuery.allDocsMode = kCBLIncludeDeleted | kCBLBySequence;

Careful: the mode constants are not masks, so you can’t ‘or’ them. (I’ve made this mistake myself before.) Just use kCBLBySequence; it implicitly includes deleted revs.

Can I specify the startKey and endKey and pass in the sequence number? 

Yes. Use query.startKey = @(firstSeq).

—Jens

[Brendan Duddridge]

Ah ok. Great. That's easy.

Thanks!

Brendan