CloudKit <--> CouchbaseLite

[Brendan Duddridge]

Hi,

I'd really like to add CloudKit syncing to my app and I was wondering what the best approach would be for integrating it with CouchbaseLite (iOS).

In a previous post I suggested generating files that contain all the changes and then processing them on each device, keeping track of the last file that was synced.

But now I'm thinking a better solution would be to use CloudKit and store CBLDocument properties into CKRecord objects. 

What I'm wondering is what would be the best approach for this?

Should I simply store a one-to-one mapping of CBLDocument objects (representing the most current revision) to CKRecord objects?

What if a CBLDocument is updated or deleted?

Instead of storing CBLDocument objects into CKRecord objects, should I store CBLRevisions into CKRecords basically keeping a running log of all the changes over time?

If a CBLDocument is updated or deleted locally, maybe I should just replace the associated CKRecord in CloudKit with the updated most recent local revision?

What about document attachments?

There's the CKAsset class for that, but I'm wondering if maybe it's best to have a separate CKRecord for each attachment and relate that to the main CKRecord that represents the CBLDocument? Then if an attachment is updated, I just just find the corresponding CKRecord and update its attachment.

Processing the changes from CloudKit would just be a matter of converting each CKRecord into a CBLDocument and calling putExistingRevisionWithProperties on it.

I know these are a lot of unanswered questions. I'm just trying to find out what the best way to support CloudKit would be.

Official support for this from Couchbase would be an amazing addition for all iOS and Mac CBL developers.

If anyone wants to collaborate with me further on this that would be amazing. Multiple minds working on the problem would be much better than just me alone.

Thanks,

Brendan

[Brendan Duddridge]

This project on GitHub looks interesting that may be beneficial as a way of getting CloudKit syncing with Couchbase Lite:

https://github.com/joulupukki/BTCloudKitSync

Or there's this one, but it's all written in Swift:

https://github.com/evermeer/EVCloudKitDao

Thanks,

Brendan

[Brendan Duddridge]

Any takers?

I'm going forward with it but I've still got the question of how to best handle CBL attachments. 

Perhaps storing each attachment as a separate CKRecord would be ideal. If an attachment is added, updated, or deleted, I could just add, update, or delete the associated CKAsset object and have it removed from the CloudKit servers.

However, I'm pretty sure that CBL uniques the attachment .blob files. That is, if multiple CBLDocuments have the same file attached, there's only one instance of that attachment .blob file stored in the CBL database. Is that a correct assumption?

That's a very efficient way of storing the attachments. I would like to mimic that on CloudKit if I can to reduce storage consumption, but I also need to manage deleting them in CloudKit only when they're purged locally. If multiple CKRecords referenced the same CKAsset via a CKReference that could get tricky to keep the server in sync with the local. How does Sync Gateway handle that? Remember, CloudKit doesn't have any back-end smarts that I could write code to manage this issue. It all has to be done from the client side.

Thanks,

Brendan

On Saturday, August 27, 2016 at 7:02:54 PM UTC-6, Brendan Duddridge wrote:

[Jens Alfke]

I think you can make this work, but keeping track of document versioning will be difficult, especially in the case of conflicts.

This is actually similar to some design work I’ve been doing this year for future versions of Couchbase Mobile. I have some good mechanisms designed for replicating to servers that can’t store extensive metadata like revision trees … but the implementations are not yet solid enough to actually try out.

Unfortunately I haven’t had a chance to dig into the details of CloudKit, so I can’t give any detailed advice. Does it have some sort of MVCC mechanism that can prevent a writer from clobbering a change made by another writer? For example something like the ETag mechanism in HTTP (i.e. PUT with If-Match.)

—Jens

[Brendan Duddridge]

Does it have some sort of MVCC mechanism that can prevent a writer from clobbering a change made by another writer? For example something like the ETag mechanism in HTTP (i.e. PUT with If-Match.) 

I don't think so. Well at least I don't really know. It's higher level than that. All conflict management needs to be done on the client I believe.

CloudKit is really just a cloud storage service, but rather than just storing files on a server, you store CKRecords which is just a key/value store. Pretty much the same as an NSDictionary. Values can be a variety of types including Strings, Numbers, Locations, NSData, and file attachments (CKAsset). Plus arrays of these types.  CKReference objects are how you related objects together.

I was thinking that if I store CBLDocument properties as an archived NSData and replace the existing corresponding CKRecord object in CloudKit, then when that's transferred to another device, I just use the regular putExistingRevisionWithProperties to store it locally and then I could call my regular code that processes conflicts. Right now I'm just picking the most recently modified document as the winner.

[Jens Alfke]

On Sep 6, 2016, at 10:23 PM, Brendan Duddridge <bren...@gmail.com> wrote:

I don't think so. Well at least I don't really know. It's higher level than that. All conflict management needs to be done on the client I believe.

Looks like it does. From the docs:

When you save records, the value in the savePolicy property determines how to proceed when conflicts are detected on the server. Because records can be modified between the time you fetch them and the time you save them, the save policy determines whether new changes overwrite existing changes. By default, the operation object reports an error if a newer version of a record is found on the server. You can change the default setting to permit your changes to overwrite the server values wholly or partially.

This gives you some ability to handle conflicts, but it means that conflict resolution has to be done immediately: you can’t save data to the server until it’s resolved. This is still workable, though less flexible than Couchbase Mobile. It doesn’t interoperate well with it, though.

I think you could build a sync system using CBL where the clients communicate only through CloudKit. The revision tree stuff in CBL would just be unused overhead. But you can’t combine this with CBL’s replicator — I’ve tried to figure out how to, and it just doesn’t work. If revisions can get transferred both through CloudKit [or something similar] and through the normal replicator, the versioning can get seriously confused and Bad Stuff can happen, like false conflicts and clobbered writes.

—Jens

[Brendan Duddridge]

I think you could build a sync system using CBL where the clients communicate only through CloudKit. The revision tree stuff in CBL would just be unused overhead. But you can’t combine this with CBL’s replicator — I’ve tried to figure out how to, and it just doesn’t work. If revisions can get transferred both through CloudKit [or something similar] and through the normal replicator, the versioning can get seriously confused and Bad Stuff can happen, like false conflicts and clobbered writes.

I think that's ok. I would basically be giving customers the ability to choose CloudKit OR (Cloudant and/or Nearby sync). So if they choose CloudKit then it won't be using any of the CBLReplicator stuff.

But I'm assuming I would still use the call to putExistingRevisionWithProperties to store the data from CloudKit to the local database, providing the ordered set of revision history IDs and attachments?