CBLReplicaton and conflicts

[Alan McKean]

I want to pull from a remote server on launch,. Here's what I'm doing in application:didFinishLaunching:withOptions:

  NSArray *replications = [database replicationsWithURL:[NSURL URLWithString:@"http://my_credentials@my_ip_and_port/my_db"] exclusively:YES];

  self.pull = [replications objectAtIndex:0];

  [self.pull start];

I assume it is not persistent since I am not setting it so, but I get a 409 conflict at the end of the pull on successive launches so it makes me think that I am creating another replication on top of the previous one.  Should I check for its existence before creating it on successive launches? Can someone suggest a better way of doing a pull replication on startup?

[Jens Alfke]

On Feb 12, 2014, at 4:49 PM, Alan McKean <alanm...@me.com> wrote:

> I assume it is not persistent since I am not setting it so,

Correct.

> but I get a 409 conflict at the end of the pull on successive launches so it makes me think that I am creating another replication on top of the previous one.

Hm, weird. You can enable the 'Sync' logging keyword [see the wiki for info on logging] and then look through the logs to see if multiple CBLPuller instances are running. Look for messages like
CBLPuller[http:/...] STARTING
and
CBLPuller[http:/...] Checkpointing sequence=…

—Jens

[Alan McKean]

I checked the wiki but didn't see anything about the SYNC keyword. I enabled logging with -LOG YES but don't see anything new.

[Alan McKean]

On another note, I would like an opinion on the decryption/encryption that I am using. I have a subclass of CBLMode called OTSCBLModel. It overrides modelForDocument:encrypted: like this:

+ (CBLModel *) modelForDocument:(CBLDocument *)document encrypted:(BOOL)encrypted {

  if(encrypted) {

    NSString *encryptionKey = [OTSKeychainHelper keychainStringFromMatchingIdentifier:PROVIDER_ENCRYPTION_KEY];

    NSDictionary *dictionary = [OTSCrypter decrypt:document.properties encryptionKey:encryptionKey];

    NSError *error;

    [document putProperties:dictionary error:&error];

  }

  return [self modelForDocument:document];

}

Unfortunately, due to the putProperties:error: call, I now have an unencrypted document in the db. What I want is to leave the original in the db encrypted and the in-memory properties unencrypted. I would like anything on disk to be encrypted but unencrypted for use in the app when I load them. Also, when I push, the server needs to get encrypted versions. Is there a way to 'putProperties' in the document that does not update the document on disk? Then, when I save the model, I would need to encrypt the in-memory document and save it back to disk for the push, I assume with putProperties:error:.

On Wednesday, February 12, 2014 4:58:02 PM UTC-8, Jens Alfke wrote:

[Jens Alfke]

On Feb 13, 2014, at 9:50 AM, Alan McKean <alanm...@me.com> wrote:

I checked the wiki but didn't see anything about the SYNC keyword. I enabled logging with -LOG YES but don't see anything new.

The wiki home page links to the doc section "Enabling Logging". Immediately below that, under "Useful Logging Channels", it lists the "Sync" keyword.

BTW, the flag is "-Log" not "-LOG" … these are all case sensitive.

To summarize, add two command-line args:

-Log YES

-LogSync YES

—Jens

[Jens Alfke]

On Feb 13, 2014, at 9:59 AM, Alan McKean <alanm...@me.com> wrote:

On another note, I would like an opinion on the decryption/encryption that I am using. I have a subclass of CBLMode called OTSCBLModel. It overrides modelForDocument:encrypted: like this:

(Nit: Technically that's not an override, it's a separate method with a different name.)

+ (CBLModel *) modelForDocument:(CBLDocument *)document encrypted:(BOOL)encrypted {

...

    [document putProperties:dictionary error:&error];

This isn't a safe thing to do. Instantiating a model for a document should not modify the document or have other side-effects.

What I want is to leave the original in the db encrypted and the in-memory properties unencrypted.

I don't think you'll be able to manage this without modifying CBLModel. If you look through its source code you'll see there are three places where `document.properties` is referenced. Those are places where the decryption would need to be inserted.

In addition, there is one call to -propertiesToSave in the -justSave: method. Right after this is where the encryption would need to happen.

Keep in mind that while this will let you access the actual properties via the CBLModel API, it won't make them available to map functions, so you still won't be able to do any queries against the encrypted properties.

Is there a way to 'putProperties' in the document that does not update the document on disk?

No. -putProperties is explicitly to save properties back to the database on disk. (And even if you could have this in-memory modification, it still wouldn't be visible to the map function so it still wouldn't work with queries.)

I think the real solution is going to be to hack some hooks into the replicator so that it can decrypt docs before adding them to the local db and encrypt them before uploading them. I don't have the bandwidth to do this right now, but I can offer advice if you want to do it.

—Jens

[Mark]

Hey Alan,

I played with this idea a few weeks ago for a future project. One suggestion is using the CBLModel and have the encrypted item as property that is your saved using the @dynamic keyword and a decrypted version that's not stored. When the model is loaded, decrypt the encrypted prop and drop it the decryptedDictionaryProperty. Then, either intercept the save function or encrypt and store on change of the decryptedDictionaryProperty.

The tricky bit is if you need something decrypted for your view. You could decrypt during the map phase, but the view indexes are stored and I didn't see a way to create an in-memory index (and I stopped working on it at this point as I didn't need it).

If that's not clear, I'll try to write up some code for you.

Good luck.

On Feb 13, 2014, at 9:59 AM, Alan McKean <alanm...@me.com> wrote:

> --
> You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/eeefb1a1-42bc-44b0-a3d6-4df00c89aee8%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Alan McKean]

The alternative (other than hacking the source for CBLModel) might be to create an alternate, decrypted representation of the CBLModel, e.g., OTSModel and use that in the app for display. Then when it comes time to save, get the original document and putProperties: (after encryption) back into the original. So far, I don't need to query the documents other than getting them by 'type' which is in the unencrypted part of the document.

[Alan McKean]

I would rather not put the decryption/encryption in the replicator because I like having the local database contents encrypted. Further, not all documents are encrypted, so it would have to do it selectively.

I have the source so I will check out those hooks in CBLModel where the properties are accessed and in justSave:'s propertiesToSave call. I'm not real happy about hacking it, though since any updates to CBL would have to have the same changes reapplied. I will follow up in another post with a few questions. Thanks.

BTW, I don't understand your comment about it not being safe. I am loading the document, modifying it with putProperties:error: and (after that) creating the model from it. Instantiating the model does not modify the document. It is already modified by the time I instantiate the model. Unless, of course, putProperties:error: is asynchronous.

On Thursday, February 13, 2014 10:24:45 AM UTC-8, Jens Alfke wrote:

[Alan McKean]

I am thinking along the same lines. A wrapper around the CBLDocument (if I don't hack CBLModel)

New class OTSModel (a subclass of NSObject):

1a) Either keeps the original CBLDocument or

or

1b) holds the original CBLDocument's documentID.

2) Decrypts the original document's properties dictionary into its own properties dictionary

3) Implement valueForUndefinedKey: and setValue:forUndefinedKey: to get at the properties in the dictionary. Or write the accessors.

4) Accessors get and set the values in the decrypted properties

5) Intercept save: with the model and encrypt the properties dictionary.

6a) Still in save:, putProperties:error: into the original document (if you have it as in 1a)

or

6b) Still in save: get the original document (if all you have is the docID as in 1b) and putProperties:error:  into it

I'm intrigued about hooking into the CBLModel though and will investigate what it will take to put the encryption/decryption into the CBLModel hooks.

[Alan McKean]

I don't understand the role of _changedNames, so I'm not sure where the decryption/encryption should occur:

- (NSDictionary*) currentProperties {

    NSMutableDictionary* properties = [_document.properties mutableCopy];

    // decrypt here?

    if (!properties)

        properties = [[NSMutableDictionary alloc] init];

    for (NSString* key in _changedNames)

        [properties setValue: _properties[key] forKey: key];

    // or here?

    return properties;

}

- (NSDictionary*) propertiesToSave {

    // encrypt here?

    NSMutableDictionary* properties = [_document.properties mutableCopy];

    if (!properties)

        properties = [[NSMutableDictionary alloc] init];

    for (NSString* key in _changedNames) {

        id value = _properties[key];

        [properties setValue: [self externalizePropertyValue: value] forKey: key];

    }

    [properties setValue: self.attachmentDataToSave forKey: @"_attachments"];

    // or here?

    return properties;

}

- (NSDictionary*) attachmentDataToSave {

    // anything to do here?

    NSDictionary* attachments = (_document.properties).cbl_attachments;

    if (!_changedAttachments)

        return attachments;

    

    NSMutableDictionary* nuAttach = attachments ? [attachments mutableCopy]

                                                : [NSMutableDictionary dictionary];

    for (NSString* name in _changedAttachments.allKeys) {

        // Yes, we are putting CBLAttachment objects into the JSON-compatible dictionary.

        // The CBLDocument will process & convert these before actually storing the JSON.

        CBLAttachment* attach = _changedAttachments[name];

        if ([attach isKindOfClass: [CBLAttachment class]])

            nuAttach[name] = attach;

        else

            [nuAttach removeObjectForKey: name];

    }

    return nuAttach;

}

On Thursday, February 13, 2014 10:24:45 AM UTC-8, Jens Alfke wrote:

[Mark]

I would look at subclassing CBLModel instead of NSObject, then you can use -didLoadFromDocument and -save to do your work.

> --
> You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/97982602-5726-4eac-830d-501821bad5a5%40googlegroups.com.

[Alan McKean]

Could I override the push mechanism in a subclass and use unsavedModels to get ahold of all of the models/documents that I have loaded and changed, then encrypt them back into the db before invoking the real push?

On Thursday, February 13, 2014 10:24:45 AM UTC-8, Jens Alfke wrote:

[Jens Alfke]

On Feb 13, 2014, at 12:57 PM, Alan McKean <alanm...@me.com> wrote:

I would rather not put the decryption/encryption in the replicator because I like having the local database contents encrypted.

The database is already encrypted, like all other document files on iOS. (And yes, the key for that encryption is in the keychain. But so is your AES key, so it's no less secure.)

But if you want to leave your docs AES-encrypted in the db, you'll have to do the hacking at a lower level, down in the CBLDatabase+*.m files and in CBLView+Internal.m — that's the only way to get map-reduce views to work. Any code that reads and writes the 'json' column of the 'revs' table will need to encrypt it on save and decrypt it on read. (And actually the same goes for the 'key' and 'value' columns of the 'maps' table, if those could contain sensitive data … which is a problem because the 'key' table has an index on it so it has to be sortable, which doesn't combine well with encryption.)

Further, not all documents are encrypted, so it would have to do it selectively.

You could add a property to the doc that indicates that it should be encrypted.

BTW, I don't understand your comment about it not being safe. I am loading the document, modifying it with putProperties:error: and (after that) creating the model from it. Instantiating the model does not modify the document. It is already modified by the time I instantiate the model.

It was your use of 'override' that threw me — I was at first reading the method as an override of +modelForDocument:, whose semantics are read-only. But you gave yours a different name so it can have different behavior.

But, now you've created new unencrypted revisions of the encrypted documents. If you have a push replication, it's going to upload those unencrypted revisions of every doc you've accessed (not just the ones you've intentionally changed) back to the server. That doesn't seem good.

—Jens

[Jens Alfke]

On Feb 13, 2014, at 1:01 PM, Alan McKean <alanm...@me.com> wrote:

I don't understand the role of _changedNames, so I'm not sure where the decryption/encryption should occur:

_changedNames is a set containing the property names that have been modified in the model but not saved.

The decryption should occur immediately after getting _document.properties, and the encryption should occur in the -justSave: method on the result of propertiesToSave.

You won't need to do any decryption in -attachmentDataToSave. That only accesses the document's _attachments property, which can't be encrypted because it's owned by Couchbase Lite itself and heavily used by the replicator.

—Jens

[Jens Alfke]

On Feb 13, 2014, at 1:45 PM, Alan McKean <alanm...@me.com> wrote:

Could I override the push mechanism in a subclass

No, CBLPusher isn't designed to be subclassed.

and use unsavedModels to get ahold of all of the models/documents that I have loaded and changed, then encrypt them back into the db before invoking the real push?

The replicator doesn't (and shouldn't) know anything about models; those are several layers above it in the architecture's hierarchy. Unsaved data in models is completely unknown to anything else in Couchbase Lite.

—Jens

[Alan McKean]

My first version of this (I may throw it and dig in to the replicator as you suggested). I can live with the db contents being unencrypted since, as you point out, it is encrypted in the Documents directory. Here is what I have so far and it seems to work with the few model classes that I have updated so far.

In my subclass of CBLModel:

// encrypted model subclasses implement init to set isEncrypted to YES

- (id) init {

  self = [super init];

  if(self) {

    self.decryptedProperties = [NSMutableDictionary dictionary];

    self.isEncrypted = NO;

  }

  return self;

}

- (void) didLoadFromDocument {

  // decrypt document

  NSString *encryptionKey = [OTSKeychainHelper keychainStringFromMatchingIdentifier:PROVIDER_ENCRYPTION_KEY];

  self.decryptedProperties = [OTSCrypter decrypt:self.document.properties encryptionKey:encryptionKey];

}

- (BOOL) save:(NSError *__autoreleasing *)outError {

  // encrypt doument

  if(self.isEncrypted) {

    NSString *encryptionKey = [OTSKeychainHelper keychainStringFromMatchingIdentifier:PROVIDER_ENCRYPTION_KEY];

    // encryption result is a dictionary: @{@“iv”:iv, @“data”:data} where iv is the initialization vector and data is a base 64 encoded string of encrypted data

    NSDictionary *encryptionResult = [OTSCrypter encrypt:self.decryptedProperties encryptionKey:encryptionKey];

    NSMutableDictionary * newProperties = [@{} mutableCopy];

    [newProperties setObject:[self.document.properties objectForKey:@"_id"] forKey:@"_id"];

    [newProperties setObject:[self.document.properties objectForKey:@"_rev"] forKey:@"_rev"];

    if(encryptionResult != nil) {

      [newProperties setObject:[self.document.properties objectForKey:@"type"] forKey:@"type"];

      [newProperties setObject:[encryptionResult objectForKey:@"iv"] forKey:@"iv"];

      [newProperties setObject:[encryptionResult objectForKey:@"data"] forKey:@"data"];

    }

    else {

      [newProperties setObject:@YES forKey:@"_deleted"];

      [newProperties setObject:[self.document.properties objectForKey:@"type"] forKey:@"type"];

    }

    [self.document putProperties:newProperties error:outError];

  }

  else {

    [super save:outError];

  }

  return !outError;

}

All models that are encrypted have to implement accessor methods to get and set the properties in self.decryptedProperties. Not elegant, so I’m still looking for a better way. Maybe the changes to the replicator as you mentioned. The real downside is that there are two copies of all of the properties in memory when the model object gets loaded: one dictionary in self.document.properties and the other in self.decryptedProperties.

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/jFADxJ8cV6Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/3FED640E-3C35-43FF-BB62-5B97BAE9713F%40couchbase.com.

[Jens Alfke]

Overriding -save: like that isn't going to work. The two immediate problems I see are:
(1) In the encrypted case you don't call the original save method, so in turn -didSave doesn't get called, which does some important bookkeeping like clearing the needsSave flag.
(2) There are code paths, like through +saveModels:error: and -[CBLDatabase saveAllModels], that don't call -save: but call the internal method -justSave: instead, so your code won't get called.

It'd be much better if you override -propertiesToSave: instead. Just call super, change the returned dictionary into one with the encrypted data, and return that.

—Jens

[Alan McKean]

Thanks. Done.

> --Jens

>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/jFADxJ8cV6Y/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/118E4877-39E8-484C-A7C6-66A432F2B968%40mooseyard.com.

[Alan McKean]

Is the method truly CBLModel propertiesToSave:? I can't override that one (I don't see it in the docs and i't doesn't autocomplete). I can override propertiesToSave (it's documented and it does autocomplete). Problem is, CBLModel propertiesToSave doesn't get called during CBLModel save:.

[Alan McKean]

I think I know what the problem is. propertiesToSave gets called on any of my CBLModel subclasses that are not encrypted. So when a model property changes, its name gets registered as having changed. But my CBLModel subclasses implement accessors to get and set the decryptedDictionary values. So the property names don't get registered as having changed. Is there a good way to tell the CBLModel that a given property has changed?

Here's an example of an accessor pair from one of my encrypted CBLModel subclasses:

- (NSNumber *) dateOfService {

  return [self.decryptedProperties objectForKey:@"dateOfService"];

}

- (void) setDateOfService:(NSNumber *)dateOfService {

  [self.decryptedProperties setObject:dateOfService forKey:@"dateOfService"];

[Jens Alfke]

On Feb 14, 2014, at 1:05 PM, Alan McKean <alanm...@me.com> wrote:

But my CBLModel subclasses implement accessors to get and set the decryptedDictionary values. So the property names don't get registered as having changed. Is there a good way to tell the CBLModel that a given property has changed?

This doesn't sound like a good direction to go in — you're working against the grain of CBLModel. Trying to make this work will get more and more hacky.

Instead, try what I suggested earlier — decode the document properties on their way into the model. There are two CBLModel methods that access _document.properties. Change those into calls to a new overrideable method, call it -documentProperties, that by default returns _document.properties.

So change both occurrences of

    NSMutableDictionary* properties = [_document.properties mutableCopy];

to

    NSMutableDictionary* properties = self.documentProperties;

and implement the method like this:

- (NSMutableDictionary*) documentProperties {

return [_document.properties mutableCopy];

}

Then you can override -documentProperties to decrypt the properties and return those. (I made the method return an NSMutableDictionary to reduce the amount of copying going on; if you're assembling a dictionary you'll probably have a mutable instance already, and both the callers want a mutable dictionary anyway.)

If this works for you, I can add it into CBLModel, especially if you send a pull request with a unit test :)

—Jens

[Alan McKean]

Your suggestions sound pretty straightforward, and I will give it a go, but if I am going to hack the source code, perhaps I should do as you suggested early on in the thread:

'I think the real solution is going to be to hack some hooks into the replicator so that it can decrypt docs before adding them to the local db and encrypt them before uploading them. I don't have the bandwidth to do this right now, but I can offer advice if you want to do it.’

Can you point me to the places in the source where I would decrypt during the pull and encrypt during the push? Then I can decide which modifications I would rather put in. I think that kind of modification would be generally more useful, even if it does leave the iOS db unencrypted. But as you pointed out, the file system is encrypted anyway.

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/jFADxJ8cV6Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/ACD6E2EA-CF16-40AB-990D-7B94BAC2AB52%40couchbase.com.

[Alan McKean]

Also, which target(s) in the CouchbaseLite project should I add my encryption classes to if I will be using the framework on iOS and, potentially, on the Mac?

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/2E0CB4AF-2C93-4E7F-A5AA-91BF65216F06%40me.com.

[Jens Alfke]

On Feb 14, 2014, at 5:05 PM, Alan McKean <alanm...@me.com> wrote:

Can you point me to the places in the source where I would decrypt during the pull and encrypt during the push?

Sure. Let's do this in the issue tracker so it's easier to find for future reference (and for eventual commits.)

I've copied over the old TouchDB issue and filed it as #253. See you there.

—Jens

[Jens Alfke]

On Feb 14, 2014, at 5:21 PM, Alan McKean <alanm...@me.com> wrote:

Also, which target(s) in the CouchbaseLite project should I add my encryption classes to if I will be using the framework on iOS and, potentially, on the Mac?

I wouldn't add the encryption directly to Couchbase Lite. To make this reuseable there should be a callback API to let apps install their own transformation functions. Then the encryption code will be in your app.

(But if you do want to start out by adding it directly to a private copy of Couchbase Lite, the targets to add it to would be "CBL Mac" and "CBL iOS Library".)

—Jens

[Alan McKean]

Just want to make sure you got the patch file that I sent you. I posted a couple of messages in the issue tracker but I'm not sure you saw them.