Replication help needed

[Thiago Alencar]

So I've started to experiment with replication last weekend (meaning: newbie), unfortunately without success with a very simple case: Maybe I'm still missing some important concept?

In my test, I just wanted to be able to generate some data (with a user logged in), then go to another device, and be able to see this data using the same user (after logging in).

Here's the scenario:

-Couchbase Lite based app using custom authentication: this is working already, I get the session cookies and I'm able to see the documents synced the Couchbase server.

Test 1: 

-User A fires the app in his first device, registers himself, the app calls:

 [push/pull setCookieNamed:@"SyncGatewaySession" withValue:sessionValue path:nil expirationDate:nil secure:NO];

He gets his own document created with a [database documentWithID: hisUniqueUsername];

The following replication methods are called:

- (void)defineSync

{

    pull = [_database createPullReplication:_remoteURL];

    push = [_database createPushReplication:_remoteURL];

    pull.continuous = push.continuous = YES; 

    [self listenForReplicationEvents:push];

    [self listenForReplicationEvents:pull];

}

- (void)startSync {

    [pull start];

    [push start];

}

The document gets created and synced to database. In my shadow bucket it looks something like this:

{

  "type": "contacInfo",

  "username": "userA"

}

So the user enters some data, and saves the document, e.g.: 

{

  "type": "contacInfo",

  "blobData" :  "Some user data",

  "username": "userA"

}

I can see in the logs that the replication works as soon as the [save] method is called, so far everything as expected...

Now, you might have predicted this one: After running the same pattern in another device (but skipping registration and using this same user), I realise that I created a conflict because: the initDocumentWith ID will generate a new document, but with the same ID. This document is newer, so I believe sync gateway will make this the winning revision from the conflict, and I won't be able to see "blobData" in the second device.

So I went ahead and changed the order of things: 

Test 2- 

After logging in, I first wait for the replication to complete, and just then I call [database existingDocumentWithID], in the hope that now I'll get the document instead of creating a new one.

But no, I'm still not able to find the document and therefore I have no access to the "blobData" generated from the first device.

Okay, maybe I could create another document with "blobData" associating the owner to his user's  unique ID and find it with a query, but it feels more like a workaround (?).

And already to advance / confirm: right now I have basically nothing in my sync function -  but if I ever call requireUser("userA") there, this will lookup the current session_id from the request, and see if it matches with the user's who is doing this request, thus one can be sure that sync gateway knows whether the client doing request to change the document, is who he claims to be, correct? In other words, if the session gets validated, I can be confident that "requireUser()"  will block any updates not coming from that user, correct ?  

Anybody could shed some light here?  

Thanks in advance,

Thiago

[Jens Alfke]

On Sep 29, 2014, at 5:25 AM, Thiago Alencar <thia...@gmail.com> wrote:

This document is newer, so I believe sync gateway will make this the winning revision from the conflict

That's correct as long as by "newer" you mean "has more revisions" (i.e. higher generation count) There isn't any chronological timestamp on a document.

After logging in, I first wait for the replication to complete, and just then I call [database existingDocumentWithID], in the hope that now I'll get the document instead of creating a new one.

But no, I'm still not able to find the document and therefore I have no access to the "blobData" generated from the first device.

Sounds like the document isn't in any channel your user account can read.

And already to advance / confirm: right now I have basically nothing in my sync function

The default sync function will assign the channels listed in the document's "channels" property. Since you didn't add such a property to your document, it's not in any channels. So unless you gave the user access to the wildcard "*" channel, the user won't be able to see the document, explaining your problem.

There are many ways to solve this, depending on your apps' goals. A simple one is:

• Define a private channel for each user, named something like "user-" + username
• When creating a user account, add that channel to their admin_channels property
• When creating documents, add a "channels" property whose value is an array containing the user's channel

An even simpler one is to give every user account access to the "*" channel, although you'd only want that for experimentation, probably.

This isn't the first time new developers have run into this, and I'm realizing that it's a problem with the Gateway's default settings — they create a situation where you can't read your own writes, which is unexpected.

—Jens

[Thiago Alencar]

Hi Jens,

Thank you so much for your answer. Sorry for this long post, but its with good intentions.

Well, I said "basically there's nothing in the sync function" because I wasn't worrying about it at that point that much, but if I am honest with you I did try as well playing around with the channels before asking here. See my latest sync function had:

function(doc) {

        channel(doc.channels);

        if(doc.type == "contactInfo"){

        channel("public");

        access(doc.username, "public");

        } }

And it still wasn't working. But after I read your answer, I was feeling more confident, so I went to review my code, again and..  Can you find the mistake: ?

+ (NSString*) type{ return @"contacInfo";  }

Yes, look again. Nooo!! missing a "t" !! damn it :-)

BUT,  this was just one of the many tests I was doing while trying to make the sync work. So before you stop reading this: 

Yes, I think as well that this situation (not the missing "t", but the channel thing) is unexpected: however, I think this is more a documentation issue than a design issue (I'm totally fine assigning a channel to a document, as long as I know I really have to in order for it to sync). 

I think the main problem is that this is explained just later on, in the channel section of the sync gateway, so one can't learn "linearly" following the guide from beginning to end. I know there's a thin / blurry line between where to document somethings that are related both to couchbase lite and the replication sync gateway. But if you agree with me, here's is my opinion on where, how and why the documentation can be improved:

The documentation says right in the beginning of the replication topic:

"The application code doesn't have to pay attention to the details: it just knows that when it makes changes to the local database they will eventually be uploaded to the server, and when changes occur on the server they will eventually be downloaded to the local database. The app's job is to make the UI reflect what's in the local database, and to reflect user actions by making changes to local documents. "

Later on, the it mentions channels again, but this time inside the *Filters* topic: 

 A special type of filter used with the Couchbase Sync Gateway is the set of channels that a pull replication will download from. It's also possible to limit a replication to an explicit set of document IDs.

I didn't want to filter anything - but just get everything owned by that user. Still, I was changing a document, but I wasn't getting it back (even without setting up "filters", and even after using the same document ID). If we follow this line of thought, seeing the replication as purely self-contained mechanism to get an identical copy of the local database everywhere, what I write on the "local" database should be always visible from the app's perspective no matter if it originated locally (initWithID), or if it came after a sync (to the local database), just like you said. So I agree with you that its kind of misleading.

Of course, once you know how it works, it becomes "obvious", but "obvious" is context-dependent and I think the main goal of the guides is to be clear for beginners on how to use the tool - so if you do agree with me, can we please ask somebody to at least add a big bold note in the cb lite's replication documentation, stating that: a document will *not* be actually synchronised if it is not set to any channel !?  Okay, the sync gateway's documentation makes this much more clear in the "channels" section (but again, since this is a requirement, I think this should be clear right in the beginning / introduction).

I don't know the internals of the project's syncing to give an opinion on its design or gw's defaults > so its more up to you whether to change that or the documentation. Either way I'd say its an improvement.

BR,

Thiago

[Thiago Alencar]

There are many ways to solve this, depending on your apps' goals. A simple one is:

• Define a private channel for each user, named something like "user-" + username

• When creating a user account, add that channel to their admin_channels property

• When creating documents, add a "channels" property whose value is an array containing the user's channel

Regarding the approach suggested of assigning each user to a private channel: does that scale well ?  By well I mean, at the degree that a couchbase server scales?

I'm asking because I read somewhere that the gateway doesn't use local storage (only RAM) - correct me if I'm wrong; So I got concerned about the scaling if doing it this way. Can we use and abuse of channels as we wish?

Thiago

[Jens Alfke]

On Sep 30, 2014, at 3:10 AM, Thiago Alencar <thia...@gmail.com> wrote:

Regarding the approach suggested of assigning each user to a private channel: does that scale well ?  By well I mean, at the degree that a couchbase server scales?

The current implementation will scale to large numbers of channels but not vast numbers. I'm guessing on the order of 10k active channels; it's hard to be specific because it depends on how the channels are used, how active the clients are, how much RAM the server has, etc.

—Jens

[Thiago Alencar]

I suppose this the same case mentioned in the Chat app data model :

"So in the chat app, we have channels for each chat room, but also a channel for user profiles. (I left it out of this document, but if you look at the actual implementation, you'll see there's also a channel for each user, which syncs the chat room documents for the rooms they have access to. This is needed because of a limitation in the implementation that we plan to fix, so maybe by the time you read this it's no longer necessary.)"

If so, has this already been implemented, or is it in the roadmap?

By the way, I've tried with the wildcard channel access(doc.username, "*") but it didn't work in my tests: the document didn't make it to the "other side". I'm running couchbase lite and sync gateway community edition 1.0.2. Either I'm missing something.. or it's this bug: https://github.com/couchbase/sync_gateway/issues/432 ? If I could set as well channel("*") for every document, that would be cheating meaning that "*" is not actually a "special keyword", but is rather being treated as any other channel. 

Anyway I'd suggest at least changing the default sync function from:

function (doc) {
   channel(doc.channels);
}

to something like: 

function (doc) {
   channel("*");
   var this_user = "Is there a way to get the username of this request context?"
   access(this_user, "*");
}

BR,

Thiago

[Jens Alfke]

On Oct 1, 2014, at 3:08 AM, Thiago Alencar <thia...@gmail.com> wrote:

This is needed because of a limitation in the implementation that we plan to fix, so maybe by the time you read this it's no longer necessary.)"

I no longer remember exactly what I meant by that, but I think the limitation was the Gateway not sending the client existing docs when the client got access to a new channel. That's since been implemented (although it didn't work reliably till 1.0.2.)

By the way, I've tried with the wildcard channel access(doc.username, "*") but it didn't work in my tests: the document didn't make it to the "other side".

We ran into this in another context yesterday. Adding the channel "*" to a document does not in reality make it visible to all users, even though it's documented that it does. (I filed that as #432. It would be a nice feature to have, but the implementation would be nontrivial.)

—Jens