Documents missing from shadow but in primary

[Jeremy Kelley]

I am seeing a case where I can create documents within a shadowed bucket (primary), and they are not showing up in any form in the shadow bucket and therefore are not getting sync'd to remote clients ever.

I have seen nothing in the sync gateway logs that could indicate errors occurred.  I did lose a node in my cluster that failed over but even after replacing it and everything being rebalanced I am seeing this occur.

TWO QUESTIONS

ONE - Is there a common misconfiguration that could cause this to happen that I could look for?

TWO - If I were to just shut down sync gateway and then delete the shadow bucket and recreate it and restart sync gateway, would it "do the right thing"?  Would it just copy all documents over into the shadow and then each device could just resync and move forward? 

Thanks,

Jeremy

[Jens Alfke]

On Mar 13, 2015, at 8:01 AM, Jeremy Kelley <jer...@33ad.org> wrote:

I am seeing a case where I can create documents within a shadowed bucket (primary), and they are not showing up in any form in the shadow bucket and therefore are not getting sync'd to remote clients ever.

Do you mean that no documents you create in the primary bucket get copied into the Sync Gateway bucket? Or is it intermittent? If the latter, is there anything those documents might have in common, like document IDs or properties?

ONE - Is there a common misconfiguration that could cause this to happen that I could look for?

No, this sounds like a bug. File a bug report, please!

TWO - If I were to just shut down sync gateway and then delete the shadow bucket and recreate it and restart sync gateway, would it "do the right thing"?  Would it just copy all documents over into the shadow and then each device could just resync and move forward? 

Sort of … but you probably don’t want to do this. You’d end up with all the same documents with the same IDs, but you’d lose all the replication metadata that was kept in the old shadow bucket. Most importantly, each document’s revision ID would reset to first generation (starting with “1-….”). The clients would interpret this as a conflict in each document, with unpredictable results. :/

Have you tried just shutting down the Gateway and restarting it? It’s been a long time since I worked on the bucket shadowing code, but I think that might re-scan the primary bucket and import missing documents.

—Jens

[Jeremy Kelley]

On Fri, Mar 13, 2015 at 10:40 AM Jens Alfke <je...@couchbase.com> wrote:

On Mar 13, 2015, at 8:01 AM, Jeremy Kelley <jer...@33ad.org> wrote:

I am seeing a case where I can create documents within a shadowed bucket (primary), and they are not showing up in any form in the shadow bucket and therefore are not getting sync'd to remote clients ever.

Do you mean that no documents you create in the primary bucket get copied into the Sync Gateway bucket? Or is it intermittent? If the latter, is there anything those documents might have in common, like document IDs or properties?

Unfortunately, intermittent.  I will paste an example of documents below.  I see no rhyme or reason as to what could cause this.

 

ONE - Is there a common misconfiguration that could cause this to happen that I could look for?

No, this sounds like a bug. File a bug report, please!

Will do!  I'll attach some example documents below.  There's no difference between the ones that are showing up and the ones that weren't.

 

TWO - If I were to just shut down sync gateway and then delete the shadow bucket and recreate it and restart sync gateway, would it "do the right thing"?  Would it just copy all documents over into the shadow and then each device could just resync and move forward? 

Sort of … but you probably don’t want to do this. You’d end up with all the same documents with the same IDs, but you’d lose all the replication metadata that was kept in the old shadow bucket. Most importantly, each document’s revision ID would reset to first generation (starting with “1-….”). The clients would interpret this as a conflict in each document, with unpredictable results. :/

Have you tried just shutting down the Gateway and restarting it? It’s been a long time since I worked on the bucket shadowing code, but I think that might re-scan the primary bucket and import missing documents.

THIS fixed it. Restarting the sync gateway process.  I have it on my slate today to spin up 3 more instances of sync gateway as we're picking up steam in our beta rollout.  For now I'll just schedule a cron job to do rolling restarts on them.  Elegance at work!

-jeremy

attached below - pastes of the documents from the primary db.

in primary but not in shadow

{

   "modifiers": [

   ],

   "co": "1a3713cb92af4b71b7847cd81ab8fef4",

   "name": "Testf",

   "descr": "None",

   "price": 5,

   "updated_at": "2015-03-13T04:39:07.678526",

   "taxable": true,

   "folder": "Materials",

   "type": "catgitem",

   "created_at": "2015-03-13T04:39:07.678456",

   "unit": ""

}

{

   "id": "249db358e4fd4093b4357a0b33e365e2",

   "rev": "1-0000511e19afba760000000000000000",

   "expiration": 0,

   "flags": 0

}

{

   "modifiers": [

   ],

   "co": "1a3713cb92af4b71b7847cd81ab8fef4",

   "name": "testd2",

   "descr": "None",

   "price": 5,

   "updated_at": "2015-03-13T04:37:50.381921",

   "taxable": true,

   "folder": "Materials",

   "type": "catgitem",

   "created_at": "2015-03-13T04:37:50.381873",

   "unit": ""

}

{

   "id": "fe6656a65c6648a5b71d43ecdece2357",

   "rev": "3-0000510c1a6e40060000000000000000",

   "expiration": 0,

   "flags": 0

}

----------------------------------------------

is there

{

   "modifiers": [

   ],

   "co": "1a3713cb92af4b71b7847cd81ab8fef4",

   "name": "testb",

   "descr": "None",

   "price": 5,

   "updated_at": "2015-03-13T04:31:18.528785",

   "taxable": true,

   "folder": "Materials",

   "type": "catgitem",

   "created_at": "2015-03-13T04:31:18.528737",

   "unit": ""

}

{

   "id": "9a61d84ae75446ec838c224c18f5924e",

   "rev": "1-000050b0de2c083b0000000000000000",

   "expiration": 0,

   "flags": 0

}

[Adam Fraser]

My initial thought is that the Sync Gateway's feed that's monitoring changes in the shadowed bucket had dropped it's connection to one or more of the shadowed vbuckets.  This should be getting handled automatically - I've added a note to https://github.com/couchbase/sync_gateway/issues/722 for further investigation.

Thanks for the detailed report - it's helpful.

Adam

[Jeremy Kelley]

On Fri, Mar 13, 2015 at 11:23 AM Adam Fraser <adamc...@gmail.com> wrote:

My initial thought is that the Sync Gateway's feed that's monitoring changes in the shadowed bucket had dropped it's connection to one or more of the shadowed vbuckets.  This should be getting handled automatically - I've added a note to https://github.com/couchbase/sync_gateway/issues/722 for further investigation.

I also `skimmed` the documentation for sync gateway looking for an api endpoint to force a "sync" (for lack of a better term) between the primary and the shadow.  Maybe tell it to do an audit of some sort... Did I miss that?

Thanks all for the help on this,

Jeremy

 

 

[Jens Alfke]

On Mar 13, 2015, at 9:27 AM, Jeremy Kelley <jer...@33ad.org> wrote:

I also `skimmed` the documentation for sync gateway looking for an api endpoint to force a "sync" (for lack of a better term) between the primary and the shadow.  Maybe tell it to do an audit of some sort... Did I miss that?

There isn’t an API for it, because it’s something that should Just Work™ :/

Restarting the Sync Gateway will effectively do that, however.

—Jens