Getting started with Sync Gateway. Documents not syncing from server

[sc...@kujilabs.com]

I am having issues getting the sync_gateway to return any documents. I am not sure if I am missing anything. How do I get the server documents to sync to the gateway? I have read I might have to do this manually but how should this be done? How do I get new documents to sync ongoing. 

My config:

{

   "interface":":4984",

   "adminInterface":":4985",

   "log":["HTTP+","CRUD","REST+"],

   "databases":{

      "lib_development":{

         "server":"http://x.x.x.x:8091",   #with real ip

         "bucket":"lib_development",

         "sync":`function(doc) {channel(doc.channels);}`

      }

   },

   "pretty": true,

   "verbose": true

}

This creates these documents on the server

_sync:syncdata

{

  "Sync": "function(doc) {channel(doc.channels);}"

}

Ive added a user to the gateway

curl -X PUT localhost:4985/$DB/_user/fgynen --data '{"password":"password","admin_channels":["library"], "name":"fgynen"}'

And this user record gets synced to the "lib_development" bucket on the server.

_sync:user:fgynen

{

  "name": "fgynen",

  "admin_channels": {

    "library": 1

  },

  "all_channels": {

    "library": 1

  },

  "passwordhash_bcrypt": "JDJhJDEwJHB6Mmo0VFA1dHh0cXBhc3Jha1BiOHVvZWUvQjFteTJueEtHcFBhUVpWNVlVbFFEQnZ4Q2J5",

  "roles": []

}

Example server document:

{

  "type": "cbase_question",

  "section_id": 5,

  "name": "question",

  "channels": [

    "library"

  ]

}

Getting _all_docs on the gateway

curl -X POST fgy...@127.0.0.1:4984/pea_development/_all_docs?include_docs=true

{"rows":[

],

"total_rows":0}

[Jens Alfke]

On Oct 21, 2013, at 5:22 AM, sc...@kujilabs.com wrote:

I am having issues getting the sync_gateway to return any documents. I am not sure if I am missing anything. How do I get the server documents to sync to the gateway? I have read I might have to do this manually but how should this be done? How do I get new documents to sync ongoing. 

It sounds like you have a pre-existing Couchbase bucket and want to use it with the Sync Gateway. The beta 1 version doesn’t support this; it wants to manage the bucket all on its own.

The latest version (in the ‘master’ branch on Github) will recognize existing documents in the bucket when it starts up, and will add the necessary metadata (a “_sync” property) to them so they’ll be replicated to clients. To enable this, in the database configuration JSON (i.e. where you configure “bucket” and “pool”) set the property “importDocs” to true. It can also optionally detect new docs being added while it’s running, if you set “importDocs” to “continuous”.

But it cannot recognize changes made to existing synced documents; we’re still working on that. In the meantime if you want to change documents you can do so through the gateway’s admin API (defaulting to port 4985.)

—Jens

[sc...@kujilabs.com]

Ah thanks for the info. I think I may have misunderstood how this whole architecture is meant to work. 

I have a web app that creates documents in couchbase server and I was expecting the sync gateway to sync these documents to my mobile app with couchbase lite. Is this the eventual goal? 

I thought it might be something similar to couchdb replication. With documents automatically bidirectionally synced. 

It seems though that I need my web app to create its documents via the sync gateway instead. Is this correct? Would it also make sense to read the documents from the gateway too? (from the web app)

Thanks again.

[Ryan Payne]

This is exactly what I am trying to accomplish as well. I came across this yesterday and am using the latest build now. It works great for picking up newly added documents now, but as mentioned does not recognize document changes yet.

Ideally I am looking to have a central server that is updated with an application on the server side through the native CB server SDK's, then have the devices just sync to that bucket. Will that be attainable in the near future or should I build my application around inserting/updating the docs through the Sync Gateway REST API as opposed to using the server directly? I realize the point of contention is the revision tracking, just wondering if that's something in the pipeline. Thanks

[sc...@kujilabs.com]

I think this answers all of the questions

https://github.com/couchbase/sync_gateway/issues/7#issuecomment-25638983

Looking forward for this to be supported!

[sc...@kujilabs.com]

and this

https://github.com/couchbase/sync_gateway/wiki/Couchbase-Server-Smart-Client-SDKs

[Jens Alfke]

On Oct 24, 2013, at 3:13 AM, sc...@kujilabs.com wrote:

Ah thanks for the info. I think I may have misunderstood how this whole architecture is meant to work. 

No, you’re just running into unimplemented features. :/

I thought it might be something similar to couchdb replication. With documents automatically bidirectionally synced. 

That is the goal. It’s pretty hard to attain, though, because Couchbase Server doesn’t have the kind of document metadata (i.e. revision IDs, revision histories) that CouchDB does. We’re trying our best to work around that.

It seems though that I need my web app to create its documents via the sync gateway instead. Is this correct? Would it also make sense to read the documents from the gateway too? (from the web app)

Right now, writing through the gateway is the best approach. Reading through it is optional, but may be simpler for you since you can then use the same API (i.e. some CouchDB client library) for both writes and reads.

—Jens

[Matt Hunter]

Right now, writing through the gateway is the best approach. Reading through it is optional, but may be simpler for you since you can then use the same API (i.e. some CouchDB client library) for both writes and reads.

I've run into the same issues described in this thread - and now understand how it's a current limitation (thanks Jens) - but I have a question about what you said regarding using the Gateway for for writing.  

We have a webapp that uses Couchbase.  We home to use couchbase lite and the sync gateway to tie our mobile app into the same data.  Obviously this won't work ATM, as the data from the webapp won't sync.  So... how can I create new documents, perform updates, etc via the Sync Gateway?  If I just make POSTs for new documents... and PUTs for existing documents, will that work?  Is there any extra metadata I need to include?  etc...  A working example using CURL would be awesome.

Thanks!

M@

[Jens Alfke]

On Nov 12, 2013, at 8:33 AM, Matt Hunter <technol...@gmail.com> wrote:

So... how can I create new documents, perform updates, etc via the Sync Gateway?  If I just make POSTs for new documents... and PUTs for existing documents, will that work?  Is there any extra metadata I need to include?  etc...  A working example using CURL would be awesome.

Yes, that works. The API isn’t yet in our documentation, but it’s basically the same as the CouchDB REST API; are you familiar with that? There’s a good intro with ‘curl’-based examples here.

It’s basically standard REST, except that when updating or deleting a document you need to provide a “_rev” property that contains the revision ID of the current revision; this is a key part of the multi-version concurrency control (MVCC) system.

—Jens

[Matt Hunter]

Yes, that works. The API isn’t yet in our documentation, but it’s basically the same as the CouchDB REST API; are you familiar with that? There’s a good intro with ‘curl’-based examples here.

It’s basically standard REST, except that when updating or deleting a document you need to provide a “_rev” property that contains the revision ID of the current revision; this is a key part of the multi-version concurrency control (MVCC) system.

Perfect - thanks :)  We're switching to Couchbase from CouchDB in order to get Couchbase Lite, so this is a no-brainer.  

-M@ 

[Matt Hunter]

Perfect - thanks :)  We're switching to Couchbase from CouchDB in order to get Couchbase Lite, so this is a no-brainer.  

^^^ - famous last words ;)

New question - I can't seem to access views via the sync gateway.  Is this something that can be configured in the sync gateway... or is it just not possible?  I was hoping to put all traffic from the webapp to Couchbase through the sync gateway (for simplicity)... but I'm guessing that I really do have to use Couchbase's REST API for views?

Thanks again,

M@ 

[Jens Alfke]

On Nov 13, 2013, at 6:39 PM, Matt Hunter <technol...@gmail.com> wrote:

New question - I can't seem to access views via the sync gateway.  Is this something that can be configured in the sync gateway... or is it just not possible?

That’s not implemented — it wasn’t necessary for replication.

You should be able to use the Couchbase API on the bucket to create and query views. Just make sure your map function

(a) ignores all documents whose IDs begin with “_sync:” (those are internal things created by the gateway)

(b) ignores the “_sync” property of a document (that’s where the gateway stores metadata like the revision history)

—Jens