Transitioning from CouchDB to Couchbase Server

[Foosh]

We're currently hosting our data with CouchDB using IrisCouch and want to transition to hosting with Couchbase Server. We're experiencing some difficulty moving our data over though. Is is possible to transition our data by replicating using the Sync Gateway? And if so, is there a guide or step-by-step instructions for moving from CouchDB to Couchbase Server? We've set up Couchbase Server on Amazon. We've set up the Sync Gateway. We created a new user using 

curl -X PUT localhost:4985/beer-sample/_user/bill --data '{"user":"bill", "password":"xyz", "disabled":false, "admin_channels":["*"]}'

Then, using Futon on IrisCouch, we tried to test a continuous pull replication with http://bill:xyz@ourServer:4984/beer-sample. We get back a message that says {"ok":true,"_local_id":"ecf3e071a5de3fee4627ff2c369b977e"} but no data seems to be replicating? If we do a one-time pull replication, it just seems to hang (no response is received and no documents are replicated). Are we missing a step? I appreciate any help!

Thanks,

Bill

[Jens Alfke]

> On Dec 30, 2014, at 8:06 PM, Foosh <bill....@kiddienotes.com> wrote:
>
> Then, using Futon on IrisCouch, we tried to test a continuous pull replication with http://bill:xyz@ourServer:4984/beer-sample.

You mean "push", not "pull", right? Pulling from SG to IrisCouch is the opposite of what you want.
That is, in your _replicate request the "source" should be the local (IrisCouch) db name, and the "target" should be the URL of the SG database.

Replication to/from CouchDB isn't part of our automated testing, but I do it periodically and it should be working. If a push from IrisCouch to SG doesn't work, the first thing to look at would be the SG logs.

—Jens

[Foosh]

You mean "push", not "pull", right? Pulling from SG to IrisCouch is the opposite of what you want.
That is, in your _replicate request the "source" should be the local (IrisCouch) db name, and the "target" should be the URL of the SG database. 

 

Replication to/from CouchDB isn't part of our automated testing, but I do it periodically and it should be working. If a push from IrisCouch to SG doesn't work, the first thing to look at would be the SG logs.

Thanks for the reply Jens and sorry for the confusion. I did mean "pull". We couldn't get push or pull replications working with our database so we decided to simplify things and just try pulling from the Beer-Sample bucket provided with Couchbase Server into an empty db on IrisCouch. I'll check the SG logs for any additional info.

Bill

[Foosh]

Looked at the logs and this is what I'm seeing 

04:05:41.638002 HTTP+: #005:     --> 200 OK  (0.0 ms)

04:07:42.308893 HTTP:  #006: HEAD /beer-sample/  (as bill)

04:07:42.308929 HTTP+: #006:     --> 200   (0.3 ms)

04:07:42.313941 HTTP:  #007: GET /beer-sample/  (as bill)

04:07:42.314102 HTTP+: #007:     --> 200   (0.3 ms)

04:07:42.319558 HTTP:  #008: GET /beer-sample/_local/e5350363872cc53f9687f65266055666  (as bill)

04:07:42.319673 HTTP: #008:     --> 404 missing  (0.3 ms)

04:07:42.325158 HTTP:  #009: GET /beer-sample/_local/5fa4ba97b640178d48f173d3f6d1c06d  (as bill)

04:07:42.325273 HTTP: #009:     --> 404 missing  (0.3 ms)

04:07:42.330379 HTTP:  #010: GET /beer-sample/_changes?style=all_docs&heartbeat=10000&since=0&feed=normal  (as bill)

04:07:42.330471 HTTP+: #010:     --> 200 OK  (0.0 ms)

Seems like it's complaining about some missing docs in the sample bucket? Is that right? Is there a way to get more detail?

[Foosh]

In addition to the log detail I posted, the error I'm getting back from the replication request is

{"error":"try_clause","reason":"{ok,stopped}"}

Here's the Response header

Cache-Control
must-revalidate
Content-Length
47
Content-Type
application/json
Date
Sat, 03 Jan 2015 21:39:26 GMT
Server
CouchDB/1.1.0 (Erlang OTP/R14B)

and the Request Header 
Accept
application/json, text/javascript, */*, application/json
Accept-Encoding
gzip, deflate
Accept-Language
en-US,en;q=0.5
Cache-Control
no-cache
Connection
keep-alive
Content-Length
73
Content-Type
application/json; charset=UTF-8
Cookie
AuthSession=YWRtaW46NTRBODYxN0M6ziX_UNdsanu2kF2GUX-yJZc5QYo
Host
myHost:5984
Pragma
no-cache
Referer
http://myHost:5984/_utils/replicator.html
User-Agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:34.0) Gecko/20100101 Firefox/34.0
X-Requested-With
XMLHttpRequest


Anywhere else I should look for details that might be helpful?

[Jens Alfke]

> On Jan 2, 2015, at 8:12 PM, Foosh <bill....@kiddienotes.com> wrote:
>
> Seems like it's complaining about some missing docs in the sample bucket? Is that right? Is there a way to get more detail?

No, the 404 isn't an error. It just means there's no existing saved checkpoint for that replication (since it's the first time that db is pulling.)

Is that all that's in the logs? In that case I'd guess that the changes feed was empty, i.e. there are no documents in the database that the user 'bill' can access.

—Jens

[Foosh]

Is that all that's in the logs? In that case I'd guess that the changes feed was empty, i.e. there are no documents in the database that the user 'bill' can access.

That's all that's in the logs. When starting the sync gateway, I used -verbose and -log=HTTP+,CRUD. Is there another flag that would yield more detail? 

I must be missing a step. I just want to replicate al documents from our IrisCouch database to Couchbase Server. So I set up a dummy user ("bill") in the target data bucket on Couchbase Server with "admin_channels":["*"] thinking that would give him permission to everything and allow all documents to replicate. Do I need to do something else to give this user the ability to pull all documents from IrisCouch? 

[Jens Alfke]

On Jan 3, 2015, at 6:45 PM, Foosh <bill....@kiddienotes.com> wrote:

I must be missing a step. I just want to replicate al documents from our IrisCouch database to Couchbase Server. 

Then why are you doing a pull from Couchbase Server to IrisCouch? That's the wrong direction.

(I pointed this out on the 1st thinking it was probably a typo in your email. Sounds like it isn't.)

—Jens

[Foosh]

Then why are you doing a pull from Couchbase Server to IrisCouch? That's the wrong direction.

(I pointed this out on the 1st thinking it was probably a typo in your email. Sounds like it isn't.)

Apologies again for the confusion. The pull was just to try get some form of replication working using the Couchbase Server sample data. 

More importantly to me, it doesn't work when I try a push replication either. When I try a push replication from IrisCouch to Couchbase Server that's when I get this error

{"error":"try_clause","reason":"{ok,stopped}"}

The Response header

Cache-Control
must-revalidate
Content-Length
47
Content-Type
application/json

Date
Sun, 04 Jan 2015 23:45:15 GMT
Server
CouchDB/1.1.0 (Erlang OTP/R14B)

The Request header

Accept
application/json, text/javascript, */*, application/json
Accept-Encoding
gzip, deflate
Accept-Language
en-US,en;q=0.5
Cache-Control
no-cache
Connection
keep-alive
Content-Length
73
Content-Type
application/json; charset=UTF-8

Cookie
AuthSession=YWRtaW46NTRBOUNGRjg6rP5aKjgbleieONyATFSurlMfYS8

Host
myHost:5984
Pragma
no-cache
Referer
http://myHost:5984/_utils/replicator.html
User-Agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:34.0) Gecko/20100101 Firefox/34.0
X-Requested-With
XMLHttpRequest

and all the log says is 

23:45:15.367032 HTTP:  #006: HEAD /dev_db/  (as bill)
23:45:15.367067 HTTP+: #006:     --> 200   (0.3 ms)
23:45:15.372316 HTTP:  #007: GET /dev_db/  (as bill)
23:45:15.372479 HTTP+: #007:     --> 200   (0.3 ms)
23:45:15.378246 HTTP:  #008: GET /dev_db/_local/4353b0d76c341a3e27cfbd24f2f3cbeb  (as bill)
23:45:15.378356 HTTP: #008:     --> 404 missing  (0.2 ms)
23:45:15.383492 HTTP:  #009: GET /dev_db/_local/ceb8208e4a2175eeb6d03953a4f7f884  (as bill)
23:45:15.383592 HTTP: #009:     --> 404 missing  (0.2 ms)
23:45:15.390480 HTTP:  #010: POST /dev_db/_missing_revs

[Jens Alfke]

On Jan 4, 2015, at 3:50 PM, Foosh <bill....@kiddienotes.com> wrote:

Apologies again for the confusion. The pull was just to try get some form of replication working using the Couchbase Server sample data. 

Well, if that's just Couchbase Server sample data then SG won't recognize it as being documents, since it wasn't created by SG and doesn't have the proper metadata.

and all the log says is 

...

23:45:15.390480 HTTP:  #010: POST /dev_db/_missing_revs

"_missing_revs" didn't ring a bell with me at all, and for a few minutes I thought it was some new API call, until I searched around and found that this has come up before, and it's actually a very old API call that was deprecated a long time ago, i.e. before CouchDB 1.2.

I have a hard time believing that IrisCouch is running that old a version of CouchDB, but you can check yourself by doing "GET /".

—Jens

[Foosh]

"_missing_revs" didn't ring a bell with me at all, and for a few minutes I thought it was some new API call, until I searched around and found that this has come up before, and it's actually a very old API call that was deprecated a long time ago, i.e. before CouchDB 1.2.

That was it! We keep our prod data on IrisCouch but the dev sandbox I was using to test the replication was actually an old instance of CouchDB (v1.1) on one of our servers. Replicating from a newer version of CouchDB to Couchbase Server worked without a hitch. 

Thanks Jens for all your help troubleshooting this!

Bill