XDCR problem with Couchbase-capi server

[Steffen Larsen]

Hi All,

I was discussing a XDCR problem with +Marty Schoch. Instead of just continuing our email discussion we both think that this might be a good subject to put in this group (to share some problems and solutions).

So to the problem. I am implementing the couchbase-capi-server, and have run into a couple of problems. From an enduser perspective in the couchbase GUI, I got this response back when adding a new replication to another host ("Create Replication" under the XDCR tab): "can't grab remote bucket info from any of known nodes". 

So after giving some debug info to Marty, we found that the first problem was to reply with a wrong response to some of the urls that the XDCR protocol needs to implement.

So when trying to request:

/pools/default

the CAPI server was returning:

"buckets":{"uri":"/pools/default/buckets?uuid=c21f969b-5f03-333d-83e0-4f8f136e7682"},"nodes":[{"ports":{"direct":9999},"couchApiBase":"http://zool.local:9999/default/","hostname":"zool.local"}]}

Here the couchApiBase was wrong. It does not have to take the bucket/pools in the URL. So the correct one was http://zool.local:9999, that is the hostname and the port.

I found out by looking at the elastic search plugin: 

https://github.com/couchbaselabs/elasticsearch-transport-couchbase/blob/master/src/main/java/org/elasticsearch/transport/couchbase/capi/ElasticSearchCouchbaseBehavior.java#L137

Now after having done the fix, I still have a problem when setting up replication from couchbase to my XDCR CAPI server. 

I get these XDCR Errors:

2013-06-20 13:59:52 - Error replicating vbucket 7: badarg

2013-06-20 13:59:52 - Error replicating vbucket 63: badarg

2013-06-20 13:59:52 - Error replicating vbucket 47: badarg

2013-06-20 13:59:52 - Error replicating vbucket 46: badarg

2013-06-20 13:59:52 - Error replicating vbucket 41: badarg

2013-06-20 13:59:52 - Error replicating vbucket 3: badarg

2013-06-20 13:59:52 - Error replicating vbucket 33: badarg

2013-06-20 13:59:52 - Error replicating vbucket 32: badarg

2013-06-20 13:59:52 - Error replicating vbucket 31: badarg

2013-06-20 13:59:52 - Error replicating vbucket 26: badarg

So I can register my CAPI server correctly, but when I setup replication of a bucket I get these errors (see also attachments). 

Marty or any others, can you guys help here?

I still don't get to the point where I receive anything from my code that implements CAPIBehaviour. 

-Cheers!

/Steffen

[Marty Schoch]

Hi Steffen,

At first glance, it looks to me like its reasonably happy with the Couchbase REST API responses and is trying to start the individual vbucket replicators.  Its unhappy with some data it received, which stops that vbucket replicator, and starts another.

A few things will be helpful at this point.

1.  Look at the Couchbase logs.  I'm fairly sure there is a more detailed message related to the "badarg", it will most likely be in the files named "error.*" or "xdcr.*".

2.  Please run the following requests against your server (substituting correct ports and authorization)

$ curl -v -XHEAD -uAdministrator:password http://localhost:9091/default

$ curl -v -XHEAD -uAdministrator:password http://localhost:9091/default%2f0

$ curl -v -XGET -uAdministrator:password http://localhost:9091/default

$ curl -v -XGET -uAdministrator:password http://localhost:9091/default%2f0

If you can post the output of these requests to pastebin or similar site that would help.

One other thing that might provide some useful information.  The couchbase-capi-server has some logging built-in.  We want to enable "trace" level debugging.  In the elasticsearch adapter we do this by setting:

com.couchbase: TRACE

Exactly how to enable it for your environment will depend on which actual logging implementation your'e using.  If you can get this enabled, the output will be very useful as at the TRACE level we log every request we get from Couchbase.

marty

/Steffen

--
You received this message because you are subscribed to the Google Groups "Couchbase" group.
To unsubscribe from this group and stop receiving emails from it, send an email to couchbase+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Steffen Larsen]

Hi again Marty!,

On Jun 20, 2013, at 4:50 PM, Marty Schoch <marty....@gmail.com> wrote:

Hi Steffen,

At first glance, it looks to me like its reasonably happy with the Couchbase REST API responses and is trying to start the individual vbucket replicators.  Its unhappy with some data it received, which stops that vbucket replicator, and starts another.

A few things will be helpful at this point.

1.  Look at the Couchbase logs.  I'm fairly sure there is a more detailed message related to the "badarg", it will most likely be in the files named "error.*" or "xdcr.*".

ok. nothing to see in the logs in the GUi. And under my server logs there is nothing than manifest.txt (Coucbase Server.app on a macOSX).

2.  Please run the following requests against your server (substituting correct ports and authorization)

$ curl -v -XHEAD -uAdministrator:password http://localhost:9091/default

HTTP/1.1 200 OK

$ curl -v -XHEAD -uAdministrator:password http://localhost:9091/default%2f0

HTTP/1.1 200 OK

$ curl -v -XGET -uAdministrator:password http://localhost:9091/default

Error 500 getDatabaseDetails are not supported

$ curl -v -XGET -uAdministrator:password http://localhost:9091/default%2f0

Error 500 getDatabaseDetails are not supported

Both responses are generated because I am throwing an unsupported exception my self. I did not thought that I needed to implement the getDatabaseDetails override method.

throw new UnsupportedOperationException("getDatabaseDetails are not supported");

So now I got a bit further. I have Implemented this method (returning an empty map instead of the Exception). What should it really return?

When I return the empty map I  now get to databaseExists in the CAPIBehaviour. Here I am just returning true?.. Is that correct?

If you can post the output of these requests to pastebin or similar site that would help.

One other thing that might provide some useful information.  The couchbase-capi-server has some logging built-in.  We want to enable "trace" level debugging.  In the elasticsearch adapter we do this by setting:

com.couchbase: TRACE

Exactly how to enable it for your environment will depend on which actual logging implementation your'e using.  If you can get this enabled, the output will be very useful as at the TRACE level we log every request we get from Couchbase.

ok Thanks!

You received this message because you are subscribed to a topic in the Google Groups "Couchbase" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/couchbase/DfLAu3MEzWk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to couchbase+...@googlegroups.com.

[Marty Schoch]

On Fri, Jun 21, 2013 at 1:54 PM, Steffen Larsen <zoo...@gmail.com> wrote:

Hi again Marty!,

On Jun 20, 2013, at 4:50 PM, Marty Schoch <marty....@gmail.com> wrote:

Hi Steffen,

At first glance, it looks to me like its reasonably happy with the Couchbase REST API responses and is trying to start the individual vbucket replicators.  Its unhappy with some data it received, which stops that vbucket replicator, and starts another.

A few things will be helpful at this point.

1.  Look at the Couchbase logs.  I'm fairly sure there is a more detailed message related to the "badarg", it will most likely be in the files named "error.*" or "xdcr.*".

ok. nothing to see in the logs in the GUi. And under my server logs there is nothing than manifest.txt (Coucbase Server.app on a macOSX).

The logs I was referring to can be found here on a Mac:

~/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/

 

2.  Please run the following requests against your server (substituting correct ports and authorization)

$ curl -v -XHEAD -uAdministrator:password http://localhost:9091/default

HTTP/1.1 200 OK

$ curl -v -XHEAD -uAdministrator:password http://localhost:9091/default%2f0

HTTP/1.1 200 OK

$ curl -v -XGET -uAdministrator:password http://localhost:9091/default

Error 500 getDatabaseDetails are not supported

$ curl -v -XGET -uAdministrator:password http://localhost:9091/default%2f0

Error 500 getDatabaseDetails are not supported

Both responses are generated because I am throwing an unsupported exception my self. I did not thought that I needed to implement the getDatabaseDetails override method.

throw new UnsupportedOperationException("getDatabaseDetails are not supported");

So now I got a bit further. I have Implemented this method (returning an empty map instead of the Exception). What should it really return?

When I return the empty map I  now get to databaseExists in the CAPIBehaviour. Here I am just returning true?.. Is that correct?

Yeah you will need to implement this method.  The only documentation about what you need to do is the Elasticsearch implementation.

1.  databaseExists() should return true for any destination database name you want to use.

2.  getDatabaseDetails() should return a map containing the field "db_name", the value should be the name of the database (without the UUID portion)

The relevant code is here:

https://github.com/couchbaselabs/elasticsearch-transport-couchbase/blob/master/src/main/java/org/elasticsearch/transport/couchbase/capi/ElasticSearchCAPIBehavior.java#L101-L103

Here are some sample requests/responses from the Elasticsearch adapter:

$ curl -uAdministrator:password -XGET http://localhost:9091/default

{"db_name":"default"}

$ curl -uAdministrator:password -XGET http://localhost:9091/default%2f0

{"db_name":"default/0"}

$ curl -uAdministrator:password -XGET "http://localhost:9091/default%2f0%3b00000000000000000000000000000000"

{"db_name":"default/0"}

NOTE: in these responses, the vbucket number should be included in the db_name (if it was in the original request), but the bucket UUID should not be included

marty

[Steffen Larsen]

Hi Marty!,

Now my reposes give the same result as your, which is good!:

$ curl -uAdministrator:password -XGET http://localhost:9091/default

{"db_name":"default"}

$ curl -uAdministrator:password -XGET http://localhost:9091/default%2f0

{"db_name":"default/0"}

$ curl -uAdministrator:password -XGET "http://localhost:9091/default%2f0%3b00000000000000000000000000000000"

{"db_name":"default/0"}

But I still get the similar fault on the GUI:

• 2013-06-29 17:05:19 - Error replicating vbucket 59: badarg
• 2013-06-29 17:05:19 - Error replicating vbucket 54: badarg
• 2013-06-29 17:04:49 - Error replicating vbucket 40: badarg
• 2013-06-29 17:04:49 - Error replicating vbucket 39: badarg
• 2013-06-29 17:04:19 - Error replicating vbucket 39: badarg
• 2013-06-29 17:04:19 - Error replicating vbucket 19: badarg
• 2013-06-29 17:03:49 - Error replicating vbucket 62: badarg
• 2013-06-29 17:03:49 - Error replicating vbucket 38: badarg
• 2013-06-29 17:03:19 - Error replicating vbucket 62: badarg
• 2013-06-29 17:03:19 - Error replicating vbucket 38: badarg

Looking in the logs (error.2), I repeatedly see these:

error_logger:error,2013-06-29T17:07:19.560,ns...@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]

=========================CRASH REPORT=========================

  crasher:

    initial call: xdc_vbucket_rep:init/1

    pid: <0.5608.0>

    registered_name: []

    exception exit: badarg

      in function  gen_server:terminate/6

    ancestors: [<0.2841.0>,<0.2836.0>,xdc_replication_sup,ns_server_sup,

                  ns_server_cluster_sup,<0.64.0>]

    messages: []

    links: [<0.2841.0>]

    dictionary: []

    trap_exit: true

    status: running

    heap_size: 28657

    stack_size: 24

    reductions: 14753

  neighbours:

[error_logger:error,2013-06-29T17:07:19.560,ns...@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.5609.0> terminating 

** Last message in was init

** When Server state == {init_state,

                            {rep,

                                <<"c21f969b-5f03-333d-83e0-4f8f136e7682/dk.yousee.metadata.epg/default">>,

                                <<"dk.yousee.metadata.epg">>,

                                <<"/remoteClusters/c21f969b-5f03-333d-83e0-4f8f136e7682/buckets/default">>,

                                [{connection_timeout,180000},

                                 {continuous,true},

                                 {http_connections,20},

                                 {retries,2},

                                 {socket_options,

                                     [{keepalive,true},{nodelay,false}]},

                                 {worker_batch_size,500},

                                 {worker_processes,4}]},

                            38,<0.2839.0>,<0.2840.0>,<0.2836.0>}

** Reason for termination == 

** badarg

-Hope you can help me in the right direction!

-Cheers!

/Steffen

[Marty Schoch]

This looks like its getting connection refused now.  I see that your curls are working against port 9091, but in the earlier examples you were return 9999 for the couchApiBase.  Did you change port you're listening on?  Did you make the corresponding change in the data returned by the couchbase API?

marty

--

[G.Maheswar reddy]

Hi Marty,


I followed this
Steps in this link
http://blog.couchbase.com/couchbase-and-full-text-search-couchbase-transport-elastic-search
in couchbase xdcr  i am getting following error while adding cluster

Attention - Error socket_closed_remotely happened during REST call get to http://127.0.0.1:9091/pools.

I am using elasticsearch version 0.90.3 and couchbase version 2.0.1 .But i am able to add cluster to couchbase in another node with elasticsearch and replicating is working.But with my system elasticsearch is not adding to couchbase cluster. Can you give me any solution for this problem.
Thanks you
.G.Mahesh

[Lasse Schou]

Hi,

You have to use ElasticSearch 0.20.2 because the transport plugin still doesn't support newer ES versions. Or you can do like this guy and build the latest version yourself: http://blog.jeroenreijn.com/2013/07/visitor-analysis-with-couchbase-elasticsearch.html

Hope that helps.

2013/8/14 G.Maheswar reddy <maheswar...@gmail.com>

[Marty Schoch]

Yes the currently released version only works with 0.20.2.  We plan on releasing an updated version in a few days.

marty