Couchbase java client writing data to cluster when 1 node failed

[Егор Голощапов]

Hi all. 

What I'm trying to do - writing time series data all the time, even if one of two nodes failed. 

Please help me figure out, how to handle this situation.

Very simple case:

• two servers (147.54.160.217; 147.54.160.218) in one cluster with shared bucket (and one replica)
  
• couchbase enterprise version 2.5 (windows 7)
• java couchbase client [<dependency><groupId>com.couchbase.client</groupId><artifactId>couchbase-client</artifactId><version>1.4.0-dp</version></dependency>]

It is the latests client and servers.

I've made a couchbase project to show what and how I'm doing (attached in zip file). It is maven project so I hope it will be very simple to start it in any IDE.

In that project there are 2 classes:

• CouchbaseUtil.java (singleton that provide nice API to interract with nodes using couchbase-API)
  
• CouchbaseMain.java (infinite cycle that produces data and asks CouchbaseUtil.java to store it )
  

I'm generating data and ask CouchbaseUtil to save it in infinite cycle. After that I'm disabling couchbase service in one of the servers and expecting client to continue writing to working one, but couchbase client trying to save data in not working host and never switches to working host. Do I need to handle it manually? If I do, how can I do it? I've read that couchbase client is smart enough to handle it for me. 

Please, take a minute,  look at my project, and show me where I'm wrong.

Thank you very much!

[Aliaksey Kandratsenka]

It looks like you are expecting couchbase server to behave in a way that it's not designed to behave.

I cannot give you some pointers to read. But will try to explain in few sentences. Couchbase Server (or client) does not automatically switch to remaining nodes. It's equivalent of sharded mysql with slaves as backup. If node dies or goes down temporarily, it's portion of data will become unavailable. Until some entity (admin for example) fails over that portion of data to replica. As a convenience we have autofailover feature than can automatically fail over node in some common and safe cases.

Some call it CP (i.e. from CAP) but it's very wrong to say that. It's just plain old master-slave replication sharded smartly for scaling and with nice cluster management on top.

The only thing where client is expected one way or other way to switch to live server(s) is client queries for cluster configuration (google for vbucket map). Because we normally keep all nodes aware of cluster configuration so any node can be asked for vbucket map.

[Егор Голощапов]

Aliaksey Kandratsenka ,

thank you very much for the answer.

Could you please tell me, is it possible to make couchbase client (java client for example) write/read data from cluster even if one node fails? If it is, how it can be done? 

[Matt Ingenthron]

From: Егор Голощапов <golo...@gmail.com>
Reply-To: "couc...@googlegroups.com" <couc...@googlegroups.com>
Date: Thursday, March 13, 2014 3:17 AM
To: "couc...@googlegroups.com" <couc...@googlegroups.com>
Subject: Re: Couchbase java client writing data to cluster when 1 node failed

Could you please tell me, is it possible to make couchbase client (java client for example) write/read data from cluster even if one node fails? If it is, how it can be done? 

The best way to get back to availability of both reads and writes is to trigger a failover.  This does imply momentary lack of availability with fairly quick recovery.

Before the failover, you can access other data with the "replica read" feature.  See:

http://www.couchbase.com/autodocs/couchbase-java-client-1.3.2/com/couchbase/client/CouchbaseClient.html#getFromReplica(java.lang.String)

http://docs.couchbase.com/couchbase-devguide-2.5/#replica-read

Hope that helps,

Matt

-- 

Matt Ingenthron

Couchbase, Inc.

[Aliaksey Kandratsenka]

I don't think it's possible with couchbase. It's simply not part of design.

There are systems that have that ability to be always available even in face of more than half nodes failing. E.g. riak. But they give up consistency while doing this. In fact even in absence of failures such systems are not fully consistent (i.e. some clients may see stale values). Another downside is because they have to accept writes in face of network partition, they have to be able to converge independent mutations of same doc. That converge logic is called conflict resolution and it has to be implemented in application. So building against this model can be quite tricky.

We believe that what couchbase implements is good set of tradeoffs:

* it's "in practice" consistent (easy to program against)

* it's fast (because client has exactly one node to work with for any given id)

* it's simple (no paxos and no vector clocks or revision trees; and that means less code and less bugs; and also ease of understanding and tuning and scaling etc)

* it's scalable

* if node fails, only subset of data becomes unavailable and only until failover is done. Given that node or network failures are infrequent and given that failover is quick operation, we believe that periods of unavailability are infrequent and short.