Hector write error when one node is down

[Shamik Bandopadhyay]

Hi,

While trying to test the failover, I ran into write error. Here's
my scenario. I've 4 nodes with 1 seed. I start writing 25000 records
using 2 concurrent threads. While the write is in progress, I brought
down one of the non-seed nodes. This is what I noticed.

1. Writing goes into a pause.

2. The 3 nodes prints the following message

INFO 02:45:30,577 error writing to /10.192.223.15
INFO 02:45:32,596 InetAddress /10.192.223.15 is now dead.

No activity is recorded after that.

3. The error from hector

Aug 15, 2010 2:46:55 AM
me.prettyprint.cassandra.service.CassandraClientFactory
createThriftClient
SEVERE: Unable to open transport to 10.192.223.15:9160
org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
me.prettyprint.cassandra.service.CassandraClientFactory.createThriftClient(CassandraClientFactory.java:
89)
at
me.prettyprint.cassandra.service.CassandraClientFactory.create(CassandraClientFactory.java:
71)

Aug 15, 2010 2:46:55 AM
me.prettyprint.cassandra.service.FailoverOperator operate
WARNING: Skip-host failed
me.prettyprint.cassandra.service.SkipHostException:
org.apache.thrift.transport.TTransportException: Unable to open
transport to 10.192.223.15:9160 , java.net.ConnectException:
Connection refused
at
me.prettyprint.cassandra.service.FailoverOperator.skipToNextHost(FailoverOperator.java:
244)
at
me.prettyprint.cassandra.service.FailoverOperator.operateSingleIteration(FailoverOperator.java:
192)
at
me.prettyprint.cassandra.service.FailoverOperator.operate(FailoverOperator.java:
84)
at
me.prettyprint.cassandra.service.KeyspaceImpl.operateWithFailover(KeyspaceImpl.java:
151)

Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)

Aug 15, 2010 2:46:55 AM
me.prettyprint.cassandra.service.FailoverOperator
operateSingleIteration
WARNING: Got a TTransportException from 174.129.112.50. Num of
retries: 3 (thread=http-8090-2)
Aug 15, 2010 2:46:55 AM
me.prettyprint.cassandra.service.FailoverOperator skipToNextHost
INFO: Skipping to next host (thread=http-8090-2). Current host is:
174.129.112.50
Aug 15, 2010 2:46:55 AM
me.prettyprint.cassandra.service.FailoverOperator invalidate
INFO: Invalidating client CassandraClient<174.129.112.50:9160-7>
(thread=http-8090-2)
Aug 15, 2010 2:46:55 AM
me.prettyprint.cassandra.service.CassandraClientFactory
createThriftClient
SEVERE: Unable to open transport to 10.192.223.15:9160
org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
me.prettyprint.cassandra.service.CassandraClientFactory.createThriftClient(CassandraClientFactory.java:
89)
at
me.prettyprint.cassandra.service.CassandraClientFactory.create(CassandraClientFactory.java:
71)
at
me.prettyprint.cassandra.service.CassandraClientFactory.makeObject(CassandraClientFactory.java:
141)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:
1148)

I ran the cassandra logs in debug, but after one died, the remaining
stopped responded after the message at point 2.

I'm bit puzzled since I thought the fail over is built in and
Cassandra is quick to respond to the situation through other running
nodes.

I'm using CL=1 and RF=1.

Am I missing something in the config?

Any pointers will be highly appreciated.

Thanks

[Shamik Bandopadhyay]

I tried changing the CL to ANY and QUARAM, but the Cassandra behavior
remain unchanged

[Nathan McCall]

With an RF of 1 means that your data exists only in one place, it
*does not* mean "make one copy". If you take a node down, that data is
inaccesible and their is no replica node for hinted handoff to work
correctly. See: http://wiki.apache.org/cassandra/HintedHandoff for
details on this.

That said, CL.ANY should have hinted locally on the receiving node,
but you may have gotten your cluster in a bad state by that point.

In general, RF=1 should be considered localhost development mode only
- I can't think of a viable use case for this in production.

[Shamik Bandopadhyay]

Nathan, thanks for your continuous support. With CL=ANY and RF=1 and 4
nodes running,my understanding was that the co-ordinating node will
write the hint locally till the actual node comes back live.It'll then
make a replica of the data.But that doesn't seem to in my test cases.

Just clearing my doubts on couple of related things. While using hint
for CL=ANY and RF=1, the co-ordinating will store the data in its own
disk, is that a correct assumption?

Also, when we write data with CL=1, and RF=3, Cassandra will return
the response to the client when the first replica is written to first
derived node.The remaining couple of replica will happen in an
asynchronous process. Am I correct on this?

Sorry for asking some basic stuff, but I'm just trying to clear my
understanding based on the result I'm observing after running my test
scenarios.

[Nathan McCall]

That is my understanding as well, but I would guarantee very few
people have tried this (RL=1 on multiple nodes) as it a very a-typical
usecase and generally falls into the category of "don't do that".
Hinted Handoff has undergone a lot of change since that wiki page was
written, so I would have to code dive to tell you the answer to that
definitely.

You are correct on your assumption of behaviour for CL=1 and RF=3.

[Shamik Bandopadhyay]

Thanks Nathan, appreciate all your help...