Distributed Cluster and Firewall

[Srikanth]

Hi,

We have a 4 node cluster, with 2 nodes in one DC and two nodes in
another DC. Nodes in each DC have been assigned a zone id, and hence,
we have two zones configured. I have setup the 4 nodes in full
replication.

We are observing that after some time, the Voldemort Client in DC1
complains that DC2 nodes are not accessible.
If we restart the voldemort nodes in DC2, then, issues goes away for
some time, but then comes back again.

I am suspecting that the issue could be the firewall that sits between
the two DCs. We had similar issue with RabbitMQ server and we
resolved it by using keep-alive messages. RabbitMQ had an option to
specify an interrval at which keep alive (or equialent) messages were
exchanged.

I was looking for similar option for Voldemort. When I did some
search in google group's past entries, I found mention of a patch that
enabled keep alive. However, I am not sure whether it is available in
0.90.1 and how to configure it.

Any inputs in this regard will be of great help.

Thanks,
Srikanth

[Srikanth]

When the nodes in DC1 is writing entries to a store, we see below
warnings.

Since we used StoreClient.applyUpdate method with 3 retries, the
opertion is attempted 4 times by the client and after which following
message appears.

2012-05-09 16:21:19,577 ["http-bio-9000"-exec-3] WARN
[routed.action.PerformSerialPutRequests] - Error in put on node
0(DC2Host1)
voldemort.store.UnreachableStoreException: Client response not read/
parsed, cannot determine result
at
voldemort.store.socket.clientrequest.AbstractClientRequest.getResult(AbstractClientRequest.java:
78)
at
voldemort.store.socket.clientrequest.BlockingClientRequest.getResult(BlockingClientRequest.java:
65)
at voldemort.store.socket.SocketStore.request(SocketStore.java:
248)
at voldemort.store.socket.SocketStore.put(SocketStore.java:
207)
at voldemort.store.socket.SocketStore.put(SocketStore.java:67)
at
voldemort.store.routed.action.PerformSerialPutRequests.execute(PerformSerialPutRequests.java:
95)
at voldemort.store.routed.Pipeline.execute(Pipeline.java:214)
at
voldemort.store.routed.PipelineRoutedStore.put(PipelineRoutedStore.java:
506)
at
voldemort.store.routed.PipelineRoutedStore.put(PipelineRoutedStore.java:
66)
at voldemort.store.logging.LoggingStore.put(LoggingStore.java:
121)
at voldemort.store.DelegatingStore.put(DelegatingStore.java:
69)
at
voldemort.store.stats.StatTrackingStore.put(StatTrackingStore.java:
128)
at
voldemort.store.stats.StatTrackingStore.put(StatTrackingStore.java:39)
at
voldemort.store.serialized.SerializingStore.put(SerializingStore.java:
137)
at voldemort.store.DelegatingStore.put(DelegatingStore.java:
69)
at
voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:256)