Could not read messages for timestamp [2016-08-25T22:43:12.538Z] (this read will be retried)

[Sachin]

Hi Guys,

I am trying to run reindex command in our DB. This is what I ran

// Open a graph

graph = TitanFactory.open("conf/titan.properties")

g = graph.traversal()

// Block until the SchemaStatus transitions from INSTALLED to REGISTERED

mgmt = graph.openManagement()

report = mgmt.awaitGraphIndexStatus(graph, "indexName").call()

// Run a Titan-Hadoop job to reindex

//mgmt = graph.openManagement()

mr = new MapReduceIndexManagement(graph)

mr.updateIndex(mgmt.getGraphIndex("indexName"), SchemaAction.REINDEX).get()

As soon as I ran this, I started getting the error -

22:46:56.236 [RingDescribeAutoDiscovery] WARN  c.n.a.impl.RingDescribeHostSupplier - Failed to get hosts from titan via ring describe.  Will use previously known ring instead

22:46:57.134 [pool-75-thread-1] WARN  c.t.t.diskstorage.log.kcvs.KCVSLog - Could not read messages for timestamp [2016-08-25T22:43:12.538Z] (this read will be retried)

com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception

at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:144) ~[titan-core-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:703) ~[titan-core-1.0.0.jar:na]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S

at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:86) ~[titan-core-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na]

... 9 common frames omitted

Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:114) ~[titan-cassandra-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:78) ~[titan-cassandra-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getSlice(AstyanaxKeyColumnValueStore.java:67) ~[titan-cassandra-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:769) ~[titan-core-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:766) ~[titan-core-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:133) ~[titan-core-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.util.BackendOperation$1.call(BackendOperation.java:147) ~[titan-core-1.0.0.jar:na]

at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na]

... 10 common frames omitted

Caused by: com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0]No hosts to borrow from

at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.<init>(RoundRobinExecuteWithFailover.java:48) ~[astyanax-core-3.8.0.jar:3.8.0]

at com.netflix.astyanax.connectionpool.impl.TokenAwareConnectionPoolImpl.newExecuteWithFailover(TokenAwareConnectionPoolImpl.java:83) ~[astyanax-core-3.8.0.jar:3.8.0]

at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352) ~[astyanax-core-3.8.0.jar:3.8.0]

at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:538) ~[astyanax-thrift-3.8.0.jar:3.8.0]

at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:112) ~[titan-cassandra-1.0.0.jar:na]

... 17 common frames omitted

22:46:58.648 [RingDescribeAutoDiscovery] WARN  c.n.a.impl.RingDescribeHostSupplier - Failed to get hosts from titan via ring describe.  Will use previously known ring instead

22:46:59.703 [RingDescribeAutoDiscovery] WARN  c.n.a.impl.RingDescribeHostSupplier - Failed to get hosts from titan via ring describe.  Will use previously known ring instead

22:47:02.284 [pool-93-thread-1] WARN  c.t.t.diskstorage.log.kcvs.KCVSLog - Could not read messages for timestamp [2016-08-25T22:43:13.787Z] (this read will be retried)

Please advice how to fix this.

[Jason Plurad]

Hi Sachin,

Have you tried switching from cassandra to cassandrathrift?

-- Jason