Could not successfully complete backend operation due to repeated temporary exceptions after PT2M

[Scott P]

I'm testing a master branch build of 0.2.0. It's a single JanusGraph with a single Cassandra. After around 14 hour of creating elements, it looks like my C stops working for a little while? Is that what this error means. My janusgraph is still up and running and appears to be working.

org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT2M

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)

at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:469)

at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:268)

at org.janusgraph.diskstorage.BackendTransaction.edgeStoreMultiQuery(BackendTransaction.java:300)

at org.janusgraph.graphdb.database.StandardJanusGraph.edgeMultiQuery(StandardJanusGraph.java:434)

at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getVertices(StandardJanusGraphTx.java:417)

at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction.vertices(JanusGraphBlueprintsTransaction.java:140)

at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph.vertices(JanusGraphBlueprintsGraph.java:134)

... 9 more

Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:123)

at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore$$Lambda$187.00000000840EB030.apply(Unknown Source)

at io.vavr.API$Match$Case0.apply(API.java:3174)

at io.vavr.API$Match.of(API.java:3137)

at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:120)

at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore$$Lambda$72.00000000EC605650.apply(Unknown Source)

at io.vavr.control.Try.getOrElseThrow(Try.java:671)

at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:277)

at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77)

at org.janusgraph.diskstorage.keycolumnvalue.cache.ExpirationKCVSCache.getSlice(ExpirationKCVSCache.java:92)

at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:271)

at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:268)

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)

... 20 more

Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [localhost/127.0.0.1:9042] Timed out waiting for server response))

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)

at io.vavr.concurrent.Future$$Lambda$78.00000000EC607520.apply(Unknown Source)

at io.vavr.control.Try.of(Try.java:62)

at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)

at io.vavr.concurrent.FutureImpl$$Lambda$79.00000000EC6079F0.run(Unknown Source)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)

... 4 more

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [localhost/127.0.0.1:9042] Timed out waiting for server response))

at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:211)

at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:46)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:275)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:440)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:418)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:778)

at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1374)

at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:588)

at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:662)

at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:385)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)

... 1 more

[Robert Dale]

That's what it looks like to me - cassandra is not responding within the timeout.  Did you ever try those batch size settings?  I wonder if you're hitting major GC pauses in cassandra.  Might want to turn on gc logging to rule it out.

Robert Dale

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/0b71b352-9e84-428f-84ce-d4561260436e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Scott P]

I did make the change the cql batch size as you suggested. I'm going to try reducing the young generation size for cassandra.

Robert Dale

To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.