Exception and can no longer search new index - Could not successfully complete backend operation due to repeated temporary exceptions after PT10S

[Jamie Scott]

I've recently created and enabled a new composite index made up of 2 properties, an index already exists for one of these properties. However I can no longer query using those parameters over millions of vertices.

Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT10S

        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:86)

        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)

        ... 74 more

Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend

        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:114)

        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:78)

        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getSlice(AstyanaxKeyColumnValueStore.java:67)

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:65)

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.ExpirationKCVSCache$2.call(ExpirationKCVSCache.java:91)

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.ExpirationKCVSCache$2.call(ExpirationKCVSCache.java:87)

        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)

        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)

        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)

        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)

        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)

        at com.google.common.cache.LocalCache.get(LocalCache.java:3937)

        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.ExpirationKCVSCache.getSlice(ExpirationKCVSCache.java:87)

        at com.thinkaurelius.titan.diskstorage.BackendTransaction$5.call(BackendTransaction.java:383)

        at com.thinkaurelius.titan.diskstorage.BackendTransaction$5.call(BackendTransaction.java:380)

        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)

        ... 75 more

Caused by: com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=172.31.42.125(172.31.42.125):9160, latency=5001(15003), attempts=3]TimedOutException()

        at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171)

        at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)

        at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)

        at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153)

        at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119)

        at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)

        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:538)

        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:112)

        ... 91 more

Caused by: TimedOutException()

        at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14696)

        at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14633)

        at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:14559)

        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)

        at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:741)

        at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:725)

        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:544)

        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:541)

        at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)

        ... 97 more

[Daniel Kuppitz]

What's the expected number of results? I guess it's just too big and thus runs into a timeout.

Cheers,

Daniel

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/587fca6d-3f0e-44c0-9b5b-6f2b9ecb1efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jamie Scott]

Is there anything that I could increase to stop the timeout?

[Jamie Scott]

I'm also just trying to return just .id()

Also tried adding .limit() but fails as looks like traversing them all.

This is the index status:

gremlin> report = mgmt.awaitGraphIndexStatus(graph, 'v_type-to_hydrate').call()

==>GraphIndexStatusReport[success=false, indexName='v_type-to_hydrate', targetStatus=REGISTERED, notConverged={to_hydrate=ENABLED, v_type=ENABLED}, converged={}, elapsed=PT1M0.108S]

gremlin> 

[Daniel Kuppitz]

Check your cassandra.yaml file, it contains a few timeout settings. However, if you have to deal with huge amounts of data, I would suggest to use OLAP jobs instead.

What's the actual query?

Cheers,

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/90167505-82f2-4e81-9db6-c160810f832b%40googlegroups.com.

[Jamie Scott]

g.V().has('to_hydrate',true).has('v_type','User').id().limit(100)

I think there is about 5 million possible nodes that match the query atm.

How would you suggest that I traverse these?

[Daniel Kuppitz]

:) Well, of course that's going to end up in timeouts (or OMEs). What are you trying to do with 5M results?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/c194e606-a067-41f9-8527-5a93367eaee3%40googlegroups.com.

[Jamie Scott]

I want to extract all the id's and store them else where for processing.

[Daniel Kuppitz]

Ok, that's an OLAP use case. The index is completely unnecessary and just produces a lot of overhead when you add new data.

In OLAP you can do this:

g.V().has('to_hydrate',true).has('v_type','User').id().store("x")

Once it's done you'll find the list of IDs stored in HDFS under the directory output/x.

Here's a simple example using HadoopGraph and Spark:

gremlin> graph = HadoopGraph.open("conf/hadoop/hadoop-graphson.properties")

==>hadoopgraph[graphsoninputformat->graphsonoutputformat]

gremlin> g = graph.traversal(computer(SparkGraphComputer))

==>graphtraversalsource[hadoopgraph[graphsoninputformat->graphsonoutputformat], sparkgraphcomputer]

gremlin> g.V().hasLabel("person").id().store("x").iterate()

gremlin> hdfs.head("output/x", ObjectWritable)

==>4

==>1

==>6

==>2

gremlin> 

It should work equally well for large result sets.

Cheers,

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/f0f487f7-c5bb-4add-9b4f-91603a153da5%40googlegroups.com.

[Daniel Kuppitz]

I found one that I used successfully a while back. You'll probably have to tune some parameters, but it should be a good foundation to start with:

#

# Hadoop Graph Configuration

#

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph

gremlin.hadoop.graphInputFormat=com.thinkaurelius.titan.hadoop.formats.cassandra.CassandraInputFormat

gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat

gremlin.hadoop.memoryOutputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat

gremlin.hadoop.deriveMemory=false

gremlin.hadoop.jarsInDistributedCache=true

gremlin.hadoop.inputLocation=none

gremlin.hadoop.outputLocation=output

#

# Cassandra Configuration

#

titanmr.ioformat.conf.storage.backend=cassandra

titanmr.ioformat.conf.storage.hostname=localhost

titanmr.ioformat.conf.storage.port=9160

titanmr.ioformat.conf.storage.keyspace=titan

#

# SparkGraphComputer Configuration

#

spark.master=local[*]

spark.executor.memory=400M

spark.serializer=org.apache.spark.serializer.KryoSerializer

spark.storage.memoryFraction=0.4

cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner

Cheers,

Daniel

On Thu, Nov 19, 2015 at 7:36 PM, Scott <ja...@atomicjets.com> wrote:

Great thanks

Do you have an example conf for connecting to cassandra along with hadoop, for the first query?

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/3ce2585b-7434-4ecd-a06e-6d9689b49c3f%40googlegroups.com.