Possible Concurrency Issue? out of sequence response

[Luke Evans]

Hello,

Currently I am interacting with Titan from a Java Spring project with a Groovy "GraphAlgorithms" class pretty much as shown here: https://github.com/tinkerpop/gremlin/wiki/Using-Gremlin-through-Groovy 

Everything works great most of the time, but on certain intense queries where many threads interact with the Groovy class at the same time, I get the following stack trace: 

[ERROR] [executorWithPoolSizeRange-19 11:36:57] (CountingConnectionPoolMonitor.java:trackError:81) com.netflix.astyanax.connectionpool.exceptions.ThriftStateException: ThriftStateException: [host=10.170.104.78(10.170.104.78):9160, latency=66858(66858), attempts=1]org.apache.thrift.TApplicationException: get_slice failed: out of sequence response
com.netflix.astyanax.connectionpool.exceptions.ThriftStateException: ThriftStateException: [host=10.170.104.78(10.170.104.78):9160, latency=66858(66858), attempts=1]org.apache.thrift.TApplicationException: get_slice failed: out of sequence response
	at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:177)
	at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
	at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:185)
	at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:177)
	at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
	at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
	at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
	at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:175)
	at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getSlice(AstyanaxOrderedKeyColumnValueStore.java:117)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferedKeyColumnValueStore.getSlice(BufferedKeyColumnValueStore.java:40)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.CachedKeyColumnValueStore.getSlice(CachedKeyColumnValueStore.java:91)
	at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore.getSlice(ConsistentKeyLockStore.java:102)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction$1.call(BackendTransaction.java:164)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction$1.call(BackendTransaction.java:161)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:61)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:230)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:161)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.edgeQuery(StandardTitanGraph.java:180)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$5$2.get(StandardTitanTx.java:589)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$5$2.get(StandardTitanTx.java:586)
	at com.thinkaurelius.titan.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:52)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$5.execute(StandardTitanTx.java:586)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$5.execute(StandardTitanTx.java:539)
	at com.thinkaurelius.titan.graphdb.query.QueryProcessor.getUnwrappedIterator(QueryProcessor.java:56)
	at com.thinkaurelius.titan.graphdb.query.QueryProcessor$OuterIterator.<init>(QueryProcessor.java:132)
	at com.thinkaurelius.titan.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:48)
	at com.google.common.collect.Iterables$7.iterator(Iterables.java:611)
	at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getProperty(AbstractVertex.java:106)
	at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getProperty(AbstractVertex.java:122)
	at com.tinkerpop.pipes.filter.PropertyFilterPipe.processNextStart(PropertyFilterPipe.java:28)
	at com.tinkerpop.pipes.filter.PropertyFilterPipe.processNextStart(PropertyFilterPipe.java:13)
	at com.tinkerpop.pipes.AbstractPipe.hasNext(AbstractPipe.java:98)
	at com.tinkerpop.pipes.util.Pipeline.hasNext(Pipeline.java:105)
	at com.tinkerpop.pipes.filter.BackFilterPipe.processNextStart(BackFilterPipe.java:33)
	at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
	at com.tinkerpop.pipes.transform.LabelPipe.processNextStart(LabelPipe.java:14)
	at com.tinkerpop.pipes.transform.LabelPipe.processNextStart(LabelPipe.java:11)
	at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
	at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
	at com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
	at com.tinkerpop.gremlin.java.GremlinPipeline.fill(GremlinPipeline.java:1575)
	at com.tinkerpop.gremlin.java.GremlinFluentPipeline$fill$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
	at com.tinkerpop.gremlin.java.GremlinFluentPipeline$fill$0.call(Unknown Source)
	at com.winston.graphdb.GraphAlgorithms.edgesBetweetTwoIDs(GraphAlgorithms.groovy:71)
	at com.winston.graphdb.GraphAlgorithms$edgesBetweetTwoIDs$3.callStatic(Unknown Source)
	at com.winston.graphdb.GraphAlgorithms.defineConnections(GraphAlgorithms.groovy:232)
	at com.winston.graphdb.GraphManager.findConnections(GraphManager.java:52)
	at com.winston.metadata.MetaDataFetcher.fetchConnections(MetaDataFetcher.java:37)
	at com.winston.metadata.MetaDataFetcher$$FastClassByCGLIB$$42d45c58.invoke(<generated>)
	at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:80)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.thrift.TApplicationException: get_slice failed: out of sequence response
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
	at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:594)
	at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:578)
	at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamilyQueryImpl.java:190)
	at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamilyQueryImpl.java:177)
	at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
	... 57 more

This results in a timeout from my application.

Here is my groovy class:

class GraphAlgorithms { 

    static {
        Gremlin.load()
    } 

    private static Graph g; 

    public static void init(Graph graph) {
        g = graph;
    } 

    public static String defineConnections(String id) {
        // call methodB
    } 

    public static Vertex edgesBetweetTwoIDs(String s) { 

                   // Some work

              }

}

And the Java Spring class:

@Component("graphmanager")
public class GraphManager { 

    // Titan Graph
    Graph graph; 

    @PostConstruct
    public void init() {
        Configuration conf = new BaseConfiguration();
        conf.setProperty("storage.backend","cassandra");
        conf.setProperty("storage.hostname","<cassandra_ip>");
        graph = TitanFactory.open(conf);
        GraphAlgorithms.init(graph);
    } 

    public String findConnections(String s) {
        String vString = GraphAlgorithms.methodA(s);
        return vString;
    }
} 

The "findConnections" method in the Java class is called from many threads simultaneously, and I'm wondering if this is where I'm getting myself into trouble. I tried making the threads use the same TransactionalGraph as described here: https://github.com/thinkaurelius/titan/wiki/Multi-Threaded-Transactions but I got the same results. 

1) Would trying to hunt down a concurrency bug be a waste of time? (could it be something with my cassandra cluster instead?)

2) Does it even make sense to organize my groovy class as shown above? Or should an entire transaction's worth of reads be contained in one method? Currently, my groovy methods pass around Vertex objects..

Any help would be great! Thanks!

Luke

[Stephen Mallette]

I'm not sure what the specific nature of your problem is, but Rexster exposes a single Graph instance just as you seem to be trying to do here.  Since it works for Rexster (and other models I've seen that try to use a single Graph instance), I would think it can work for Spring (though, i'm not terribly familiar with Spring these days).  One thing that is important to remember is that you close the transaction in some way (rollback/commit) for each request (even for just reads).  I remember strange things happening in Rexster when it wasn't doing that properly (though I can't say it specifically produced this error).

Stephen

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Luke Evans]

Thanks for the reply, Stephen.

I wasn't sure if I needed to commit when none of my transactions had writes so this helps a lot. 

Also, over the weekend I came across your post about Script Engines with Rexster, and I think this would be the most performant solution for me anyway. I really like the idea of moving the heavy lifting to the server with <init-script> and <imports>.

I'm still going to play around with my current implementation with Spring + Groovy for curiosity's sake. 

Thanks!

Luke