Lock Exceptions when creating Index

[Abhinav A]

Hi all,

I'm running into a strange exception that I've been trying to solve for the past few hours. 

Setup

1. 3 Cassandra nodes

2. 1 Titan node running Titan Server with Rexster

Problem:

When trying to create an index on an empty graph using gremlin, it keeps giving the familiar "Could not commit transaction due to exception during persistence" error.

The strange thing is this only happens when creating an index. We've ensured that no clients are writing to Cassandra or Titan and that transactions from gremlin are committing successfully when doing things like adding vertices and edges. Another strange issue is that the same code that's failing here runs successfully when running Titan locally with embedded Cassandra.

Code

The gremlin session looks like this:

gremlin> g = TitanFactory.open("bin/remote.yaml")
13/07/02 18:18:27 INFO impl.ConnectionPoolMBeanManager: Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterTitanConnectionPool,ServiceType=connectionpool
13/07/02 18:18:28 INFO impl.CountingConnectionPoolMonitor: AddHost: c1.asdf.com
13/07/02 18:18:28 INFO impl.CountingConnectionPoolMonitor: AddHost: c2.asdf.com
13/07/02 18:18:28 INFO impl.CountingConnectionPoolMonitor: AddHost: c3.asdf.com
13/07/02 18:18:28 INFO impl.ConnectionPoolMBeanManager: Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
13/07/02 18:18:28 INFO impl.CountingConnectionPoolMonitor: AddHost: c1.asdf.com
13/07/02 18:18:28 INFO impl.CountingConnectionPoolMonitor: AddHost: c2.asdf.com
13/07/02 18:18:28 INFO impl.CountingConnectionPoolMonitor: AddHost: c3.asdf.com
==>titangraph[cassandra:c1.asdf.com]
gremlin> g.makeType().name("uuid").dataType(String.class).indexed(Vertex.class).unique(Direction.BOTH).makePropertyKey();
==>v[36028797018964170]
gremlin> g.commit()
13/07/02 18:18:41 INFO util.BackendOperation: Temporary storage exception during backend operation [PersistingTransaction]. Attempting incremental retry
com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Lock failed: exceeded max timeouts [LockClaim [backer=com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore@3acdb1f1, key=0x1-117-117-105-228, col=0x132, expectedValue=null]]
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockTransaction.writeBlindLockClaim(ConsistentKeyLockTransaction.java:250)
...

And the contents of remote.yaml are:

storage.backend=cassandra

storage.hostname=c1.asdf.com,c2.asdf.com,c3.asdf.com

I've tried everything I can think of. Any advice on how to proceed would be appreciated.

Thanks,

--Abhi

Here's the full stack trace in case it helps:

13/07/02 18:22:44 INFO util.BackendOperation: Temporary storage exception during backend operation [PersistingTransaction]. Attempting incremental retry
com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Lock failed: exceeded max timeouts [LockClaim [backer=com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore@25ba59dc, key=0x1-117-117-105-228, col=0x132, expectedValue=null]]
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockTransaction.writeBlindLockClaim(ConsistentKeyLockTransaction.java:250)
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore.acquireLock(ConsistentKeyLockStore.java:137)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireVertexIndexLock(BackendTransaction.java:153)
at com.thinkaurelius.titan.graphdb.database.IndexSerializer.lockKeyedProperty(IndexSerializer.java:111)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:255)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:203)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:61)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.save(StandardTitanGraph.java:277)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:839)
at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:41)
at com.tinkerpop.blueprints.TransactionalGraph$commit.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at groovysh_evaluate.run(groovysh_evaluate:56)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:67)
at org.codehaus.groovy.tools.shell.Interpreter$evaluate.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:65)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:78)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.main(Console.java:104)
13/07/02 18:22:45 INFO util.BackendOperation: Temporary storage exception during backend operation [PersistingTransaction]. Attempting incremental retry
com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Lock failed: exceeded max timeouts [LockClaim [backer=com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore@25ba59dc, key=0x1-117-117-105-228, col=0x132, expectedValue=null]]
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockTransaction.writeBlindLockClaim(ConsistentKeyLockTransaction.java:250)
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore.acquireLock(ConsistentKeyLockStore.java:137)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireVertexIndexLock(BackendTransaction.java:153)
at com.thinkaurelius.titan.graphdb.database.IndexSerializer.lockKeyedProperty(IndexSerializer.java:111)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:255)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:203)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:61)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.save(StandardTitanGraph.java:277)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:839)
at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:41)
at com.tinkerpop.blueprints.TransactionalGraph$commit.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at groovysh_evaluate.run(groovysh_evaluate:56)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:67)
at org.codehaus.groovy.tools.shell.Interpreter$evaluate.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:65)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:78)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.main(Console.java:104)
13/07/02 18:22:46 INFO util.BackendOperation: Temporary storage exception during backend operation [PersistingTransaction]. Attempting incremental retry
com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Lock failed: exceeded max timeouts [LockClaim [backer=com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore@25ba59dc, key=0x1-117-117-105-228, col=0x132, expectedValue=null]]
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockTransaction.writeBlindLockClaim(ConsistentKeyLockTransaction.java:250)
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore.acquireLock(ConsistentKeyLockStore.java:137)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireVertexIndexLock(BackendTransaction.java:153)
at com.thinkaurelius.titan.graphdb.database.IndexSerializer.lockKeyedProperty(IndexSerializer.java:111)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:255)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:203)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:61)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.save(StandardTitanGraph.java:277)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:839)
at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:41)
at com.tinkerpop.blueprints.TransactionalGraph$commit.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at groovysh_evaluate.run(groovysh_evaluate:56)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:67)
at org.codehaus.groovy.tools.shell.Interpreter$evaluate.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:65)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:78)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.main(Console.java:104)
13/07/02 18:22:47 INFO util.BackendOperation: Temporary storage exception during backend operation [PersistingTransaction]. Attempting incremental retry
com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Lock failed: exceeded max timeouts [LockClaim [backer=com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore@25ba59dc, key=0x1-117-117-105-228, col=0x132, expectedValue=null]]
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockTransaction.writeBlindLockClaim(ConsistentKeyLockTransaction.java:250)
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore.acquireLock(ConsistentKeyLockStore.java:137)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireVertexIndexLock(BackendTransaction.java:153)
at com.thinkaurelius.titan.graphdb.database.IndexSerializer.lockKeyedProperty(IndexSerializer.java:111)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:255)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:203)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:61)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.save(StandardTitanGraph.java:277)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:839)
at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:41)
at com.tinkerpop.blueprints.TransactionalGraph$commit.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at groovysh_evaluate.run(groovysh_evaluate:56)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:67)
at org.codehaus.groovy.tools.shell.Interpreter$evaluate.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:65)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:78)
at com.thinkaurelius.titan.tinkerpop.gremlin.Console.main(Console.java:104)
Could not commit transaction due to exception during persistence

[Matthias Broecheler]

Could this be related to this issue?

https://github.com/thinkaurelius/titan/issues/302

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Matthias Broecheler
http://www.matthiasb.com

[Abhinav]

Matthias,

I don't think it was the timestamp mismatch issue you mentioned. After a few more hours of experiments, I was able to successfully get it to create the indexes. 

I think it was a latency issue. 

Our cassandra servers are sitting in a data center in Germany. 

The Gremlin shell was running locally on my laptop in Toronto, Canada when the aforementioned locking exceptions occurred. 

When I ssh'd into a nearby server in Amsterdam and ran the same commands from a Gremlin shell, everything went smoothly.

I was able to reproduce this issue numerous times, which led me to think it was a latency issue. Curiously enough, adding/removing a vertex/edge was working from my laptop's Gremlin shell.

Am I correct in assuming that latency was the culprit or is this something totally different?

Thanks,

--Abhi

[Dan LaRocque]

Hi Abhi,

I think you're right that this is separate from #302.  It seems like the max timeouts exception happens before #302 could potentially kick in.

What was the round-trip write latency from your laptop in Toronto to the German datacenter's Cassandra cluster?  The exception you cited can happen when Titan's lock-related Cassandra write takes longer than LOCK_WAIT_MS (default 100 ms) more than LOCK_RETRY_COUNT times (default 3).  The write has to succeed slowly each time; a failed write would take a different path and wouldn't log that particular exception.

The locking code is bypassed for transactions that take no locks.  That may be why adding and removing vertices and (presumably non-unique) edges worked fine.

There's nothing special or particularly deep about those default values mentioned above.  They're configurable.  However, increasing LOCK_WAIT_MS can slow throughput of small transactions that take locks.  Commits on transactions taking locks can wait up to LOCK_WAIT_MS before committing as they verify their locks.  If you don't use unique edges/properties, then your transactions generally won't lock, and there will be no wait regardless of the LOCK_WAIT_MS setting.

thanks,
Dan

[Abhinav]

Hi Dan,

What you said makes a lot of sense. Thanks for the in-depth explanation.

Cheers,

--Abhi

[Billy Newport]

I think I'm having the same problem. I have the titan client on the same box as a cassandra node in NY. I have another cass node in NY, one in London and one in Hong Kong. I've configured the cassandra-topology stuff accordingly. The two in NY are different RACs.

I think netflix is picking the HK node and all I get is timeouts unless I change the timeouts but clearly, if my client is on the same box as a NY node, thats the one I want to talk with. My consistency level is ONE/ONE. Is there a way to force titan to connect to a cassandra node 'close' to the client by specifying DC_NY:RAC1 or some similar string so it can do something cleverly than picking the furthest node :)

I'm using 0.3.2 and 1.2.10
Thanks
Billy

[Billy Newport]

I patched the cassandra plugin to allow the setLocalDataCenter method to be specified in the AstanyxStoreManager when creating a getContextBuilder and this allows the initial index creates to succeed. However, it's running brutally slow with frequency max timeout exceeded because I suspect titan is using a QUORUM setting somewhere rather than what I've specified in the storage properties as ONE/ONE.

Does titan use quorum itself for id locks and so forth? Should they really be LOCAL_QUORUM for example instead?

[Matthias Broecheler]

Hi Billy,

see my previous message on this. Since Titan needs to ensure that vertex/edge ids are globally unique, we require global quorum at the moment. We are considering alternative implementations based on zookeeper or using local_quorum with a configurable prefix_id. The latter is a little risky because it requires the user to configure that prefix correctly.

Best,

Matthias

On Wed, Sep 25, 2013 at 12:39 PM, Billy Newport <bi...@billynewport.com> wrote:

I patched the cassandra plugin to allow the setLocalDataCenter method to be specified in the AstanyxStoreManager when creating a getContextBuilder and this allows the initial index creates to succeed. However, it's running brutally slow with frequency max timeout exceeded because I suspect titan is using a QUORUM setting somewhere rather than what I've specified in the storage properties as ONE/ONE.

Does titan use quorum itself for id locks and so forth? Should they really be LOCAL_QUORUM for example instead?

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dev Kumkar]

Hi Matthias,

Am using Titan 1.0.0 and Cassandra as the storage back end.

If there are 2 nodes running in cluster and there is high latency  like 250 ms between these instances then whenever there is write/commit on one node via Titan following exception is being faced:

Caused by: com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Temporary locking failure

ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph - Could not commit transaction [25] due to exception 

com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Temporary locking failure

        at com.thinkaurelius.titan.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:295) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:89) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:40) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:240) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:554) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:683) [titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263) [titan-core-1.0.0.jar:?]

        at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94) [gremlin-core-3.0.1-incubating.jar:3.0.1-incubating]

At the Cassandra layer, from CQL with quorum data is being inserted without any issues in database, so this could be issue with some configuration setting for lock wait at Titan level.

With quorum things are not working whenever there is latency between 2 instances.

Its really strange that even 100 ms latency is facing with this issues, am I missing something really basic?