Temporary locking failure

[Dev Kumkar]

Am using Titan 1.0.0 and Cassandra as the storage back end.

If there are 2 nodes running in cluster and there is high latency  like 250 ms between these instances then whenever there is write/commit on one node via Titan following exception is being faced:

Caused by: com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Temporary locking failure

ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph - Could not commit transaction [25] due to exception

com.thinkaurelius.titan.diskstorage.locking.TemporaryLockingException: Temporary locking failure

        at com.thinkaurelius.titan.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:295) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:89) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:40) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:240) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:554) ~[titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:683) [titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:?]

        at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263) [titan-core-1.0.0.jar:?]

        at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94) [gremlin-core-3.0.1-incubating.jar:3.0.1-incubating]

At the Cassandra layer, from CQL with quorum data is being inserted without any issues in database, so this could be issue with some configuration setting for lock wait at Titan level.

Its really strange that even 100 ms latency is facing with this issues, am I missing something really basic?

[Florian Hockmann]

Titan has to use locks when you have uniqueness constraints in your schema. This exception is telling you that a transaction could not acquire a lock because another transaction was executed at the same time that had the lock. You can either simply retry transactions that fail because of this TemporaryLockingException or remove all uniqueness constraints from your schema so locks aren't used any more. Uniqueness constraints are indexes defined as unique or edge labels with a multiplicity other than MULTI. You would then of course have to ensure uniqueness some other way in your application when you need it for your data model. One way to do that is for example with a separate database like Redis just for this purpose.

In case you didn't know, Titan isn't maintained any more since 2016. You can instead use JanusGraph which is a fork of Titan and which is under active development. JanusGraph also really supports CQL, whereas you can only use Thrift for Cassandra with Titan which is already deprecated.

[Dev Kumkar]

Thanks Florian for the details. Know about JanusGraph and I believe the unique constraints would continue as design requirement to make sure data consistency across multiple instances.

Regarding my issue, actually since there is network latency issue between nodes, the suspect is time to acquire lock resulting into issues.

If there is no latency then for the same data there is no issue for writes and commits.

 

In addition to lock exception, also in the exception stack trace can see "Lock retry count exceeded". 

Did try changing titan configuration settings like storage.lock.wait-time, storage.lock.retries but they didn't work and further lock exceptions were reported with following stack trace.

Tried these settings in titan-cassandra.properties and verified from gremlin, any pointers here to make these settings work?

Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Lock write retry count exceeded

              at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:325) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:109) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:290) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:89) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:40) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:240) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:554) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:683) ~[titan-core-1.0.0.jar:?]

              at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) ~[titan-core-1.0.0.jar:?]