(Titan) Could not commit transaction due to exception during persistence

Allan Johns

unread,

Feb 20, 2013, 12:32:36 AM2/20/13

to gremli...@googlegroups.com, aureliu...@googlegroups.com

Hi,

I'm getting the following error when executing a particular groovy script:

javax.script.ScriptException: com.thinkaurelius.titan.core.TitanException: Could not commit transaction due to exception during persistence

I'm running:
rexster-0.3.0-SNAPSHOT @ 20th Feb 2013 (via REST)
titan-0.2.0 RELEASE, with embedded Cassandra backend

Here is the script being executed:

def create_or_append_vertex(element_type, uri, fields) {
    def createOrAppendVertex = {
        verts = g.V("_uri", uri).toList();
        nverts = verts.size();
        new_v = false;

        if(nverts == 0) {
            v = g.addVertex();
            v.setProperty("_uri", uri);
            v.setProperty("element_type", element_type);
            new_v = true;
        }
        else if(nverts == 1) {
            v = verts[0];
        }
        else {
            s = String.format("%s lookup on uri '%s' matched %d elements: %s",
                element_type, uri, nverts, verts.toString());
            throw new Exception(s);
        }

        for(entry in fields.entrySet()) {
            if(entry.value == null) continue;
            if(v.getProperty(entry.key) == null) {
                v.setProperty(entry.key, entry.value);
            }
        }

        return [v.getId(),new_v];
    }

    def transaction = { final Closure closure ->
        try {
            results = closure();
            g.stopTransaction(TransactionalGraph.Conclusion.SUCCESS);
            results[0] = g.v(results[0])
            return results;
        } catch (e) {
            g.stopTransaction(TransactionalGraph.Conclusion.FAILURE);
            throw e;
        }
    }

    return transaction(createOrAppendVertex);
}

I have a test running which executes this script approximately 700 times, divided over 32 concurrent processes, and approx 600 of those times the vertex already exists in the database, and no new fields are being added. Under this scenario, between approx 5-30% of create_or_append_vertex() invocations are failing with this exception.

This is a huge problem for us at the moment, any help greatly appreciated.

Thanks
Allan

Allan Johns

unread,

Feb 20, 2013, 1:26:06 AM2/20/13

to gremli...@googlegroups.com, aureliu...@googlegroups.com

I've been able to dig a little more and introspect the exception being raised:

EXCEPTION TYPE:
class com.thinkaurelius.titan.core.TitanException

CAUSE:
com.thinkaurelius.titan.diskstorage.locking.PermanentLockingException: Lock could not be acquired because it is held by a local transaction [LockClaim [backer=com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore@68fe414e, key=0x066a6f623a6d65, col=0x400000000000008a, expectedValue=null]]

STACKTRACE:
com.thinkaurelius.titan.graphdb.transaction.StandardPersistTitanTx.commit(StandardPersistTitanTx.java:181)
com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsTransaction.stopTransaction(TitanBlueprintsTransaction.java:33)
com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.stopTransaction(TitanBlueprintsGraph.java:37)
com.tinkerpop.blueprints.TransactionalGraph$stopTransaction$0.call(Unknown Source)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
Script7$_run_closure2.doCall(Script7.groovy:34)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
Script7.run(Script7.groovy:49)
com.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:232)
com.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:103)
javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
com.tinkerpop.rexster.extension.GremlinExtension.tryExecuteGremlinScript(GremlinExtension.java:242)
com.tinkerpop.rexster.extension.GremlinExtension.evaluatePostOnGraph(GremlinExtension.java:168)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:321)
com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:229)
com.tinkerpop.rexster.GraphResource.executeGraphExtension(GraphResource.java:269)
com.tinkerpop.rexster.GraphResource.postGraphExtension(GraphResource.java:192)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:147)
org.glassfish.grizzly.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:106)
org.glassfish.grizzly.servlet.ServletHandler.doServletService(ServletHandler.java:252)
org.glassfish.grizzly.servlet.ServletHandler.service(ServletHandler.java:188)
org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
org.glassfish.grizzly.http.server.HttpHandlerChain.service(HttpHandlerChain.java:196)
org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:175)
org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:815)
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
org.glassfish.grizzly.strategies.LeaderFollowerNIOStrategy.executeIoEvent(LeaderFollowerNIOStrategy.java:102)
org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:88)
org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:398)
org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:368)
org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:334)
org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:264)
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)
java.lang.Thread.run(Thread.java:722)

thx
A

Matthias Broecheler

unread,

Feb 20, 2013, 3:58:11 AM2/20/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hey Allan,

that just means that another transaction is holding a lock which conflicts with the current one. This means the transaction cannot move forward because it has been operating on invalid (stale) data and hence fails. Under the high concurrency load that you are describing this is to be expected and hence its quite normal for a database to fail with a locking exception.

To get around this you can:

1) retry the transaction

2) try to work around the source of contention: unique values, or other aspects that require locks.

HTH,

Matthias

Allan

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Matthias Broecheler
http://www.matthiasb.com

Allan Johns

unread,

Feb 20, 2013, 6:17:33 PM2/20/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hey Matthias,

Thanks for that, I'll be implementing both of your suggestions.

Just another related question - say I have a unique, indexed key 'foo' that I want to set on a vertex. Consider the following query:

g.v(101).setProperty('foo','bah')

Given that 'foo' is unique and indexed, should I ever expect this query to fail due to lock contention, even rarely?

cheers
A

Allan Johns

unread,

Feb 20, 2013, 9:38:37 PM2/20/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hi,

So I've revised my groovy script, with mixed results. Here it is:

def create_unique_vertex(element_type, uri, fields) {
    retries = 8;
    delay = 32;
    v = null;
    is_new = true;

    while(!v && (retries > 0)) {
        retries -= 1;

        try {
            r = g.V("_uri", uri);
            if(r) {
                v = r.next();
                is_new = false;
            } else {

v = g.addVertex();
v.setProperty("_uri", uri);

                is_new = true;
            }
            g.stopTransaction(TransactionalGraph.Conclusion.SUCCESS);
            v = g.getVertex(v);
        } catch(e) {
            g.stopTransaction(TransactionalGraph.Conclusion.FAILURE);
            if(retries) {
                // sleep and try again. Total possible cumulative sleep time is ~8 secs
                i = (int)(Math.random()*(delay/4));
                Thread.sleep(delay+i);
                delay += delay;
            } else {
                throw e;
            }
        }
    }

    //if(v) {
    //    v.setProperty("element_type", element_type);
    //    for(entry in fields.entrySet()) {
    //        if(entry.value == null)
    //            continue;
    //        if(v.getProperty(entry.key) == null)
    //            v.setProperty(entry.key, entry.value);
    //    }
    //}

    return [v, is_new];
}

If the commented-out section is left out, then everything works great, no failures any more. However, introducing the commented section back in causes the "Could not commit transaction due to exception during persistence" exception once more.

The "_uri" key is the only unique key (set using g.makeType().unique() on db initialisation). How should I best go about setting the other fields while avoiding lock contention? Do I have to force each setProperty() call into its own transaction, retrying on failure etc? Which brings me back to my previous post - is it *ever* guaranteed that you can use setProperty() and not have it fail?

I've seen other people's code where they are quite happily traversing through the graph, using setProperty along the way, eg:

g.V('key','lots').sideEffect{it.setProperty('foo','bah')}

If the number of vertices traversed is high, isn't this code highly susceptible to failure, or am I missing something? Is this one query treated as a single transaction?

Sorry for all the questions, I feel like I'm missing something here.

thx
A

Matthias Broecheler

unread,

Feb 21, 2013, 4:12:56 AM2/21/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hi Allan,

the part that is causing the failure is that you defined your property _uri to be unique. So, when you do a

v.setProperty("_uri","blah"), then Titan will ensure that v is the only vertex that has "blah" as its uri. However, there could be multiple transactions concurrently, so the only way to ensure this is for Titan to acquire a lock. It does so toward the end of the transaction (when committed) so that locks aren't held any longer than needed (otherwise throughput would go down a lot). In the meantime, another transaction might have set the same value "blah" on another vertex and it might acquire the lock sooner, which inevitably causes the former transaction to fail since there is nothing it can do to ensure uniqueness.

When other people use setProperty it is typically not with a unique property. Unique properties are mostly set when a vertex is first created (like a unique username) and typically not during a traversal. That's why the issue does not arise.

You should analyze your code to determine why multiple vertices are trying to set the same uri and see if there is something at the application level that can be done to remove that bottle neck. There isn't much the database can do about that.

HTH,

Matthias

Allan Johns

unread,

Feb 21, 2013, 5:15:15 PM2/21/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hi Matthias,

Sorry perhaps I wasn't clear. I completely get the situation you're describing, and I'm successfully dealing with that now, by minimising the code setting "_uri" and doing so in a smaller transaction, combined with retries on failure.

The surprising part was the second part of my code failing - the loop which does a setProperty() on other properties, which are *not* unique. A typical type definition of these keys is:

g.makeType().name("user").dataType(String.class).functional().indexed().makePropertyKey();

Reading into the docs further just now, I think I see the problem - I've defined locking functional types. In my case blind overwrites are acceptable, so I'm going to change my types to functional(false) and see if this solves the problem.

thx
A

Allan Johns

unread,

Feb 21, 2013, 7:42:58 PM2/21/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hi again,

I've entirely removed the setProperties loop and put that into its own separate script, which looks like this:

def update_vertex(_id, fields, overwrite) {
    v = g.v(_id);
    for(entry in fields.entrySet()) {
        if(overwrite || (v.getProperty(entry.key) == null))
            v.setProperty(entry.key, entry.value);
    }
    return v;
}

I've also changed all my property types to functional(false) so that locking contention should not occur. A typical type now looks like so:

g.makeType().name("user").dataType(String.class).functional(false).indexed().makePropertyKey();

However, I am still experiencing occasional "Could not commit transaction due to exception during persistence" exceptions in this function. Why would this be... what am I doing that could still be causing lock contention?

Thanks again,
A

Allan Johns

unread,

Feb 21, 2013, 8:23:04 PM2/21/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Ok I've changed the function to the following, and I no longer get the exception, but I'm not sure why:

def update_vertex(_id, fields, overwrite) {
    v = g.v(_id);
    for(entry in fields.entrySet()) {
        if(overwrite || (v.getProperty(entry.key) == null))
            v.setProperty(entry.key, entry.value);
    }

g.stopTransaction(TransactionalGraph.Conclusion.SUCCESS);
return g.getVertex(v);
}

Thx
A

Jonathan Haddad

unread,

Feb 22, 2013, 12:07:09 PM2/22/13

to gremli...@googlegroups.com, aureliu...@googlegroups.com

I'm not sure if this is relevant, but I've only got a second so i can't read the whole thread, but you can't set a unique vertex value to one that's already used, even if it's the same value it already had. (if it's not related to the thread disregard - i just wanted to put it out there on the off chance it's related)

Jon

Jonathan Haddad

unread,

Feb 22, 2013, 12:08:46 PM2/22/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Trying to say

x.unique_id = "blah"

// do stuff, later

x.unique_id = "blah" // fails, if i remember correctly

Matthias Broecheler

unread,

Feb 22, 2013, 12:35:09 PM2/22/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hey Allan,

gotcha. Yes, if you do functional() in the property definition then it will also acquire a lock to prohibit other (concurrent) threads to overwrite a value that you might be writing at that time to ensure consistency. By setting it to false, it won't do that anymore. I guess the latest change is committing the transaction more frequently which means there is less data to commit at once which generally reduces failure rate.

That seems as expected. Anything I am missing?

Cheers,

Matthias

Allan Johns

unread,

Feb 24, 2013, 5:46:46 PM2/24/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Yeah sorry for the long thread :) Here's the summary:
I had a groovy function that looked like this:

def update_vertex(_id, fields, overwrite) {
    v = g.v(_id);
    for(entry in fields.entrySet()) {
        if(overwrite || (v.getProperty(entry.key) == null))
            v.setProperty(entry.key, entry.value);
    }
    return v;
}

This was failing occasionally with the "exception during persistence". However, none of the properties being set are unique, and all are of type functional(false). The function has since been changed to this:

def update_vertex(_id, fields, overwrite) {
try {

        v = g.v(_id);
        for(entry in fields.entrySet()) {
            if(overwrite || (v.getProperty(entry.key) == null))
                v.setProperty(entry.key, entry.value);
        }

        g.stopTransaction(TransactionalGraph.Conclusion.SUCCESS);
        return g.getVertex(v);

    } catch(e) {
        g.stopTransaction(TransactionalGraph.Conclusion.FAILURE);
        throw e;
    }
}

This new version works always, and I don't know why. I would have thought that Rexster would be effectively doing this for me anyway (a single query through REST is a single transaction iirc). But that aside, I don't know why the persistence error was occurring here at all, given that I'm not touching unique or locking functional types.

Thanks!
A

Matthias Broecheler

unread,

Apr 1, 2013, 3:58:07 PM4/1/13

to aureliu...@googlegroups.com, gremli...@googlegroups.com

Hey Andrew,

if you don't define any locks, then I believe the error occurs because you don't define your types prior to multiple threads trying to create the same types.

if you have a code snippet like this:

v.addEdge("somelabel",u)

where v,u are vertices and you run this in multiple threads without first defining "somelabel" then both threads will attempt to do so and one failing because of lock congestions.

Bottom line: In multi-threaded applications, always define your labels and keys up front (in a single thread).

HTH,

Matthias

On Fri, Mar 29, 2013 at 1:47 PM, AN <and...@bodydatasystems.com> wrote:

Hello,

I am getting both the error referenced below (Lock could not be acquired...) as well as "Updated state: lock acquired but value has changed since read..."

I do not have any functional or unique properties but do have the following:

graph.makeType().name('prop').dataType(String.class).indexed().makePropertyKey()

for several different properties. What else would be causing locking?

I am attempting a multithreaded load of a Titan+Cassandra with a single, local instance.

Thanks,
Andrew

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Reply all

Reply to author

Forward