[TinkerPop3] Gremlin-server driver and Transactions

Dmill

unread,

Aug 27, 2014, 12:02:39 PM8/27/14

to gremli...@googlegroups.com

Hey guys,

I'm finishing up the rexpro-php (driver) support of TP3/gremlin-server. I'm basically missing Transaction support. Once that is all done I will refactor my code and let you guys know. I might separate it into another repo, still debating.

In the meantime I'm trying to figure out how transactions work in TP3. Does the following look correct? I've been playing around with various methods available but can't seem to get something functional.

n.tx().open(); 
n.addVertex();
n.tx().commit(); // or .rollback()

Here's my configuration file, I have the neo4j jar in place:

host: localhost
port: 8182
threadPoolWorker: 1
gremlinPool: 8
scriptEvaluationTimeout: 30000
serializedResponseTimeout: 30000
channelizer: com.tinkerpop.gremlin.server.channel.WebSocketChannelizer
graphs: {
  g: config/tinkergraph-empty.properties,
  n: config/neo4j-empty.properties}
use:
  - [com.tinkerpop, neo4j-gremlin, "3.0.0.M1"]
  - [org.apache.commons, commons-math3, "3.2"]
scriptEngines: {
  gremlin-groovy: {
    imports: [java.lang.Math, org.apache.commons.math3.util.FastMath],
    staticImports: [java.lang.Math.PI],
    scripts: [scripts/generate-classic.groovy]},
  nashorn: {
      imports: [java.lang.Math, org.apache.commons.math3.util.FastMath],
      staticImports: [java.lang.Math.PI]}}
serializers:
  - { className: com.tinkerpop.gremlin.driver.ser.KryoMessageSerializerV1d0 }
  - { className: com.tinkerpop.gremlin.driver.ser.KryoMessageSerializerV1d0, config: { serializeResultToString: true }}
  - { className: com.tinkerpop.gremlin.driver.ser.JsonMessageSerializerGremlinV1d0 }
  - { className: com.tinkerpop.gremlin.driver.ser.JsonMessageSerializerV1d0 }
processors:
  - { className: com.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
metrics: {
  consoleReporter: {enabled: true, interval: 180000},
  csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
  jmxReporter: {enabled: true},
  slf4jReporter: {enabled: true, interval: 180000},
  gangliaReporter: {enabled: true, interval: 180000, addressingMode: MULTICAST},
  graphiteReporter: {enabled: true, interval: 180000}}
threadPoolBoss: 1
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferHighWaterMark: 32768
writeBufferHighWaterMark: 65536
ssl: {
  enabled: false}

Thanks guys

Stephen Mallette

unread,

Aug 27, 2014, 12:34:10 PM8/27/14

to gremli...@googlegroups.com

You could be asking two questions here so I'll answer both. You might be asking about how transactions work in the Gremlin Structure API in which case, you basically have it correct with:

n.tx().open();

n.addVertex();

n.tx().commit()

Of course, transactions are implicit by default (as they were with TP2) so you technically don't need to start transactions (unless you configure it otherwise):

http://www.tinkerpop.com/docs/3.0.0.M1/#transactions

You might also be asking about how transactions are handled by Gremlin Server. By default, Gremlin Server works in "sessionless" mode. In that mode, it will auto-commit, thus each request is encapsulated in a transaction so each request must be independent of the next. In recent weeks/months, I realized that this wasn't good for non-jvm languages trying to bind to Gremlin Server. You need transactions that exceed request boundaries. It's sort of a different use case. Luckily, Gremlin Server is pretty flexible and so i was able to add that in by creating a new OpProcessor that supports sessions (looks like you have that configured in your yaml file - so that's good). If you want to support an in-session mode with your PHP client, then you need to supply a "session-id" in your RequestMessage "args". Every request processed with that "session-id" will execute with the same bindings as the previous request and in the same thread on the server which means your transaction is completely managed by the client. This isn't documented yet unfortunately - Bob mentioned it to me and I forgot.

Under this model, i've seen some pretty decent bulk loading speeds that have exceeded Rexster's capabilities - I'm hoping we can get it fast enough so that non-jvm languages can actually bulk load in their language of choice. I doubt it will ever be as fast as raw Neo4jGraph, but perhaps it will be good enough that it becomes an option for people just getting started.

HTH,

Stephen

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/db197b50-cdb7-4c5c-b2b3-e9a5f148fe3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dmill

unread,

Aug 27, 2014, 12:55:08 PM8/27/14

to gremli...@googlegroups.com

Great info.

I've been using the session opProcessor with the argument 'session' instead of 'session-id' so far. I based myself off of com.tinkerpop.gremlin.driver.Tokens (http://www.tinkerpop.com/javadocs/3.0.0.M1/constant-values.html) have I been using the wrong arg or was that just a by memory "typo"?

So if I get this right, any exchanges made over a 'session' are automatically encapsulated within a transaction. How is this transaction either committed or rolled back?

Also I've forked TP3 and will probably make a pull request with some clarification to the documentation regarding making drivers (with things I struggled to figure out and some of the extra info I got from threads right and left, including this one).

Stephen Mallette

unread,

Aug 27, 2014, 12:59:14 PM8/27/14

to gremli...@googlegroups.com

i didn't mean "session-id" literally - what's in tokens is right.

> So if I get this right, any exchanges made over a 'session' are automatically encapsulated within a transaction.

no - that's only if do NOT include a "session" identifier in the request message. it is up to the client to manage the commit() when the "session" identifier is supplied.

> make a pull request with some clarification to the documentation

that would be great....thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/736f97a8-1d17-44bf-8f79-a248bbbb8d9f%40googlegroups.com.

Dmill

unread,

Aug 27, 2014, 1:43:53 PM8/27/14

to gremli...@googlegroups.com

Ok thanks for clearing that out.

Is it safe to assume that request script bindings and variables set in the previous request of a same session aren't carried over to the future requests? Or am I doing something wrong? (this is more of a personal question rather than a driver specific one.)

I've tried the following:

Request 1 : {"requestId":"16161de5-af6f-4d3c-a801-c7b5a7223e82","processor":"session","op":"eval","args":{"session":"4848a217-bf9e-4f3c-a695-5550571c27ed","gremlin":"g.v(CUSTO_BINDING)","bindings":{"CUSTO_BINDING":2},"language":"gremlin-groovy"}}

Response : works like a charm, I get my vertex or not depending on the graph.

Request 2 : {"requestId":"2e2ab225-c6b7-4ffd-bc88-1746a8302505","processor":"","op":"eval","args":{"session":"4848a217-bf9e-4f3c-a695-5550571c27ed","gremlin":"g.v(CUSTO_BINDING)","bindings":null,"language":"gremlin-groovy"}}

Response : Error (597) - java.lang.RuntimeException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: CUSTO_BINDING for class: Script2

as well as :

Request 1 : {"requestId":"65a348ae-e5d8-4984-8177-9b658a797266","processor":"session","op":"eval","args":{"session":"6c07fa1b-e21a-404f-af8e-1622ac2296ce","gremlin":"cal = 5+5","bindings":null,"language":"gremlin-groovy"}}

Response : works perfectly again. I get 10 back

Request 2 : {"requestId":"9744d80f-30ee-4dbd-b93e-f446621d4a81","processor":"","op":"eval","args":{"session":"6c07fa1b-e21a-404f-af8e-1622ac2296ce","gremlin":"cal","bindings":null,"language":"gremlin-groovy"}}

Response : Error (597) - java.lang.RuntimeException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: cal for class: Script3 Possible solutions: class

And on the transaction front, I'm having some issues as well. I'm running these against a neo4j Graph (n from the config earlier).

Request 1 : {"requestId":"9737ed29-1959-4894-9d1a-f3745927e9db","processor":"session","op":"eval","args":{"session":"933262e0-44e3-4207-ac7f-304f7550a88c","gremlin":"n.V.count()","bindings":null,"language":"gremlin-groovy"}}

Response : 12

Request 2 : {"requestId":"fafc2bd7-9b25-487c-bed1-3ff3cfbdfabc","processor":"","op":"eval","args":{"session":"933262e0-44e3-4207-ac7f-304f7550a88c","gremlin":"n.tx().open()","bindings":null,"language":"gremlin-groovy"}}

Response : null

Request 3 : {"requestId":"e0333eba-0406-4fdf-ac5e-99fa781eef97","processor":"","op":"eval","args":{"session":"933262e0-44e3-4207-ac7f-304f7550a88c","gremlin":"n.addVertex()","bindings":null,"language":"gremlin-groovy"}}

Response : Get the vertex back with id 39981

Request 4 : {"requestId":"cad5824a-0f2d-4d5c-a3a8-cfb4107aaa0f","processor":"","op":"eval","args":{"session":"933262e0-44e3-4207-ac7f-304f7550a88c","gremlin":"n.tx().rollback()","bindings":null,"language":"gremlin-groovy"}}

Response : null

Request 5 : {"requestId":"2ce44850-bbee-4dd8-b3a2-1e4cd0fb91d5","processor":"","op":"eval","args":{"session":"933262e0-44e3-4207-ac7f-304f7550a88c","gremlin":"n.V.count()","bindings":null,"language":"gremlin-groovy"}}

Response : 13

Any help in figuring out what I've got wrong would be great :)

Thanks again!

Dmill

unread,

Aug 27, 2014, 1:55:48 PM8/27/14

to gremli...@googlegroups.com

I'll spare you some time I had an error in my code (that's why we set these unit tests up eh ;p) so the above don't apply.

On the other hand I'm getting a message from gremlin server about a transaction already existing. So it seems like it opens one on creation. I'll toy with it a bit and see if I need anymore input.

Thanks a lot though!

Stephen Mallette

unread,

Aug 27, 2014, 3:45:23 PM8/27/14

to gremli...@googlegroups.com

Glad you solved your problem - i was going to point you to this test which confirms session behavior with bindings:

https://github.com/tinkerpop/tinkerpop3/blob/2bc704a6354ef0d55112e373e5f6b22720429af6/gremlin-server/src/test/java/com/tinkerpop/gremlin/server/GremlinDriverIntegrateTest.java#L297

You have to configure Neo4jGraph to use "manual" transactions. For purposes of your work and assumptions you may make with your client, I suggest you assume that users will be configured with auto-transactions which is the default.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/f7ea2c8d-2141-4e2e-bd7e-05127842fa23%40googlegroups.com.

Dmill

unread,

Aug 27, 2014, 4:22:15 PM8/27/14

to gremli...@googlegroups.com

Thanks for the link. Yeah I'll go with automatic transactions as default.

My tests on transactions and sessions are passing marvelously. I'm having a bit of an issue regarding neo4j and n.addVertex(). I'll systematically get an error from gremlin server. Even though the vertex is added. It seems like the serializer (json) can't serialize the output from addVertex (if I do: n.addVertex();5 it will succeed and send 5 back):

Error during serialization: (was java.util.NoSuchElementException) (through reference chain: java.util.HashMap["result"]->java.util.ArrayList[0])

I can't seem to make this go away. Any idea what it could be?

Dmill

unread,

Aug 27, 2014, 4:31:19 PM8/27/14

to gremli...@googlegroups.com

Oh and while I'm at it I can't find anything on the subject of authentification with gremlin-server. If you could point me in the right direction I'd be grateful. (src files etc.. anything really)

Thanks again.

Stephen Mallette

unread,

Aug 28, 2014, 6:12:53 AM8/28/14

to gremli...@googlegroups.com

> I'll systematically get an error from gremlin server. Even though the vertex is added. It seems like the serializer (json) can't serialize the output from addVertex (if I do: n.addVertex();5 it will succeed and send 5 back)

It would seem that "serialization" is the problem. Do you have any more to the exception? is the little bit you pasted what you get in the return message?

> Oh and while I'm at it I can't find anything on the subject of authentification with gremlin-server. If you could point me in the right direction I'd be grateful. (src files etc.. anything really)

At the moment, there is no "authentication" on gremlin server. Please add an issue in github and i'll think about it.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/96ebbc89-87e1-42fe-93e0-f5e813575a7f%40googlegroups.com.

Dmill

unread,

Aug 28, 2014, 7:12:19 AM8/28/14

to gremli...@googlegroups.com

The little bit I posted was the full message back but I have some additional information from the server console.

I'll try and provide more information from my testing.

Starting gremlin-server with my configuration (from a few posts back) and a virgin neo4j graph.

- All requests are made against the session opProcessor. There's no difference if they're in the same session or not. There are however NO issues in the standard opProcessor.

- All requests are made against a Neo4j graph. There are no issues with tinkergraph.

- All requests are made within a binary websocket frame using application/json as the mimeType.

requesting n.V works as expected (returns empty set) but the server output gives the following warning:

[WARN] GremlinExecutor - Could not initialize gremlin-groovy ScriptEngine with scripts/generate-classic.groovy as script could not be evaluated - javax.script.ScriptException: java.lang.IllegalArgumentException: Vertex with id already exists: 1

requesting n.addVertex() returns an error : Error during serialization: (was java.util.NoSuchElementException) (through reference chain: java.util.HashMap["result"]->java.util.ArrayList[0])

It's worth noting that the vertex is created. And the server console output is :

[INFO] Session - New session established for 6acefc06-b3f7-4e80-a8f4-6c9a65b9400d

[INFO] ScriptEngines - Loaded nashorn ScriptEngine

[INFO] ScriptEngines - Loaded gremlin-groovy ScriptEngine

[INFO] GremlinExecutor - Getting dependencies for [[com.tinkerpop, neo4j-gremlin, 3.0.0.M1]]

[INFO] GremlinExecutor - Getting dependencies for [[org.apache.commons, commons-math3, 3.2]]

[WARN] GremlinExecutor - Could not initialize gremlin-groovy ScriptEngine with scripts/generate-classic.groovy as script could not be evaluated - javax.script.ScriptException: java.lang.IllegalArgumentException: Vertex with id already exists: 1

[WARN] AbstractJsonMessageSerializerV1d0 - Response [ResponseMessage{requestId=f6a128f9-f534-4445-a9d4-ed158c6f795e, code=SUCCESS, result=[v[1]], resultType=COLLECTION}] could not be serialized by com.tinkerpop.gremlin.driver.ser.AbstractJsonMessageSerializerV1d0.

[WARN] WsGremlinResponseEncoder - The result [[v[1]]] in the request f6a128f9-f534-4445-a9d4-ed158c6f795e could not be serialized and returned.

com.tinkerpop.gremlin.driver.ser.SerializationException: com.fasterxml.jackson.databind.JsonMappingException: (was java.util.NoSuchElementException) (through reference chain: java.util.HashMap["result"]->java.util.ArrayList[0])

at com.tinkerpop.gremlin.driver.ser.AbstractJsonMessageSerializerV1d0.serializeResponseAsBinary(AbstractJsonMessageSerializerV1d0.java:59)

at com.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder.encode(WsGremlinResponseEncoder.java:36)

at com.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder.encode(WsGremlinResponseEncoder.java:24)

at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)

at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658)

at io.netty.channel.AbstractChannelHandlerContext.access$2000(AbstractChannelHandlerContext.java:32)

at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:939)

at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:991)

at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:924)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:370)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.util.NoSuchElementException) (through reference chain: java.util.HashMap["result"]->java.util.ArrayList[0])

at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:232)

at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:211)

at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:212)

at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:105)

at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21)

at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183)

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:467)

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:388)

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:27)

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:114)

at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:2811)

at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:2292)

at com.tinkerpop.gremlin.driver.ser.AbstractJsonMessageSerializerV1d0.serializeResponseAsBinary(AbstractJsonMessageSerializerV1d0.java:50)

... 13 more

Caused by: java.util.NoSuchElementException

at java.util.ArrayList$Itr.next(ArrayList.java:854)

at com.tinkerpop.gremlin.neo4j.structure.Neo4jElement.label(Neo4jElement.java:43)

at com.tinkerpop.gremlin.neo4j.structure.Neo4jVertex.label(Neo4jVertex.java:24)

at com.tinkerpop.gremlin.structure.io.graphson.GraphSONModule$VertexJacksonSerializer.ser(GraphSONModule.java:90)

at com.tinkerpop.gremlin.structure.io.graphson.GraphSONModule$VertexJacksonSerializer.serialize(GraphSONModule.java:76)

at com.tinkerpop.gremlin.structure.io.graphson.GraphSONModule$VertexJacksonSerializer.serialize(GraphSONModule.java:67)

at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100)

... 22 more

After this error, any subsequent n.V scripts I run throw an error. Here's the console output for the first run:

[INFO] Session - New session established for 60661585-ff1b-430a-9037-9b90b13c68ad

[INFO] ScriptEngines - Loaded nashorn ScriptEngine

[INFO] ScriptEngines - Loaded gremlin-groovy ScriptEngine

[INFO] GremlinExecutor - Getting dependencies for [[com.tinkerpop, neo4j-gremlin, 3.0.0.M1]]

[INFO] GremlinExecutor - Getting dependencies for [[org.apache.commons, commons-math3, 3.2]]

[WARN] GremlinExecutor - Could not initialize gremlin-groovy ScriptEngine with scripts/generate-classic.groovy as script could not be evaluated - javax.script.ScriptException: java.lang.IllegalArgumentException: Vertex with id already exists: 1

[WARN] AbstractJsonMessageSerializerV1d0 - Response [ResponseMessage{requestId=e55fc8c4-49c3-4f4e-a293-10065712cec5, code=SUCCESS, result=[v[2]], resultType=COLLECTION}] could not be serialized by com.tinkerpop.gremlin.driver.ser.AbstractJsonMessageSerializerV1d0.

[WARN] WsGremlinResponseEncoder - The result [[v[2]]] in the request e55fc8c4-49c3-4f4e-a293-10065712cec5 could not be serialized and returned.