GLV, bytecode & GraphSON

[Ray Scott]

Hi, 

I've been looking at the Gremlin-Python code base, to get more of a clue regarding the role of bytecode and GraphSON with regard to creating a GLV in Go. Then majority of a Go GLV is actually written, it's just the networking aspect that needs addressing. 

I've written a driver in Go too, and from the Driver Provider docs it appears that the bytecode operation on the traversal processor should accept bytecode which is then websocketed over to the server. Is this correct, or do I have to convert the bytecode into a GraphSON form before sending it? I sent the below bytecode via the websocket and I received a 500 error for my effort.

[[], [V(), propertyMap()]]

java.lang.NullPointerException\n\tat org.apache.tinkerpop.gremlin.structure.io.graphson.TraversalSerializersV2d0$BytecodeJacksonDeserializer.deserialize(TraversalSerializersV2d0.java:263)\n\tat org.apache.tinkerpop.gremlin.structure.io.graphson.TraversalSerializersV2d0$BytecodeJacksonDeserializer.deserialize(TraversalSerializersV2d0.java:250)\n\tat org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypeDeserializer.deserialize(GraphSONTypeDeserializer.java:212)\n\tat org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypeDeserializer.deserializeTypedFromAny(GraphSONTypeDeserializer.java:101)\n\tat org.apache.tinkerpop.shaded.jackson.databind.deser.std.StdDeserializer.deserializeWithType(StdDeserializer.java:120)\n\tat org.apache.tinkerpop.shaded.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:63)\n\tat org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)

What have a got wrong?

Another question I have is related to the alias required parameter for the bytecode operation. The docs indicate that it's not an optional parameter, but if look at the bytecode above, I'm struggling to understand how an alias is involved with that code. Plus the reference docs keep referring to a script when discussing bytecode aliases, but I'm not sending a script, I'm sending bytecode. Confusion.

I've been reading through some of the threads on here about bytecode and I'm unsure about handling responses from Gremlin Server. Will I need to write a GraphSON deserialiser to convert the response into which ever Golang structures I choose to develop? Again I have had a look at the Gremlin-Python code base. 

In general, from what I can gather, it's only sent once a "halt" step is encountered. Is that correct? I'm not too bothered about lambdas at this stage, and my knowledge of this area re:tinkerpop isn't very deep. It's pretty easy to develop code to convert a traversal into bytecode, it's just a bit unclear as to what happens after that.  

Thanks.

[David Brown]

Hi Ray,

I think I can point you in the right direction by guiding you around the gremlin-python source. Then you can use this as an example for your Go GLV. Before messages are sent to the server, there are two phases of serialization that occur, but they both begin with the GraphSONMessageSerializer class [1]. The serializer prepares the request message that will be sent, handling both message metadata (things like request id, processor, ops) as well as calling the GraphSONWriter.toDict method. This method is responsible for serializing the bytecode to GraphSON. The GraphSONWriter, as well as GraphSONReader and individual serializers are found in the io subpackage [2]. Regarding aliases, they are required by the traversal processor when performing bytecode operations, there is a lot more info about this here [3] in the provider documentation. Finally, yes, a traversal is only submitted when one of the special steps is called (toList, toSet, iterate, next, etc.). These steps are implemented here [4].

Hopefully this will help get you going. I (or someone else) can provide a more detailed answer when I have a bit more time on my hands.

1. https://github.com/apache/tinkerpop/blob/master/gremlin-python/src/main/jython/gremlin_python/driver/serializer.py

2. https://github.com/apache/tinkerpop/tree/master/gremlin-python/src/main/jython/gremlin_python/structure/io

3. http://tinkerpop.apache.org/docs/3.3.0/dev/provider/#_traversal_opprocessor

4. https://github.com/apache/tinkerpop/blob/master/gremlin-python/src/main/jython/gremlin_python/process/traversal.py#L51-L102

[Ray Scott]

Hi David,

Thank you for helping to clear some of that up for me. While I understand aliases, (and I'm familiar with docs you point out) I'm still non the wiser as to why I need them. A Google site search of the TinkerPop docs fails to illuminate.

So it looks like I need to develop the GraphSON serialisation code. I may have a few more questions for you as I progress, if you don't mind David.

Thanks.

[Kevin Gallardo]

Hey, 

While I understand aliases, (and I'm familiar with docs you point out) I'm still non the wiser as to why I need them

Aliases in the context of bytecode traversals only allow you to choose which traversal source configured in the gremlin server to use. Example:

I have configured the Gremlin server with 2 different graphs, one's empty (graph), the other one is populated with the "Modern" graph once the gremlin server starts up (graph2). Here's my gremlin-server.yaml:

host: localhost
port: 8182
scriptEvaluationTimeout: 30000
channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
graphs: {
  graph: conf/tinkergraph-empty.properties,
  graph2: conf/tinkergraph-empty.properties}
scriptEngines: {
[.... same as default]

Then in the startup script that will populate the traversal sources I have the following:

// an init script that returns a Map allows explicit setting of global bindings.
def globals = [:]


// defines a sample LifeCycleHook that prints some output to the Gremlin Server console.
// note that the name of the key in the "global" map is unimportant.
globals << [hook : [
  onStartUp: { ctx ->
    ctx.logger.info("Executed once at startup of Gremlin Server.")

    // populate graph2 with the "modern" graph
    org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerFactory.generateModern(graph2)

  },
  onShutDown: { ctx ->
    ctx.logger.info("Executed once at shutdown of Gremlin Server.")
  }
] as LifeCycleHook]


// define two traversal sources in the server
globals << [g1 : graph.traversal()]


globals << [g2 : graph2.traversal()]

So now I can either choose to use "g1" or "g2" in my remote traversal source thanks to aliases:

gremlin> cluster = Cluster.open()
==>localhost/127.0.0.1:8182
gremlin> client = cluster.connect()
==>org.apache.tinkerpop.gremlin.driver.Client$ClusteredClient@5eb97ced
gremlin> clientg1 = client.alias("g1")
==>org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient@57c88764
gremlin> clientg2 = client.alias("g2")
==>org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient@43effd89
gremlin> g = EmptyGraph.INSTANCE.traversal()
==>graphtraversalsource[emptygraph[empty], standard]
gremlin> clientg1.submit(g.V().count()).all().get()
==>result{object=0 class=org.apache.tinkerpop.gremlin.process.remote.traversal.DefaultRemoteTraverser}
gremlin> clientg2.submit(g.V().count()).all().get()
==>result{object=6 class=org.apache.tinkerpop.gremlin.process.remote.traversal.DefaultRemoteTraverser}

Or more simply:

gremlin> cluster = Cluster.open()
==>localhost/127.0.0.1:8182
gremlin> g1 = EmptyGraph.INSTANCE.traversal().withRemote(DriverRemoteConnection.using(cluster, "g1"))
==>graphtraversalsource[emptygraph[empty], standard]
gremlin> g2 = EmptyGraph.INSTANCE.traversal().withRemote(DriverRemoteConnection.using(cluster, "g2"))
==>graphtraversalsource[emptygraph[empty], standard]
gremlin> g1.V().count()
==>0
gremlin> g2.V().count()
==>6

The TraveralOpProcessor on the Gremlin Server will use the "alias" argument in the Request Message to determine which traversal source defined in the gremlin server to use.

Also for writing GraphSON serializers, you might want to look at the I/O Tinkerpop documentation for GraphSON2 as well this sort of "formal spec" I had written a while ago for GraphSON2 if that may help. (reminder, the gremlin-server can only deserialize bytecode if it's serialized in GraphSON V2.0+ format)

Cheers.

[Ray Scott]

Hi Kevin, 

Thanks for confirming my understanding of aliases. I've been inspecting the same places in the code and configuration that you have referenced. 

You say that 

the Gremlin Server will use the "alias" argument in the Request Message to determine which traversal source defined in the gremlin server to use

and I don't doubt that it does. So when I read the documentation for the bytecode operation re: the alias property:

A map with a single key/value pair that refers to a globally bound TraversalSource object to be aliased to different variable names for purposes of the current request. The value represents the name of the global variable and its key represents the new binding name as it will be referenced in the Gremlin query.

I'm lead to believe that the alias property is about more than just selecting a key (and passing it with the request) to a previously defined mapping such as:

globals << [g : graph.traversal()]

instead it's to be used to define a new completely new (temporary & scoped to the current request) alias. 

So I would be creating an alias to g:  g1 => g. 

Except I get an error that says 

A message with [bytecode] op code requires the [aliases] argument to be a Map containing one alias assignment named 'g'.

OK so it's hardcoded to expect "g" as the key so, what do I pass as the value, mapped to the key "g"?  

The docs say 

 The value represents the name of the global variable

Which I though was "g" (defined as a global)  

globals << [g : graph.traversal()]

So I redefined my global reference to the traversal as "t", in the line directly above this, so that my request code can remap it to the expected key "g". I now get the exception described in my first post. 

Could it be failing because the global mapping defined in the default empty-sample.groovy, is pulled into the "gremlin-groovy" scriptEngines definition (found in gremlin-server.yaml), and I'm using bytecode and not the "gremlin-groovy" language specifier used when passing a script to the Gremlin Server for evaluation? I'm just wild guessing here.

Also, just to confirm, Bytecode should not be serialised as GraphSON 3? Only as GraphSON 2. The reason I ask is because the Python GLV code contains the follow lines in serializer.py:

# KEEP TRACK OF CURRENT DEFAULTS
DEFAULT_READER_CLASS = graphsonV3d0.GraphSONReader
DEFAULT_WRITER_CLASS = graphsonV3d0.GraphSONWriter
DEFAULT_VERSION = b"application/vnd.gremlin-v3.0+json"

def __init__(self, reader=None, writer=None, version=None):
    if not version:
        version = self.DEFAULT_VERSION

The code base does also contain a v2 GraphSON serializer, however. 

Anyone else have experience in this area? 

Thanks.

[Robert Dale]

If you're using TinkerPop 3.2.5, then you should use GraphSON 2.0.  If you're using TinkerPop 3.3.0, then you should use GraphSON 3.0.  I believe both Gremlin Servers are preconfigured and ready to go with the correct serializers.

Let's take python on 3.3.0:

graph = Graph()

g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','g'))

g.V().toList()

It sends the following to the server:

!application/vnd.gremlin-v3.0+json{"op": "bytecode", "requestId": {"@type": "g:UUID", "@value": "1ff5ff44-af1a-4e06-be13-5df1f1a11c8c"}, "args": {"gremlin": {"@type": "g:Bytecode", "@value": {"step": [["V"]]}}, "aliases": {"g": "g"}}, "processor": "traversal"}

Robert Dale

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/747aa1f3-b1b5-4ad4-88c0-facc4bd10978%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Dale]

If I change scripts/empty-sample.groovy:

globals << [server1 : graph.traversal()]

Then I have to update python:

g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','server1'))

Finally, it sends:

"aliases": {"g": "server1"}

Robert Dale

On Tue, Nov 7, 2017 at 8:06 AM, Robert Dale <rob...@gmail.com> wrote:

If you're using TinkerPop 3.2.5, then you should use GraphSON 2.0.  If you're using TinkerPop 3.3.0, then you should use GraphSON 3.0.  I believe both Gremlin Servers are preconfigured and ready to go with the correct serializers.

Let's take python on 3.3.0:

graph = Graph()

g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','g'))

g.V().toList()

It sends the following to the server:

!application/vnd.gremlin-v3.0+json{"op": "bytecode", "requestId": {"@type": "g:UUID", "@value": "1ff5ff44-af1a-4e06-be13-5df1f1a11c8c"}, "args": {"gremlin": {"@type": "g:Bytecode", "@value": {"step": [["V"]]}}, "aliases": {"g": "g"}}, "processor": "traversal"}

Robert Dale

[Ray Scott]

Thanks for the example serialized request Robert, I think I should code up the version 3 GraphSON De-Serializer, and then see what happens. Pointless otherwise, as I'm running Janus 0.2.0. 

Just on the point of serialising requests to GraphSON: is it only necessary for bytecode submissions? Because I don't serialize any of my other request messages through GraphSON, and they all work as expected. I just package them in the JSON as described in the Driver docs. This is the reason I haven't written any GraphSON serialization functionality. There doesn't seem to be any mention of GraphSON in the section on Graph Driver Provider Requirements. Perhaps it's mentioned in general terms somewhere else? Even so, I feel the docs could be improved in this area, although I realise I'm in the minority with regard to driver development.

One anomaly I've noticed is that when the server code borks, it's in GraphSON2 code. Also, when do the serialisers specified in the gremlin-server.yaml file come into play? I have no idea when or where they are used, just that I don't touch them in the config :-) There is nothing about GraphSON 3 in the config file. 

Does there need to be for this server processing of bytecode from a websocket to be deserialised correctly in Janus 0.2.0? 

Thanks for your help.

On Tuesday, 7 November 2017 13:20:47 UTC, Robert Dale wrote:

If I change scripts/empty-sample.groovy:

globals << [server1 : graph.traversal()]

Then I have to update python:

g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','server1'))

Finally, it sends:

"aliases": {"g": "server1"}

Robert Dale

On Tue, Nov 7, 2017 at 8:06 AM, Robert Dale <rob...@gmail.com> wrote:

If you're using TinkerPop 3.2.5, then you should use GraphSON 2.0.  If you're using TinkerPop 3.3.0, then you should use GraphSON 3.0.  I believe both Gremlin Servers are preconfigured and ready to go with the correct serializers.

Let's take python on 3.3.0:

graph = Graph()

g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','g'))

g.V().toList()

It sends the following to the server:

!application/vnd.gremlin-v3.0+json{"op": "bytecode", "requestId": {"@type": "g:UUID", "@value": "1ff5ff44-af1a-4e06-be13-5df1f1a11c8c"}, "args": {"gremlin": {"@type": "g:Bytecode", "@value": {"step": [["V"]]}}, "aliases": {"g": "g"}}, "processor": "traversal"}

Robert Dale

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

[Robert Dale]

JanusGraph 0.2.0 uses TinkerPop 3.2.6 so you should use GraphSON 2.0 serializer.

There was an issue with the 3.3.0 GraphSON 3.0 documentation that has been resolved in 3.3.1-SNAPSHOT however I don't think they are published any where.

Robert Dale

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/4e4a3f55-5bd4-494a-817c-202c0b575e89%40googlegroups.com.

[Ray Scott]

OK thanks, I guess I'll just have to be patient. 

Robert Dale

[Kevin Gallardo]

Quite a few questions, I'll try to clear that up with what I can/know.

So far you can communicate with 2 main "interface" with the Gremlin server in requests' messages. 

1. a String script. The Gremlin server interprets a script via a script engine which has tinkerpop libraries loaded in it, by default the script language is Gremlin-groovy. Which means you'll typically write "g.V().has('name', 'marko')" in the "gremlin" argument of the request message, no need for serialization (except if the script has parameters) because the string is then forwarded and interpreted by a Groovy script engine on the gremlin server. 
2. gremlin bytecode. Bytecode is a programming language-agnostic representation of Gremlin, and that's where serialization needs to kick in. Because in the Gremlin server, the bytecode received has to be deserialized (according to the chosen serialization protocol) into Bytecode instructions of the Gremlin traversal machine of the server, to be then translated into a Traversal object and be executed by the Gremlin traversal machine (trying to be extra careful about the terms I use :) ) of the server. 

One anomaly I've noticed is that when the server code borks, it's in GraphSON2 code 

Now, the request "interface" (one of the two explained above) and the serialization protocol (GraphSON, Gryo, ..) can be configured separately. You can choose GraphSON2 as the serialization protocol, but the request can be in "gremlin-groovy" string, meaning the request will be the string "g.V().has('name', 'marko')", but the results will be sent as GraphSON2. Different configuration: you can send the request as "bytecode" encoded in GraphSON2 and will get results in GraphSON2. You can chose which interface to use via the "op" argument of the request message. "op" will specify which OpProcessor to choose for the request, "op" = "eval" means you send a script for interpretation and the default language is Gremlin-groovy (StandardOpProcessor by default). "op" = "bytecode" means you send a serialized bytecode for the query ("traversal" OpProcessor should be set too).

The serialization protocol on another side, is chosen via the header of the request message and also indicated via the header in a response message, "application/vnd.gremlin-v3.0+json" for example for GraphSON3. More details on all of that is in the driver docs you linked.

Just on the point of serialising requests to GraphSON: is it only necessary for bytecode submissions? Because I don't serialize any of my other request messages through GraphSON, and they all work as expected.

What I suspect is that you send a message with "op" == "eval" and thus you don't need to serialize your request as Bytecode-GraphSON, because what the server expects is a gremlin-groovy string. If you're not too familiar with what Bytecode is for, I'd invite you to read more on the Gremlin Language Variants in the Tinkerpop docs.

Now back on aliases

When a query is a gremlin-groovy script, aliases allow to do 2 things, define which traversal source defined on the Gremlin server should be used and define how the traversal source is named in the script. Taking my previous example where I have "g1" and "g2" traversal sources defined in the server at init, I may want to name these traversal sources differently in my script, and the alias map helps me do that. Example:

def count1 = aliasedSource1.V().count().next();
def count2 = aliasedSource2.V().count().next();

[count1, count2]

If I write such a script, and have "g1" and "g2" defined on the server, I would need to send in the request the alias map: ['aliasedSource1':'g1', 'aliasedSource2':'g2'] because I have named my traversal sources differently in the script than what they are named in the server. And the server would normally return [0, 6].

gremlin> cluster = Cluster.open()
==>localhost/127.0.0.1:8182
gremlin> client = cluster.connect()
==>org.apache.tinkerpop.gremlin.driver.Client$ClusteredClient@5eb97ced

gremlin> clientWithAliases = client.alias(['aliasedSource1':'g1', 'aliasedSource2':'g2'])
==>org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient@57c88764
gremlin> clientWithAliases.submit('def count1 = aliasedSource1.V().count().next(); def count2 = aliasedSource2.V().count().next(); [count1, count2]').all().get()
==>result{object=0 class=java.lang.Long}
==>result{object=6 class=java.lang.Long}

So that's what the aliases are for gremlin-groovy scripts, but this does not apply for a bytecode query, since there is no use to "name your traversal source differently" in a bytecode query (because it's not a script), renaming the traversal source is unnecessary and so the argument for the renaming part of the alias map is always "g". So what you send in the alias map of a bytecode request should always be ["theNameOfYourTraversalSource":"g"].

Also, just to confirm, Bytecode should not be serialised as GraphSON 3? Only as GraphSON 2

I mentioned GraphSON V2.0+ which means either V2 or V3, it depends on which Tinkerpop version as Robert mentioned and also on what serializers you have configured in the server's .yaml.

Hope some of these explanations help.

[Kevin Gallardo]

Quick correction, 

So what you send in the alias map of a bytecode request should always be ["theNameOfYourTraversalSource":"g"]

According to the example I provided above it should rather be ["g":"theNameOfYourTraversalSource"]. 

[Ray Scott]

Kevin,

Thank you very much for clearing up some of the questions I had. In particular the sending of an alias with a bytecode query. That and the explanation regarding what/when and what not/when not to serialise with GraphSON. I'm fairly far along with the development. Just a few catch points that have stopped progress. Already completed session and session less processor code.

Thanks again!

[jcapo...@cbinsights.com]

Hi Ray,

We were thinking about working on a Go GLV as well.  It looks like you have a good start (at least).  Would you be willing to share the code on github?

Thanks,

Jeff

[Ray Scott]

Hi Jeff, 

Sure, I can release it for you. I have been/am ill, so haven't coded much recently. GraphSON work would still need to be coded up. I was in 2 minds regarding ever releasing this work because a) it doesn't always follow existing GLV naming conventions/pattens. b) my code depends on a logging framework I wrote myself. But if someone wants to put the code to some better use, then it's probably better to share the work. A rewrite in Rust is something planned for the future. 

I just need to sort out code licences before I release the code. The logging framework provides package level configuration and routing to (if I remember correctly off the top of my head) files, tcp addresses and system logs, per environment. 

I'll let you know when it's on GitHub.

Ray 

[jcapo...@cbinsights.com]

Thanks Ray! Looking forward to seeing your work.

[Ray Scott]

Hi Jeff, 

I've released the code on GitHub with an MIT license under the Runways company org: https://github.com/Runways

The repos you'll need. 

nb - https://github.com/Runways/nb

uuid - https://github.com/Runways/uuid

aeronaut - https://github.com/Runways/aeronaut

gremlin - https://github.com/Runways/gremlin

aeronaut - the Gremlin driver that supports standard, session & traversal processors. 

gremlin -  The GLV. Work in the gremlin repo is based off the Python and Java designs. Those GLVs are very capable. I've attempted to slim down to what I required, omitting classes and code I didn't require, at this stage. 

Have a look at the _test.go files for some clues as to how it's used. I say clues because they aren't formal test suites. I just used them to verify WIP. Ideally, I wouldn't release it in the state it's in. I haven't looked at, or run the code for nearly a year, and I was in the middle of development when I stopped. If you have any questions, I can try to answer them.    

Have you had a look at the Python GLV? 

[jeffro...@gmail.com]

Hi Ray,

Thanks for releasing this. I tried the code in gremlin_test.go on Janusgraph 0.2.0 and getting error message you'd posted in your first post. Is this the part that needs some work related to GraphSON or is it something on my side? 

janus_1  | java.lang.NullPointerException

janus_1  | at org.apache.tinkerpop.gremlin.structure.io.graphson.TraversalSerializersV2d0$BytecodeJacksonDeserializer.deserialize(TraversalSerializersV2d0.java:262)

janus_1  | at org.apache.tinkerpop.gremlin.structure.io.graphson.TraversalSerializersV2d0$BytecodeJacksonDeserializer.deserialize(TraversalSerializersV2d0.java:250)

janus_1  | at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypeDeserializer.deserialize(GraphSONTypeDeserializer.java:212)

janus_1  | at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypeDeserializer.deserializeTypedFromAny(GraphSONTypeDeserializer.java:101)

Thanks,

JR

[Ray Scott]

Hi Jeff,

What you’re experiencing is “correct”. That’s as far as I got, at the time. The gremlin repo code is missing a GraphSON Serializer & Deserializer, that are used before and after transmission. It’s failing because it’s not receiving the expected GraphSON message format. I found Kevin & Dave’s explanations above to be very helpful.

Any other questions, I’m happy to try help with.