[Tinkerpop3] Load data into RemoteGraph/Connection

[Matt Burgess]

Hello all,

I have been a fairly-long-time user of Tinkerpop 2, and am now getting into Tinkerpop 3 (congrats on Apache Incubator status btw!). I'm using 3.2.0-incubating.

My use case is this:  I have a Java app (Apache NiFi) collecting graph data, let's say it's in Tinkerpop3 GraphSON format. I'd like to offer a service in NiFi for sending the vertices and edges from GraphSON into a Graph hosted on a different server (so, a RemoteGraph or using a RemoteConnection).

I have been prototyping in Groovy Console, to see how I could do it without the Gremlin shell, but I'm not having any luck being able to read a GraphSON doc into a RemoteGraph.  When I try this:

graph = RemoteGraph.open('conf/remote-graph.properties')

graph.io(IoCore.graphson()).readGraph("data/tinkerpop-modern.json")

I get: "java.lang.UnsupportedOperationException: The default implementation is not capable of validating custom ids - please override".  I see that RemoteGraph is not a full graph implementation, but was hoping to be able to do some custom conversions.  Following the doc here:

http://tinkerpop.apache.org/docs/3.2.0-incubating/reference/#graphson-reader-writer

I tried:

stream = new FileInputStream("data/tinkerpop-modern.json")

graph.io(IoCore.graphson()).reader().vertexIdKey("name").create().readGraph(stream, graph)

I get: "groovy.lang.MissingMethodException: No signature of method: org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONReader$Builder.vertexIdKey() is applicable for argument types: (java.lang.String) values: [name]". A quick look at GitHub returns no results for the vertexIdKey method, save for the docs themselves:

https://github.com/apache/incubator-tinkerpop/search?utf8=%E2%9C%93&q=vertexIdKey

Do I need to do a manual graph copy, like reading the GraphSON into a TinkerGraph and then adding the elements (well, new elements with the same properties to avoid ID collisions) one by one via sending Traversals to the RemoteGraph?

Thank you in advance,

Matt

[Stephen Mallette]

RemoteGraph isn't a full Graph implementation as you have found. It is mainly for spawning traversals which can be remotely executed against Gremlin Server, which is nicer than embedding Gremlin strings all over your code. I suppose there might be some way to make RemoteGraph support that method, I don't know if that's worth the effort involved. If you wanted to stick to this approach of shipping around GraphSON files, i guess you would drop down to use the driver and submit a gremlin script with the file bytes to gremlin server and have it execute over there.

> Do I need to do a manual graph copy, like reading the GraphSON into a TinkerGraph and then adding the elements (well, new elements with the same properties to avoid ID collisions) one by one via sending Traversals to the RemoteGraph?

I guess you could do a local read of the GraphSON and ship it over RemoteGraph, though that seems "wrong" but I don't know much about Nifi so it's hard to say. Generally speaking, if your only use for RemoteGraph is to addV() and addE(), then the primary benefit of RemoteGraph is sorta lost because you aren't really building any complex traversals that would be hard to maintain as strings. In your case, it sounds like you're just doing data loading. For data loading of this nature, I might just use the gremlin-driver directly and rely on the structure APIs (rather than Traversal - process API) to do the job.

If you are collecting data as GraphSON strings in Nifi, then you could just forward each bit GraphSON to Gremlin Server over the driver and parse it server side with the script you send from the client. In that way you could even batch up a bunch of vertex/edge adds in a single transaction (no support for batching yet in the driver). Again, not knowing Nifi, may have me way off base here - just throwing out ideas.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/1810e705-8385-4180-a4cc-019c06ec78db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.