Import / Export graph as json from remote server

[James Render]

Please may I have some example code on how to export an entire graph from a remote server, and then how to import.

I'm using Java and the Gremlin console.  I'm using an older version of the server, 3.3.4 (as I'm also working with Neptune that uses 3.3.2).

All the examples I've seen use the in-memory graphs (Tinkergraph).

My attempt is:

var traversal = EmptyGraph.instance().traversal().withRemote(DriverRemoteConnection.using(getCluster()))
traversal.getGraph().io(graphson()).writeGraph("my-graph.json");

results in an empty file.

thanks

I've a question up on Stackoverflow if you want to get points: https://stackoverflow.com/questions/56836705/how-do-i-export-a-remote-graph-to-json-using-tinkerpop-gremlin-and-neptune

[Kelvin Lawrence]

Hi James, here's some code that will do it for you with Neptune and should work with most Gremlin Server implementations I would think.

https://github.com/awslabs/amazon-neptune-tools/tree/master/neptune-export

The results of the export can be used to load via Neptune's bulk loader.

Hope that is useful

If that is more than you needed hopefully it will at least give you some pointers that help.

With  hosted graphs, including Neptune, it is not uncommon to find that they do not expose the Graph object or give access to the io() classes.

Cheers

Kelvin

[Kelvin Lawrence]

I should have added that the bulk loader expects CSV and not JSON so to load the results back into Neptune export as CSV rather than JSON.

[James Render]

thanks for the responses Kelvin.

I was looking for examples for use with Gremlin server.

My use case is to do integration testing by using the testcontainers framework and a docker image of gremlin server. Then I would like to load in some pre-defined data that I can write tests against.  I was looking to use Json as I find that most easily comprehended.

You're absolutely right about Neptune not giving access to the io() classes , probably to stop newbies like me trying to export a petabyte production graph!  I could export my graph from Neptune as JSON but not sure if it would then be compatible with Gremlin server (presumably this is backed by Tinkergraph in-memory by default).

[Kelvin Lawrence]

Ah I see, I had understood that you wanted to export from Neptune.

If you are using Gremlin server you have access to all of the io() classes.

From the Gremlin console (using the older graph object based approach) it would be as simple as:

gremlin> graph.io(graphson()).writeGraph("my-graph.json")

or using the newer TinkerPop 3.4 way

g.io("my-graph.json").write().iterate()

If you want to load that data into a different graph database (like Neptune) you will need to convert that JSON into a format that the graph DB can bulk import/load  or alternatively write a small program to load the data.

[Kelvin Lawrence]

One additional note, if you export your data as GraphML rather than JSON, there is a simple to run Python script/utility [1] that can convert GraphML into CSV

[1] https://github.com/awslabs/amazon-neptune-tools/tree/master/graphml2csv

[James Render]

Thanks Kelvin, that's not working for me think I'm lacking the understanding of something fundamental.

I've a gremlin-server running on localhost:8182

I fire up the console and try the command but it doesn't know the graph object locally. When I remotely execute it does know the graph but doesn't export.

~/apache-tinkerpop-gremlin-console-3.3.7/bin/gremlin.sh

gremlin> :remote connect tinkerpop.server conf/remote.yaml

gremlin> graph.io(graphson()).writeGraph("my-graph.json")
No such property: graph for class: groovysh_evaluate

gremlin> :> graph
==>tinkergraph[vertices:357 edges:883]
gremlin> :> graph.io(graphson()).writeGraph("yooblr-graph.json")
==>null


[James Render]

Ah my bad!

logged onto the docker container and there sitting in the

/opt/gremlin-server

was a lovely graph.json

I shall now try to import!  thank you very much, helpful having someone work through the process

[Fabio Lorenzi]

Just as a clarification, if you plan to move to a managed cloud graphDB provider (e.g. Compose) It's not a given that you'll get access to the gremlin server or have privileges to navigate the filesystem hence importing/exporting to files could not be a feasible option

[James Render]

Agreed.

This is for integration testing against a testcontainer running gremlin-server such that I can exercise my repository/dao layer.