Performance numbers of Gremlin Server (3.3.1) with Neo4j plugin and golang driver

[Amit Chandak]

Hi,

      I am trying to get some performance numbers for the gremlin server with neo4j plugin. Our rationale behind using gremlin server was to have the flexibility of using any tinker pop enabled db like neo4j, janus, etc. and the applications don't have to change as they talk to the gremlin server. We mostly use golang for development. 

Setup details

m4.large EC2 instance - 2 vCPUs and 8GiB Memory

Gremlin server running inside a docker

Client written in go

Here is the client code:

  start := time.Now()

  query = `g.addV(x)`

  for i := 0; i < 100000; i++ {

    node := "node" + strconv.Itoa(i)

    req := gremlin.Query(query).Bindings(gremlin.Bind{"x": node})

    conn.Exec(req)

    //fmt.Println(string(res), err)

  }

  elapsed := time.Since(start)

  fmt.Println("Adding vertices, time taken", elapsed)

Here are the time elapsed nos, for different no. of nodes

1000 nodes - 7.274606675s

10000 nodes - 51.048624896s

100000 nodes - the client code didn't return for almost 10 mins, didn't have the patience, so killed it.

But these numbers aren't looking good, especially when i have used parametrized script as mentioned here https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/gremlin-users/zskLgoE2bK8/_BgRKVPECAAJ

I remember seeing some discussion using byte code serialization/deserialization seem to perform better than scripts. Do we have performance numbers using byte-code serialization published somewhere?

I see gremlin variants have been written to make use of byte code to perform better, but i don't see a variant in golang. My question would be 2 fold:

1. Is Gremlin server ready for production? Does it have HA/autoscale support? Is there any documentation on this?

2. Is golang not the choice of language for interfacing with gremlin???

Thanks and appreciate your help on this front. One of the most active google groups i have seen in recent times, kudos for that !!!

Amit

 

[Stephen Mallette]

> 1. Is Gremlin server ready for production? 

yes - and has been for many years now. i've heard of many production implementations in that time.

> Does it have HA/autoscale support?  Is there any documentation on this?

not really - typically, people just put Gremlin Server behind a load balancer or rely on the client side load balancing strategy in the java driver (which is just round-robin at this point - it's worth noting that non-java drivers may not support that).

> 2. Is golang not the choice of language for interfacing with gremlin???

TinkerPop officially supports, java, groovy, python, c#/.net, and very very soon - javascript. Any other libraries for other languages are supported by the general TinkerPop community and we typically refer to them as "third-party" libraries. Some of these libraries are very well maintained and managed like gremlin-scala and Ogre (clojure) and others are maintained to the level that their developers can manage. I don't know much about the Go drivers that are out there and which might be "the best" or how well any of them work. I will say that Go has to be the next most requested language to get "official" TinkerPop support - I don't know if that will happen any time soon.

I don't know offhand what to say about your performance test as I don't do much with Neo4j. A few observations I had:

1. stop returning the vertex if you don't really need it: g.addV(x).iterate()

2. how much memory did you give Gremlin Server? just the default? maybe throw some more -Xmx at it? same for neo4 - what were your memory configs there? their docs talk about a heap size of 8G minimum for reliable operations. you only have 8G on your whole VM.

3. consider your use case...is your test representative of what you will be doing in your application? lots of little addV() operations? is that for bulk loading requirements? if so, you might want to look at other methods for doing that (perhaps even something specific to Neo4j's APIs - Gremlin Server likely won't beat their bulk loading features).

4. The 100000 vertex test not returning for 10 minutes seems kinda odd - it would be interesting to see what would happen if you re-tested with more memory. Not sure what else would cause that to happen.

5. As I said, earlier I don't know how the Go driver works. When you do:  conn.Exec(req) - does that block? or does it fire-and-forget? if it fires and forgets then you might be overrunning Gremlin Server with requests to some degree maybe??? Gremlin Server will queue requests as they arrive. the Java driver has the capability to slows things down if the server gets overloaded (the server slows down writes back to the client if the cilent can't respond fast enough too btw), but if the Go driver isn't that advanced it might not be doing anything smart there. Just throwing ideas out there as to what might be wrong.....i could be way off on that one.

>  One of the most active google groups i have seen in recent times, kudos for that !!!

That's a really nice compliment. I'm sure everyone who participates on this list appreciates hearing that. Thanks!

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/83d25ff4-2081-42fd-ac7e-87cab7b1148b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jean-Philippe B]

Excerpts from Amit Chandak's message of February 26, 2018 11:59 pm:

> Hi,
> I am trying to get some performance numbers for the gremlin server
> with neo4j plugin. Our rationale behind using gremlin server was to have
> the flexibility of using any tinker pop enabled db like neo4j, janus, etc.
> and the applications don't have to change as they talk to the gremlin
> server. We mostly use golang for development.
>

> *Setup details*

> m4.large EC2 instance - 2 vCPUs and 8GiB Memory
> Gremlin server running inside a docker
> Client written in go
>
> Here is the client code:
>
> start := time.Now()
> query = `g.addV(x)`
> for i := 0; i < 100000; i++ {
> node := "node" + strconv.Itoa(i)
> req := gremlin.Query(query).Bindings(gremlin.Bind{"x": node})
> conn.Exec(req)
> //fmt.Println(string(res), err)
> }
> elapsed := time.Since(start)
> fmt.Println("Adding vertices, time taken", elapsed)
>
> Here are the time elapsed nos, for different no. of nodes
>
> 1000 nodes - 7.274606675s
> 10000 nodes - 51.048624896s
> 100000 nodes - the client code didn't return for almost 10 mins, didn't
> have the patience, so killed it.
>

Hello,

Which go driver do you use ?

Personnaly I started to use https://github.com/go-gremlin/gremlin but
this driver is making a new WS connection for each request and it's not
maintained, so I forked it and added a concurrent client in it (https://github.com/eonpatapon/gremlin).

With your code for 10 000 vertices against a local gremlin-server (bin/gremlin-server.sh conf/gremlin-server-modern.yaml) it takes ~11ms.


package main

import (
"fmt"
"strconv"
"time"

"github.com/eonpatapon/gremlin"
)

func main() {
nb := 10000
client := gremlin.NewClient("ws://localhost:8182/gremlin")

start := time.Now()
query := `g.addV(x)`

for i := 0; i < nb; i++ {

node := "node" + strconv.Itoa(i)
req := gremlin.Query(query).Bindings(gremlin.Bind{"x": node})

client.Send(req)
}
elapsed := time.Since(start)
fmt.Printf("Adding %d vertices, time taken %s\n", nb, elapsed)
}

[Jean-Philippe B]

Excerpts from Jean-Philippe B's message of February 27, 2018 10:12 am:

Ops I forgot the client.Connect() ^^

It was to good to be true.

So for 10 000 vertices:

$ ./test
Adding 10000 vertices, time taken 5.66836833s
$ ./test
Adding 10000 vertices, time taken 2.90066247s
$ ./test
Adding 10000 vertices, time taken 1.882543537s
$ ./test
Adding 10000 vertices, time taken 1.785202587s
$ ./test
Adding 10000 vertices, time taken 1.791969617s

The server needs to warmup a little bit :)

JP

[Amit Chandak]

Thanks Stephen for your comments. My comments inline:

On Monday, February 26, 2018 at 4:31:05 PM UTC-8, Stephen Mallette wrote:

> 1. Is Gremlin server ready for production? 

yes - and has been for many years now. i've heard of many production implementations in that time.

> Does it have HA/autoscale support?  Is there any documentation on this?

not really - typically, people just put Gremlin Server behind a load balancer or rely on the client side load balancing strategy in the java driver (which is just round-robin at this point - it's worth noting that non-java drivers may not support that).

AMIT>> Does gremlin server support talking to each other? I thought JanusGraph (which uses gremlin server) doesn't support inter-gremlin communication. It relies on the storage backend like cassandra, HBASE for coordination. However, neo4j cluster model is different, it relies on RAFT protocol (which requires the servers to communicate) to distribute the data. Since neo4j doesn't have this concept of storage backend, we would require the gremlin servers to talk to each other, which AFAIK is not there.

 

> 2. Is golang not the choice of language for interfacing with gremlin???

TinkerPop officially supports, java, groovy, python, c#/.net, and very very soon - javascript. Any other libraries for other languages are supported by the general TinkerPop community and we typically refer to them as "third-party" libraries. Some of these libraries are very well maintained and managed like gremlin-scala and Ogre (clojure) and others are maintained to the level that their developers can manage. I don't know much about the Go drivers that are out there and which might be "the best" or how well any of them work. I will say that Go has to be the next most requested language to get "official" TinkerPop support - I don't know if that will happen any time soon.

I don't know offhand what to say about your performance test as I don't do much with Neo4j. A few observations I had:

1. stop returning the vertex if you don't really need it: g.addV(x).iterate()

AMIT>> That did improve the performance. 

2. how much memory did you give Gremlin Server? just the default? maybe throw some more -Xmx at it? same for neo4 - what were your memory configs there? their docs talk about a heap size of 8G minimum for reliable operations. you only have 8G on your whole VM.

AMIT>> You are right, i didn't do any memory tuning, as i wanted to compare the following combinations with baseline config:

1. Gremlin server with Neo4j plugin

2. Cypher driver with native Neo4j server, using BOLT protocol

3. Gremlin driver with Neo4j (with gremlin plugin)

Surprisingly, the results for 1 and 2 look quite similar.

3. consider your use case...is your test representative of what you will be doing in your application? lots of little addV() operations? is that for bulk loading requirements? if so, you might want to look at other methods for doing that (perhaps even something specific to Neo4j's APIs - Gremlin Server likely won't beat their bulk loading features).

AMIT>> Neo4j specific apis will be exposed by the neo4j plugin???

 

4. The 100000 vertex test not returning for 10 minutes seems kinda odd - it would be interesting to see what would happen if you re-tested with more memory. Not sure what else would cause that to happen.

AMIT>> I will do the analysis and send u an update.

 

5. As I said, earlier I don't know how the Go driver works. When you do:  conn.Exec(req) - does that block? or does it fire-and-forget? if it fires and forgets then you might be overrunning Gremlin Server with requests to some degree maybe??? Gremlin Server will queue requests as they arrive. the Java driver has the capability to slows things down if the server gets overloaded (the server slows down writes back to the client if the cilent can't respond fast enough too btw), but if the Go driver isn't that advanced it might not be doing anything smart there. Just throwing ideas out there as to what might be wrong.....i could be way off on that one.

AMIT>> I don't think golang driver does handle this backoff kind of feature.

 

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

[Amit Chandak]

hi Jean,

           I was using the go-gremlin driver too. Why do you say it makes a web socket connection for each request. Here's i was using it.

func main() {

 18   wsUrl := "ws://192.168.99.100:30085/gremlin"

 19   conn, _ := gremlin.NewClient(wsUrl)

 20   //session := "1234"

 21   //slice := map[string]interface{}{"type": "s3", "org": "achandak-org"}

 22   //slice := map[string]interface{}{"type": 4, "orgName": "orkus-org"}

 23   //query := `g.V()`

 24   //query := `graph.addVertex(T.label, x, y, z)`

 25   //req := gremlin.Query(query).Bindings(gremlin.Bind{"x": "hello3", "y": "org", "z": "achandak-org"})

 26   count := []int{1000, 5000, 10000, 50000, 100000, 5000000, 10000000}

 27   query := `g.addV(x)`

 28   for j := range count {

 29     start := time.Now()

 30     for i := 0; i < count[j]; i++ {

 31       node := "node" + strconv.Itoa(i)

 32       req := gremlin.Query(query).Bindings(gremlin.Bind{"x": node})

 33       conn.Exec(req)

 34       //fmt.Println(string(res), err)

 35     }

 36     elapsed := time.Since(start)

 37     fmt.Printf("Adding %d vertices, time taken %v\n", count[j], elapsed)

 38   }

NewClient was opening the web socket connection and Exec was sending the request on the connected web socket.

Looking at your nos, thats a huge improvement, i can give it a try and post the nos. with that.

Amit

[Stephen Mallette]

>  Since neo4j doesn't have this concept of storage backend, we would require the gremlin servers to talk to each other, which AFAIK is not there.

Wouldn't you just configure each neo4j instance in Gremlin Server with an HA configuration?

http://tinkerpop.apache.org/docs/current/reference/#_high_availability_configuration

>  That did improve the performance.

avoid returning results you don't need and prefer returning primitive Map/List over graph elements like Vertex/Edge

>  Surprisingly, the results for 1 and 2 look quite similar.

That's good - I don't think that's too surprising given your test. It's just doing a simple addV() - so basically inserts. there's not a lot of difference there in terms of the underlying code to execute between the two systems.

>   Neo4j specific apis will be exposed by the neo4j plugin???

If you are sending scripts to Gremlin Server, you technically have access to the entire underlying Neo4j API. However, I was really trying to get across the point that you TinkerPop generally recommends using graph provider specific tooling for things like bulk loading and you probably don't want to do that through Gremlin Server. Better to just use bulk loading tools directly if you can. I don't think there's anything wrong in bypassing Go for that kind of usage. It's batch, it's one time (typically) - done.

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/2c460f6d-f960-45a6-942d-1db45512c66c%40googlegroups.com.

[Jean-Philippe B]

Excerpts from Amit Chandak's message of February 27, 2018 10:58 pm:

Hi Amit,

You are right, somebody has included a client 2 months ago
(https://github.com/go-gremlin/gremlin/commit/2a18e9929fc74f80c521f051f1ead539af75202b). At the time I started to use it there wasn't.

This client doesn't support concurrency though.

JP