Slow down on edge addition as the no. of edges increase

[Amit Chandak]

Hi,

       I am using JanusGraph Server, with berkeleyJE as storage backend, on a 8vCPU, 32Gb memory machine. I have set JAVA_OPTIONS = "-Xms512m -Xmx32768m". Here is my gremlin server config:

host: 0.0.0.0

port: 8182

threadPoolWorker: 16

gremlinPool: 0

scriptEvaluationTimeout: 180000

channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer

graphs: {

  graph: conf/gremlin-server/janusgraph-berkeleyje-server.properties

}

scriptEngines: {

  gremlin-groovy: {

    plugins: { org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {},

               org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},

               org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},

               org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},

               org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/empty-sample.groovy]}}}}

serializers:

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

  # Older serialization versions for backwards compatibility:

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0, config: {ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}

processors:

  - { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}

  - { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}

metrics: {

  consoleReporter: {enabled: false, interval: 180000},

  csvReporter: {enabled: false, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},

  jmxReporter: {enabled: false},

  slf4jReporter: {enabled: false, interval: 180000},

  gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},

  graphiteReporter: {enabled: false, interval: 180000}}

maxInitialLineLength: 4096

maxHeaderSize: 8192

maxChunkSize: 8192

maxContentLength: 65536000

maxAccumulationBufferComponents: 1024

resultIterationBatchSize: 64

writeBufferLowWaterMark: 32768

writeBufferHighWaterMark: 65536000

Here is the config for the janus graph

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=berkeleyje

storage.directory=/data/janus

storage.berkeleyje.lock-mode=LockMode.READ_UNCOMMITTED

#schema.default=non

I am seeing after 500k edges, edge insert time went up from under 10seconds to 2+ minutes. All the edges have same property keys.
All the inserts have parametrization to avoid script cache build-up in gremlin.

How can i debug this slow down? Any obvious config i need to turn on to speed this up. I am using websocket based golang driver https://github.com/go-gremlin/gremlin.

Thanks in advance,

Amit

[Pavel Ershov]

Related problem https://groups.google.com/forum/#!topic/janusgraph-users/rGOmhVn9fC4

пятница, 17 мая 2019 г., 10:43:14 UTC+3 пользователь Amit Chandak написал:

[Amit Chandak]

Thanks Pavel, I dont think i have this issue, reason being, my add Edge queries are like this:

g.addEdge('foo').from(V(sourceID)).to(V(destID)) - then i store the ID returned from this

g.E(IDfromAbove).property('k1', 'v1').property('k2', 'v2)...

Since its index based, i dont have any *has* operator.