Query using SparkGraphComputer in Janus Graph giving error

Himanshu Gupta

unread,

Oct 24, 2017, 9:56:51 AM10/24/17

to JanusGraph users

Hi,

I am using Janus 0.1.1 with Cassandra 3.0.14 as a backend. Can anyone help me how to traverse graph using Spark with Canssandra as a backend ? Can you specify the configurations?

current I'm using following configuration file: read-cassandra.properties

--------------------------------------------------------------------------------

#

# Hadoop Graph Configuration

#

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph

gremlin.hadoop.graphInputFormat=org.janusgraph.hadoop.formats.cassandra.CassandraInputFormat

gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat

gremlin.hadoop.jarsInDistributedCache=true

gremlin.hadoop.inputLocation=none

gremlin.hadoop.outputLocation=output

#

# JanusGraph Cassandra InputFormat configuration

#

janusgraphmr.ioformat.conf.storage.backend=cassandra

janusgraphmr.ioformat.conf.storage.hostname=localhost

janusgraphmr.ioformat.conf.storage.port=9160

janusgraphmr.ioformat.conf.storage.cassandra.keyspace=janusgraph

janusgraphmr.ioformat.cf-name=edgestore

storage.backend=cassandra

#

# Apache Cassandra InputFormat configuration

#

cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner

spark.driver.host=localhost

#

# SparkGraphComputer Configuration

#

spark.master=local[4]

spark.serializer=org.apache.spark.serializer.KryoSerializer

--------------------------------------------------------------------------------

I do following operations:

gremlin> graph = GraphFactory.open("conf/hadoop-graph/read-cassandra.properties")

==>hadoopgraph[cassandrainputformat->gryooutputformat]

gremlin> g = graph.traversal().withComputer(SparkGraphComputer)

==>graphtraversalsource[hadoopgraph[cassandrainputformat->gryooutputformat], sparkgraphcomputer]

gremlin> g.V().count()

java.io.IOException: Could not get input splits

What could be the reason for this error? How do i resolve this?

Ted Wilmes

unread,

Oct 24, 2017, 4:39:12 PM10/24/17

to JanusGraph users

Hello,

The record reader that comes with JanusGraph 0.1.1 is not compatible with Cassandra 3. Can you try using the latest JanusGraph 0.3.0 release and the read-cassandra-3.properties configuration? Here's the relevant issue that was fixed in 0.2.0: https://github.com/JanusGraph/janusgraph/issues/172

Thanks,

Ted

Message has been deleted

Himanshu Gupta

unread,

Oct 26, 2017, 1:58:59 AM10/26/17

to JanusGraph users

Hi Ted Wilmes,

Even using latest janus graph 0.2.0 and read-cassandra-3.properties still I'm facing same error.

actully I want to use spark graph computer from java api

I did following operations in my java code

---------------------------------------------------------------------------------------------

import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;

import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;

import org.apache.tinkerpop.gremlin.structure.Graph;

import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;

public class Sample{

public static void main(String[] args) {

Graph graph = GraphFactory.open("read-cassandra-3.properties");

GraphTraversalSource g = graph.traversal().withComputer(SparkGraphComputer.class);

System.out.println(g.V().count().next());

}

---------------------------------------------------------------------------------------------

Error I'm getting is

java.lang.IllegalStateException: java.lang.ExceptionInInitializerError

at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.VertexProgramStep.processNextStart(VertexProgramStep.java:88)

at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)

at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)

at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:68)

at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:128)

at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:38)

at org.apache.tinkerpop.gremlin.process.traversal.Traversal.fill(Traversal.java:177)

at org.apache.tinkerpop.gremlin.process.traversal.Traversal.toList(Traversal.java:115)

at com.apm.main.Sample.main(Sample.java:18)

Caused by: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError

at java.util.concurrent.FutureTask.report(Unknown Source)

at java.util.concurrent.FutureTask.get(Unknown Source)

at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.VertexProgramStep.processNextStart(VertexProgramStep.java:68)

... 8 more

Caused by: java.lang.ExceptionInInitializerError

at org.apache.spark.SparkContext.withScope(SparkContext.scala:714)

at org.apache.spark.SparkContext.newAPIHadoopRDD(SparkContext.scala:1129)

at org.apache.spark.api.java.JavaSparkContext.newAPIHadoopRDD(JavaSparkContext.scala:507)

at org.apache.tinkerpop.gremlin.spark.structure.io.InputFormatRDD.readGraphRDD(InputFormatRDD.java:42)

at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$0(SparkGraphComputer.java:215)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too old 2.4.4

at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:56)

at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)

at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:549)

at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:81)

at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

... 9 more

- show quoted text -

Ankur Goel

unread,

Oct 26, 2017, 5:44:12 AM10/26/17

to JanusGraph users

Please upgrade your Jackson dependency version.

~

Ted Wilmes

unread,

Oct 26, 2017, 8:21:13 AM10/26/17

to Ankur Goel, JanusGraph users

Hello,

Good suggestion on the Jackson version Ankur. Himansha, you can also try running it from the Gremlin console to confirm it works.

Thanks,

Ted

--
You received this message because you are subscribed to a topic in the Google Groups "JanusGraph users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/janusgraph-users/cWAasFVmgS8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to janusgraph-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/a9b5018a-809e-4f08-a522-3f75c89468c5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Himanshu Gupta

unread,

Oct 26, 2017, 8:46:28 AM10/26/17

to JanusGraph users

hi all,

I tried with all latest version but still I got this error -

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.2

at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)

at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)

at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)

at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:81)

at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

... 9 more

Ted I did same thing in janusgraph 0.2.0 gremlin console also I got same error which I was getting in previous version "java.io.IOException: Could not get input splits"

As per https://github.com/JanusGraph/janusgraph/releases/ janusgraph 0.2.0 support gremlin 3.2.6 and I created a maven project, So these

dependencies came from there only I'm sharing my pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>com.apm.myapp</groupId>

<artifactId>myapp_spark</artifactId>

<version>0.0.1-SNAPSHOT</version>

<name>myapp-Spark</name>

<dependencies>

<dependency>

<groupId>org.apache.tinkerpop</groupId>

<artifactId>gremlin-core</artifactId>

<version>3.2.6</version>

</dependency>

<dependency>

<groupId>org.apache.tinkerpop</groupId>

<artifactId>hadoop-gremlin</artifactId>

<version>3.2.6</version>

</dependency>

<dependency>

<groupId>org.apache.tinkerpop</groupId>

<artifactId>spark-gremlin</artifactId>

<version>3.2.6</version>

</dependency>

<dependency>

<groupId>org.janusgraph</groupId>

<artifactId>janusgraph-core</artifactId>

<version>0.2.0</version>

</dependency>

<dependency>

<groupId>org.janusgraph</groupId>

<artifactId>janusgraph-cassandra</artifactId>

<version>0.2.0</version>

</dependency>

<dependency>

<groupId>org.janusgraph</groupId>

<artifactId>janusgraph-solr</artifactId>

<version>0.2.0</version>

</dependency>

<dependency>

<groupId>log4j</groupId>

<artifactId>log4j</artifactId>

<version>1.2.17</version>

</dependency>

<dependency>

<groupId>org.janusgraph</groupId>

<artifactId>janusgraph-hadoop-core</artifactId>

<version>0.2.0</version>

<exclusions>

<exclusion>

<groupId>org.janusgraph</groupId>

<artifactId>janusgraph-hbase-core</artifactId>

</exclusion>

</exclusions>

</dependency>

</dependencies>

</project>

To unsubscribe from this group and all its topics, send an email to janusgraph-use...@googlegroups.com.

Himanshu Gupta

unread,

Oct 27, 2017, 8:50:50 AM10/27/17

to JanusGraph users

do you think I missed something ?

Ted Wilmes

unread,

Oct 27, 2017, 8:53:02 AM10/27/17

to JanusGraph users

Hi Himansha,

Maybe I missed it, but can you post the full IOException you're getting when you can't get the input splits?

Thanks,

Ted

Himanshu Gupta

unread,

Oct 30, 2017, 1:42:05 AM10/30/17

to JanusGraph users

Hi Ted,

here it is full IOException

\,,,/

(o o)

-----oOOo-(3)-oOOo-----

plugin activated: janusgraph.imports

plugin activated: tinkerpop.server

plugin activated: tinkerpop.utilities

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/ubuntu/software/janusgraph-0.2.0-hadoop2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/ubuntu/software/janusgraph-0.2.0-hadoop2/lib/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

05:16:45 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

plugin activated: tinkerpop.hadoop

plugin activated: tinkerpop.spark

plugin activated: tinkerpop.tinkergraph

gremlin> graph = GraphFactory.open("conf/hadoop-graph/read-cassandra-3.properties");

==>hadoopgraph[cassandra3inputformat->gryooutputformat]

gremlin> g = graph.traversal().withComputer(SparkGraphComputer);

==>graphtraversalsource[hadoopgraph[cassandra3inputformat->gryooutputformat], sparkgraphcomputer]

gremlin> g.V().count()

05:19:00 WARN org.apache.spark.util.Utils - Your hostname, ubuntuamaster resolves to a loopback address: 127.0.0.1; using 10.0.0.10 instead (on interface eth0)

05:19:00 WARN org.apache.spark.util.Utils - Set SPARK_LOCAL_IP if you need to bind to another address

java.io.IOException: Could not get input splits

Type ':help' or ':h' for help.

Display stack trace? [yN]y

java.lang.IllegalStateException: java.io.IOException: Could not get input splits

at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.VertexProgramStep.processNextStart(VertexProgramStep.java:88)

at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)

at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)

at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:68)

at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)

at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)

at org.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:234)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)

at org.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:447)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)

at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:191)

at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)

at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)

at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:72)

at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)

at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)

at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)

at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:166)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)

at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:478)

Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not get input splits

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.VertexProgramStep.processNextStart(VertexProgramStep.java:68)

... 56 more

Caused by: java.io.IOException: Could not get input splits

at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:203)

at org.janusgraph.hadoop.formats.cassandra.CassandraBinaryInputFormat.getSplits(CassandraBinaryInputFormat.java:62)

at org.janusgraph.hadoop.formats.util.GiraphInputFormat.getSplits(GiraphInputFormat.java:62)

at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:120)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

at scala.Option.getOrElse(Option.scala:120)