TP3 3.1 SparkGraphComputer doesn't support spark on YARN?

Christine Li

unread,

Nov 30, 2015, 12:02:24 PM11/30/15

to Gremlin-users

Spark is deployed on Yarn. I've tried blvp sample with by set spark.master=yarn-client and got the follow error:

Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil

Looks like Spark on Yarn isn't supported. Is it the case?

Thanks,

MeteorMarc

unread,

Dec 2, 2015, 3:25:14 PM12/2/15

to Gremlin-users

Hi Christine,

I built TP 3.1 against Hortonworks 2.3.2 and added configs in the conf/hadoop folder of gremlin-console to support spark on yarn. See:

https://github.com/vtslab/incubator-tinkerpop

Good luck,

Marc

Op maandag 30 november 2015 18:02:24 UTC+1 schreef Christine Li:

Marko Rodriguez

unread,

Dec 2, 2015, 3:32:35 PM12/2/15

to gremli...@googlegroups.com, d...@tinkerpop.incubator.apache.org

Hi MeteorMarc,

That is really cool of you to do that.

Can you issue a pull request that updates the respective hadoop-gremlin/conf files with ONLY the necessary properties and then # comment them out so people will be able to see what is required if they want to use Spark+YARN.

Thank you,

Marko.

http://markorodriguez.com

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/d7472deb-db15-4ab6-b6fb-e6766ad24c69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

HadoopMarc

unread,

Dec 3, 2015, 9:51:33 AM12/3/15

to Gremlin-users, d...@tinkerpop.incubator.apache.org

Hi Marko,

Thanks for your interest. I will do this, but give me a few days until I have an evening free.

Regards, Marc

Op woensdag 2 december 2015 21:32:35 UTC+1 schreef Marko A. Rodriguez:

Christine Li

unread,

Dec 4, 2015, 1:03:30 PM12/4/15

to Gremlin-users, d...@tinkerpop.incubator.apache.org

Hi, Marc

I don't see a hadoop folder on https://github.com/vtslab/incubator-tinkerpop/tree/3.1.0-hdp-2.3.2.0-2950/gremlin-console/conf

Did I miss something?
Thanks,

Jason Plurad

unread,

Dec 4, 2015, 1:16:41 PM12/4/15

to Gremlin-users, d...@tinkerpop.incubator.apache.org

Christine, look in here https://github.com/vtslab/incubator-tinkerpop/tree/3.1.0-hdp-2.3.2.0-2950/hadoop-gremlin/conf

Christine Li

unread,

Dec 7, 2015, 3:30:03 PM12/7/15

to Gremlin-users, d...@tinkerpop.incubator.apache.org

Sorry for this 101 question. When I ran blvp sample with spark on yarn. I've got ClassNoFound for org.apache.spark.deploy.yarn.YarnSparkHadoopUtil. I built TP3.1 from https://github.com/vtslab/incubator-tinkerpop/tree/3.1.0-hdp-2.3.2.0-2950 and didn't see jars contains YarnSparkHadoopUtil either. Did I miss something?

HadoopMarc

unread,

Dec 8, 2015, 3:10:32 AM12/8/15

to Gremlin-users, d...@tinkerpop.incubator.apache.org

Hi Cristine,

Did you start the repl with gremlinhdp.sh? It does all the exports needed and sets the CLASSPATH. Check if the configs are right for your cluster.

https://github.com/vtslab/incubator-tinkerpop/blob/3.1.0-hdp-2.3.2.0-2950/gremlin-console/bin/gremlinhdp.sh