TP3 3.1 SparkGraphComputer doesn't support spark on YARN?

150 views
Skip to first unread message

Christine Li

unread,
Nov 30, 2015, 12:02:24 PM11/30/15
to Gremlin-users
Spark is deployed on Yarn. I've tried blvp sample with by set spark.master=yarn-client and got the follow error:

Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil

Looks like Spark on Yarn isn't supported. Is it the case?

Thanks,

MeteorMarc

unread,
Dec 2, 2015, 3:25:14 PM12/2/15
to Gremlin-users
Hi Christine,

I built TP 3.1 against Hortonworks 2.3.2 and added configs in the conf/hadoop folder of gremlin-console to support spark on yarn. See:

https://github.com/vtslab/incubator-tinkerpop

Good luck,

Marc



Op maandag 30 november 2015 18:02:24 UTC+1 schreef Christine Li:

Marko Rodriguez

unread,
Dec 2, 2015, 3:32:35 PM12/2/15
to gremli...@googlegroups.com, d...@tinkerpop.incubator.apache.org
Hi MeteorMarc,

That is really cool of you to do that.

Can you issue a pull request that updates the respective hadoop-gremlin/conf files with ONLY the necessary properties and then # comment them out so people will be able to see what is required if they want to use Spark+YARN.

Thank you,
Marko.
--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/d7472deb-db15-4ab6-b6fb-e6766ad24c69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

HadoopMarc

unread,
Dec 3, 2015, 9:51:33 AM12/3/15
to Gremlin-users, d...@tinkerpop.incubator.apache.org
Hi Marko,

Thanks for your interest. I will do this, but give me a few days until I have an evening free.

Regards,    Marc

Op woensdag 2 december 2015 21:32:35 UTC+1 schreef Marko A. Rodriguez:

Christine Li

unread,
Dec 4, 2015, 1:03:30 PM12/4/15
to Gremlin-users, d...@tinkerpop.incubator.apache.org
Hi, Marc

I don't see a hadoop folder on https://github.com/vtslab/incubator-tinkerpop/tree/3.1.0-hdp-2.3.2.0-2950/gremlin-console/conf

Did I miss something?
Thanks,

Jason Plurad

unread,
Dec 4, 2015, 1:16:41 PM12/4/15
to Gremlin-users, d...@tinkerpop.incubator.apache.org

Christine Li

unread,
Dec 7, 2015, 3:30:03 PM12/7/15
to Gremlin-users, d...@tinkerpop.incubator.apache.org
Sorry for this 101 question. When I ran blvp sample with spark on yarn. I've got ClassNoFound for org.apache.spark.deploy.yarn.YarnSparkHadoopUtil. I built TP3.1 from https://github.com/vtslab/incubator-tinkerpop/tree/3.1.0-hdp-2.3.2.0-2950 and didn't see jars contains YarnSparkHadoopUtil either. Did I miss something?

HadoopMarc

unread,
Dec 8, 2015, 3:10:32 AM12/8/15
to Gremlin-users, d...@tinkerpop.incubator.apache.org
Hi Cristine,

Did you start the repl with gremlinhdp.sh? It does all the exports needed and sets the CLASSPATH. Check if the configs are right for your cluster.


Marc

Op maandag 7 december 2015 21:30:03 UTC+1 schreef Christine Li:
Reply all
Reply to author
Forward
0 new messages