Spark-console with YARN hangs during startup

[Sarah Gerweck]

I've started playing around with Spark, Shark and YARN recently. I'm able to get the π calculator example to run correctly using YARN, but when I try to run shark or spark-console, it always hangs before I get to an actual prompt—right after Adding Shutdown Hook. 

I built with both SPARK_YARN & SHARK_YARN set and I'm using MASTER=yarn-standalone. I'm building against Hadoop 0.23.9.

I've attached by my startup logs and my stack dumps. 

Thanks for any help in advance,

  Sarah Gerweck

[Matei Zaharia]

Hi Sarah,

The Spark interactive shell can't currently run on YARN, because our current YARN mode requires the driver program to be in a container on the cluster. There's a PR for it here: https://github.com/mesos/spark/pull/868 that we'll probably merge soon.

Similar for shark -- bin/shark won't work, but one thing you could do is run the Shark server (shark --server) in a container. I believe this PR made it possible but I haven't tried it myself: https://github.com/amplab/shark/pull/164.

Matei

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
<hang.jstack><hang.log>

[Sarah Gerweck]

I want to offer you a (belated) thanks for your help on this. I was eventually able to get everything working properly by cutting a build of Spark with that pull request merged in.

Thanks,

  Sarah

You received this message because you are subscribed to a topic in the Google Groups "Spark Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spark-users/NY9WJEUj9v0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spark-users...@googlegroups.com.