Spark Action issue - unable to run spark action on CDAP distributed mode

[shyam....@gmail.com]

Hey Guys,

I have a simple CDAP application having only one spark program, currently the implementation is empty. However, I am seeing issues while running, for your reference I have copied entire log here. I am using CDAP 3.4.2 in distributed mode.

My project dependencies are -

<dependency>
  <groupId>co.cask.cdap</groupId>
  <artifactId>cdap-api</artifactId>
  <version>${cdap.version}</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>co.cask.cdap</groupId>
  <artifactId>cdap-unit-test</artifactId>
  <version>${cdap.version}</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>${slf4j.version}</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>${junit.version}</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>${guava.version}</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.10</artifactId>
  <version>1.5.1</version>
  <scope>provided</scope>
  <exclusions>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>co.cask.cdap</groupId>
  <artifactId>cdap-api-spark</artifactId>
  <version>${cdap.version}</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>com.oracle</groupId>
  <artifactId>ojdbc6</artifactId>
  <version>11.2.0.1.0</version>
</dependency>

2016-06-28 15:36:35,758 - INFO [Spark - TestAbstractSpark:c.c.t.d.AbstractClientProvider@108] - Service discovered at chikpet.amers1e.cis.trcloud:15165

2016-06-28 15:36:35,782 - INFO [Spark - TestAbstractSpark:c.c.t.d.AbstractClientProvider@116] - Attempting to connect to tx service at chikpet.amers1e.cis.trcloud:15165 with timeout 30000 ms.

2016-06-28 15:36:35,796 - INFO [Spark - TestAbstractSpark:c.c.t.d.AbstractClientProvider@130] - Connected to tx service at chikpet.amers1e.cis.trcloud:15165

2016-06-28 15:36:42,785 - INFO [NettyHttpService STARTING:c.c.h.NettyHttpService@258] - Starting service on address doddapet.amers1e.cis.trcloud/10.204.191.249:0...

2016-06-28 15:36:42,863 - INFO [NettyHttpService STARTING:c.c.h.NettyHttpService@263] - Started service on address /10.204.191.249:40401

2016-06-28 15:37:19,204 - INFO [NettyHttpService STOPPING:c.c.h.NettyHttpService@275] - Stopping service on address doddapet.amers1e.cis.trcloud/10.204.191.249:40401...

2016-06-28 15:37:19,221 - INFO [NettyHttpService STOPPING:c.c.h.NettyHttpService@289] - Done stopping service on address doddapet.amers1e.cis.trcloud/10.204.191.249:40401

2016-06-28 15:37:19,406 - ERROR [Spark - TestAbstractSpark:c.c.c.i.a.r.ProgramControllerServiceAdapter@81] - Program terminated with exception java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Application application_1467010999268_0031 finished with failed status at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:246) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.app.runtime.spark.SparkRuntimeService$3$1.run(SparkRuntimeService.java:300) [co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: org.apache.spark.SparkException: Application application_1467010999268_0031 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:940) ~[spark-assembly.jar:na] at org.apache.spark.deploy.yarn.Client$.main(Client.scala:986) ~[spark-assembly.jar:na] at org.apache.spark.deploy.yarn.Client.main(Client.scala) ~[spark-assembly.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_67] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_67] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67] at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-assembly.jar:na] at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:172) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:56) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:114) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_67] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_67] ... 1 common frames omitted

2016-06-28 15:37:19,418 - ERROR [pcontroller-program:shyama.SparkSqlTestApplication.spark.TestAbstractSpark-0bd348ea-3d46-11e6-aedf-fa163ecf4df4:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@337] - Program runner error out. java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Application application_1467010999268_0031 finished with failed status at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:246) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.app.runtime.spark.SparkRuntimeService$3$1.run(SparkRuntimeService.java:300) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: org.apache.spark.SparkException: Application application_1467010999268_0031 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:940) ~[spark-assembly.jar:na] at org.apache.spark.deploy.yarn.Client$.main(Client.scala:986) ~[spark-assembly.jar:na] at org.apache.spark.deploy.yarn.Client.main(Client.scala) ~[spark-assembly.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_67] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_67] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67] at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) ~[spark-assembly.jar:na] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-assembly.jar:na] at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:172) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:56) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:114) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_67] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67] ... 1 common frames omitted

[Terence Yim]

Hi,

Have you tried new a SparkContext object instead of leaving the method empty?

Terence

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/f1fcc797-763a-407d-be1e-7dafca72ddd8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[shyam....@gmail.com]

Hey Terrence,

I do have empty method, here is my code -

class SparkSqlAction extends SparkMain {
  val LOG: Logger = LoggerFactory.getLogger(classOf[SparkSqlAction])

  override def run(implicit sec: SparkExecutionContext): Unit = {
    LOG.info(">>>>>>>>>>>> INSIDE SPARK_ACTION_RUN <<<<<<<<<<<<<<<<<")

  }
}

One more question, how do I get sparkContext inside run method. I want to have SqlContext and then I want to do some dataframe operations.

[Terence Yim]

Hi,

You simply new it like a regular spark program

val sc = new SparkContext val sql = new SQLContext(sc)

See this as a complete example.

https://github.com/caskdata/cdap/blob/develop/cdap-unit-test/src/test/scala/co/cask/cdap/spark/app/DatasetSQLSpark.scala#L33

Terence

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/f2338ad8-4c39-4512-82bd-311943b8da96%40googlegroups.com.

[shyam....@gmail.com]

Hey Terrence,

Thanks, I am still unable to run spark program, I am getting the same issue.

[shyam....@gmail.com]

Hey Terrence, 

Here is my App -

public class SparkSqlTestApplication extends AbstractApplication {
    @Override
    public void configure() {
        setName("SparkSqlTestApplication");
        setDescription("SparkSqlTestApplication");
        addSpark(new TestAbstractSpark());
    }
}

public class TestAbstractSpark extends AbstractSpark {

    @Override
    protected void configure() {
        setName("TestAbstractSpark");
        setMainClass(SparkSqlAction.class);
        setExecutorResources(new Resources(1024, 1));
    }
}

class SparkSqlAction extends SparkMain {
  val LOG: Logger = LoggerFactory.getLogger(classOf[SparkSqlAction])

  override def run(implicit sec: SparkExecutionContext): Unit = {
    LOG.info(">>>>>>>>>>>> INSIDE SPARK_ACTION_RUN <<<<<<<<<<<<<<<<<")

  }
}

The control is not reaching SparkAction, it fails before with the exception information that I copied in my previous response.

[Terence Yim]

Hi,

Seems like the spark submit failed on the client side. Can you help check if you see the Spark program get submitted to YARN or not? If it is submitted, you’ll see two lines like this in the YARN RM UI:

application_1452044814841_0465

cdap

PageRankSpark

SPARK

root.trex

Mon May 23 13:31:18 -0700 2016

Mon May 23 13:32:05 -0700 2016

FINISHED

SUCCEEDED

-1

N/A

N/A

0.0

0.0

History

N/A

application_1452044814841_0464

cdap

workflow.trex.SparkPageRank.PageRankWorkflow

YARN

root.trex

Mon May 23 13:30:38 -0700 2016

Mon May 23 13:33:40 -0700 2016

FINISHED

SUCCEEDED

-1

N/A

N/A

0.0

0.0

History

N/A

Also, the logs you get is not the complete log. Can you help to get more log by just keep scrolling up in the UI? Also, is the Spark version also 1.5.1 on your cluster?

Terence

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/d5861a7d-fece-4142-a113-fe4ffc81fdb2%40googlegroups.com.

[shyam joshi]

Hey Terence, I did check that, right now I am seeing nothing. Job is not getting submitted to YARN. Any thoughts why? CDAP node has the gateway to YARN. Map reduce is getting submitted but not spark.