NoClassDefFoundError for jar with classpath set in /etc/cdap/conf.mycdap/cdap-site.xml

[nico.pap...@gmail.com]

Hello,

I have an application that uses phoenix-spark. I have marked this dependency as provided in the cdap application pom.xml and the jar resides in every machine on the cluster. I have specified the location of the jar in /etc/cdap/conf.mycdap/cdap-site.xml using the app.program.extra.classpath configuration. 

I'm not sure what else to try, hopefully someone can provide some insight. Please let me know if there's any other info I can provide.

Thanks!

Below is the exception:

2016-12-07 22:32:29,980 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,077 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,128 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,255 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,468 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:31,442 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:31,519 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:33,685 - INFO  [main:c.c.c.l.a.LogAppenderInitializer@68] - Initializing log appender KafkaLogAppender
2016-12-07 22:32:33,745 - INFO  [main:c.c.c.c.g.LocationRuntimeModule@104] - HDFS namespace is /sfdc
2016-12-07 22:32:33,807 - WARN  [main:c.c.c.e.c.ExploreFacade@52] - Explore functionality for datasets is disabled. All calls to enable explore will be no-ops
2016-12-07 22:32:33,809 - WARN  [main:c.c.c.e.c.ExploreFacade@52] - Explore functionality for datasets is disabled. All calls to enable explore will be no-ops
2016-12-07 22:32:33,953 - INFO  [NettyHttpService STARTING:c.c.h.NettyHttpService@258] - Starting service on address istgforce1-dnds1-5-sfm.ops.sfdc.net/10.224.111.3:0...
2016-12-07 22:32:34,009 - INFO  [NettyHttpService STARTING:c.c.h.NettyHttpService@263] - Started service on address /10.224.111.3:40987
2016-12-07 22:32:34,102 - WARN  [Driver:o.a.s.SparkConf@71] - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
2016-12-07 22:32:34,677 - INFO  [sparkDriver-akka.actor.default-dispatcher-3:a.e.s.Slf4jLogger@80] - Slf4jLogger started
2016-12-07 22:32:34,738 - INFO  [sparkDriver-akka.actor.default-dispatcher-3:Remoting@74] - Starting remoting
2016-12-07 22:32:34,970 - INFO  [sparkDriver-akka.actor.default-dispatcher-2:Remoting@74] - Remoting started; listening on addresses :[akka.tcp://spark...@10.224.111.3:44118]
2016-12-07 22:32:35,112 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,158 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,508 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,618 - WARN  [Driver:o.a.s.m.MetricsSystem@71] - Using default name DAGScheduler for source because spark.app.id is not set.
2016-12-07 22:32:36,085 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:36,874 - WARN  [ContainerLauncher #1:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:43,892 - INFO  [Driver:c.c.c.s.BigObjectLeadRatingProgram@26] - Running with args [Ljava.lang.String;@2e3ace81
2016-12-07 22:32:44,295 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:46,096 - INFO  [sparkDriver-akka.actor.default-dispatcher-3:a.r.RemoteActorRefProvider$RemotingTerminator@74] - Shutting down remote daemon.
2016-12-07 22:32:46,099 - INFO  [NettyHttpService STOPPING:c.c.h.NettyHttpService@275] - Stopping service on address /10.224.111.3:40987...
2016-12-07 22:32:46,099 - INFO  [sparkDriver-akka.actor.default-dispatcher-3:a.r.RemoteActorRefProvider$RemotingTerminator@74] - Remote daemon shut down; proceeding with flushing remote transports.
2016-12-07 22:32:46,108 - INFO  [NettyHttpService STOPPING:c.c.h.NettyHttpService@289] - Done stopping service on address /10.224.111.3:40987
2016-12-07 22:32:46,121 - ERROR [Driver:o.a.s.d.y.ApplicationMaster@96] - User class threw exception: java.lang.NoClassDefFoundError: org/apache/phoenix/spark/package$
java.lang.NoClassDefFoundError: org/apache/phoenix/spark/package$
	at co.cask.cdap.scala.BigObjectLeadRatingProgram.run(BigObjectLeadRatingProgram.scala:66) ~[program.expanded.jar/:na]
	at co.cask.cdap.app.runtime.spark.SparkMainWrapper$.main(SparkMainWrapper.scala:92) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
	at co.cask.cdap.app.runtime.spark.SparkMainWrapper.main(SparkMainWrapper.scala) ~[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_102]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_102]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:525) ~[spark-assembly-1.5.2-sfdc-1.0.0-hadoop2.6.0-cdh5.4.2-sfdc-1.2.2.jar:1.5.2-sfdc-1.0.0]
Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.spark.package$
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_102]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_102]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_102]
	... 8 common frames omitted
2016-12-07 22:32:46,124 - INFO  [Thread-6:c.c.c.l.a.LogAppenderInitializer@85] - Stopping log appender KafkaLogAppender
2016-12-07 22:32:46,125 - INFO  [sparkDriver-akka.actor.default-dispatcher-3:a.r.RemoteActorRefProvider$RemotingTerminator@74] - Remoting shut down.

[Sreevatsan Raman]

Hey Nico,

Could you try this suggestion documented in the JIRA: https://issues.cask.co/browse/CDAP-7696

app.program.extra.classpath expects the jars to be available the location specified on all the slave does and not localize these jars in YARN containers.

Thanks,

Sree

On Wed, Dec 7, 2016 at 2:40 PM, <nico.pap...@gmail.com> wrote:

Hello,

I have an application that uses phoenix-spark. I have marked this dependency as provided in the cdap application pom.xml and the jar resides in every machine on the cluster. I have specified the location of the jar in /etc/cdap/conf.mycdap/cdap-site.xml using the app.program.extra.classpath configuration. 

I'm not sure what else to try, hopefully someone can provide some insight. Please let me know if there's any other info I can provide.

Thanks!

Below is the exception:

2016-12-07 22:32:29,980 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,077 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,128 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,255 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:30,468 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:31,442 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:31,519 - WARN  [main:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:33,685 - INFO  [main:c.c.c.l.a.LogAppenderInitializer@68] - Initializing log appender KafkaLogAppender
2016-12-07 22:32:33,745 - INFO  [main:c.c.c.c.g.LocationRuntimeModule@104] - HDFS namespace is /sfdc
2016-12-07 22:32:33,807 - WARN  [main:c.c.c.e.c.ExploreFacade@52] - Explore functionality for datasets is disabled. All calls to enable explore will be no-ops
2016-12-07 22:32:33,809 - WARN  [main:c.c.c.e.c.ExploreFacade@52] - Explore functionality for datasets is disabled. All calls to enable explore will be no-ops
2016-12-07 22:32:33,953 - INFO  [NettyHttpService STARTING:c.c.h.NettyHttpService@258] - Starting service on address istgforce1-dnds1-5-sfm.ops.sfdc.net/10.224.111.3:0...
2016-12-07 22:32:34,009 - INFO  [NettyHttpService STARTING:c.c.h.NettyHttpService@263] - Started service on address /10.224.111.3:40987
2016-12-07 22:32:34,102 - WARN  [Driver:o.a.s.SparkConf@71] - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
2016-12-07 22:32:34,677 - INFO  [sparkDriver-akka.actor.default-dispatcher-3:a.e.s.Slf4jLogger@80] - Slf4jLogger started
2016-12-07 22:32:34,738 - INFO  [sparkDriver-akka.actor.default-dispatcher-3:Remoting@

74] - Starting remoting
2016-12-07 22:32:34,970 - INFO  [sparkDriver-akka.actor.default-dispatcher-2:Remoting@74] - Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.224.111.3:44118]
2016-12-07 22:32:35,112 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,158 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,508 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,618 - WARN  [Driver:o.a.s.m.MetricsSystem@71] - Using default name DAGScheduler for source because spark.app.id is not set.
2016-12-07 22:32:36,085 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:36,874 - WARN  [ContainerLauncher #1:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:43,892 - INFO  [Driver:c.c.c.s.BigObjectLeadRatingProgram@26] - Running with args [Ljava.lang.String;@2e3ace81

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/3061a734-3cdf-4d80-b4c7-15db5fb97d16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[nico.pap...@gmail.com]

I tried putting the jar in /opt/cdap/ext/runtimes/spark/ but I'm getting the same exception.

The jar is available on all nodes in the cluster, in the same location.

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

[Sreevatsan Raman]

Nico:

You don't need to put the jars on all the nodes if you are adding it to /opt/cdap/ext/runtimes/spark/, you only need to add it to master hosts (where ever cdap-master package is installed). After that you will need to restart cdap-master service. 

Thanks,

Sree

] - Starting remoting
2016-12-07 22:32:34,970 - INFO  [sparkDriver-akka.actor.default-dispatcher-2:Remoting@74] - Remoting started; listening on addresses :[akka.tcp://spark...@10.224.111.3:44118]
2016-12-07 22:32:35,112 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,158 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,508 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:35,618 - WARN  [Driver:o.a.s.m.MetricsSystem@71] - Using default name DAGScheduler for source because spark.app.id is not set.
2016-12-07 22:32:36,085 - WARN  [Driver:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:36,874 - WARN  [ContainerLauncher #1:o.a.h.c.Configuration@2670] - mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-12-07 22:32:43,892 - INFO  [Driver:c.c.c.s.BigObjectLeadRatingProgram@26] - Running with args [Ljava.lang.String;@2e3ace81

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/3b5b7314-9bcb-4880-ae33-5260c1eff394%40googlegroups.com.