Spark error: User class threw exception: java.lang.reflect.InvocationTargetException
1,181 views
Skip to first unread message
marco.sa...@gmail.com
Oct 4, 2018, 12:10:30 PM10/4/18
to CDAP User
Hi everyone,
I am facing an error with a very simple pipeline in CDAP using scala Spark code (CDAP 5.0 with MapR 6.0).
The following image shows the pipeline
Here the scala Spark code:
import co.cask.cdap.api.spark._
import org.apache.spark._
import org.slf4j._
import org.apache.spark.sql.SQLContext
class SparkProgram extends SparkMain {
case class MyClass(number: Int)
override def run(implicit sec: SparkExecutionContext): Unit = {
val sc = new SparkContext()
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
val tester1= List(1,1)
val tester1RDD =sc.parallelize(tester1,1)
val tester1DF = tester1RDD.map(a => MyClass(a)).toDF()
tester1DF.write.json("maprfs:///iper/scontrini/staging/test1_write")
System.out.println("CIAO")
}
}
Unfortunately I have the following error from YARN: User class threw exception: java.lang.reflect.InvocationTargetException
The pipeline works if I don't use the case class (I tried with spark-shell and the code works).
import co.cask.cdap.api.spark._
import org.apache.spark._
import org.slf4j._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
class SparkProgram extends SparkMain {
override def run(implicit sec: SparkExecutionContext): Unit = {
val sc = new SparkContext()
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
val testSchema = StructType(Array(StructField("number", IntegerType)))
val tester1= List(1,1)
val tester1RDD =sc.parallelize(tester1,1).map( x=> Row(x))
val tester1DF = sqlContext.createDataFrame(tester1RDD, testSchema)
tester1DF.write.json("maprfs:///iper/scontrini/staging/test1_write")
System.out.println("CIAO")
}
}
Below the complete CDAP pipeline log with the errors.
Any help would be appreciated.
Regards,
Marco
2018-10-04 17:15:13,796 - INFO [program.status:c.c.c.i.a.r.d.DistributedProgramRunner@475] - Starting Workflow Program 'DataPipelineWorkflow' with Arguments [logical.start.time=1538666107568]
2018-10-04 17:15:13,796 - INFO [program.status:c.c.c.i.a.r.d.DistributedProgramRunner@475] - Starting Workflow Program 'DataPipelineWorkflow' with Arguments [logical.start.time=1538666107568]
2018-10-04 17:15:32,800 - INFO [WorkflowDriver:c.c.c.d.SmartWorkflow@440] - Pipeline 'scala_debug_2_MATTEO_v2' is started by user 'cdap' with arguments {logical.start.time=1538666107568}
2018-10-04 17:15:32,829 - INFO [WorkflowDriver:c.c.c.d.SmartWorkflow@474] - Pipeline 'scala_debug_2_MATTEO_v2' running
2018-10-04 17:15:32,873 - INFO [WorkflowDriver:c.c.c.i.a.r.w.WorkflowDriver@611] - Starting workflow execution for 'DataPipelineWorkflow' with Run id '4734c1f3-c7e8-11e8-9822-0200ac1822a0'
2018-10-04 17:15:34,860 - INFO [action-phase-1-0:c.c.c.i.a.r.w.WorkflowDriver@342] - Starting Spark Program 'phase-1' in workflow
2018-10-04 17:15:35,987 - INFO [SparkExecutionService STARTING:c.c.h.NettyHttpService@172] - Starting HTTP Service phase-1-spark-exec-service at address dnode1.finiper.it/172.24.34.160:0
2018-10-04 17:15:36,983 - WARN [spark-submitter-phase-1-57769751-c7e8-11e8-a456-0200ac1822a0:o.a.s.d.y.s.ConfigurableCredentialManager@66] - spark.yarn.security.tokens.hbase.enabled is deprecated, using spark.yarn.security.credentials.hbase.enabled instead
2018-10-04 17:15:36,986 - WARN [spark-submitter-phase-1-57769751-c7e8-11e8-a456-0200ac1822a0:o.a.s.d.y.s.ConfigurableCredentialManager@66] - spark.yarn.security.tokens.hive.enabled is deprecated, using spark.yarn.security.credentials.hive.enabled instead
2018-10-04 17:15:47,539 - INFO [SparkDriverHttpService STARTING:c.c.h.NettyHttpService@172] - Starting HTTP Service phase-1-http-service at address dnode1.finiper.it/172.24.34.160:0
2018-10-04 17:15:47,925 - INFO [SparkDriverService:c.c.c.a.r.s.d.SparkDriverService@98] - SparkDriverService started.
2018-10-04 17:15:47,933 - INFO [Driver:c.c.c.a.r.s.SparkMainWrapper$@77] - Launching user spark class class co.cask.cdap.datapipeline.JavaSparkMainWrapper
2018-10-04 17:15:55,169 - INFO [Driver:o.s.j.u.log@186] - Logging initialized @13889ms
2018-10-04 17:15:55,278 - INFO [Driver:o.s.j.s.Server@327] - jetty-9.2.z-SNAPSHOT
2018-10-04 17:15:55,301 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@7004c919{/jobs,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,302 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3048da9c{/jobs/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,302 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@386752da{/jobs/job,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,302 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@2e5589c0{/jobs/job/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,302 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@784fc112{/stages,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,303 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@ea568bd{/stages/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,303 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@cd61a31{/stages/stage,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,303 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@e4f99e2{/stages/stage/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,303 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@80e8e68{/stages/pool,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,303 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@2dcbef54{/stages/pool/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,304 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@11f07f4e{/storage,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,304 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@6302741b{/storage/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,304 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@1f03cbdd{/storage/rdd,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,304 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@6dc03626{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,305 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@7f583edf{/environment,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,307 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@f8a2f83{/environment/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,307 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@ed6408c{/executors,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,307 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@62cb2676{/executors/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,307 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@5c5b5236{/executors/threadDump,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,308 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@2738809{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,313 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3e9d7aeb{/static,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,314 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@5de13d8{/,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,316 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3122bb7{/api,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,316 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@1dbfc81c{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,316 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@4820a362{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-10-04 17:15:55,322 - INFO [Driver:o.s.j.s.ServerConnector@266] - Started Spark@4ea89d23{HTTP/1.1}{0.0.0.0:33453}
2018-10-04 17:15:55,322 - INFO [Driver:o.s.j.s.Server@379] - Started @14042ms
2018-10-04 17:15:55,759 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@25d624a9{/metrics/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:08,537 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@f2e3a3d{/SQL,null,AVAILABLE,@Spark}
2018-10-04 17:16:08,540 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@74a4d283{/SQL/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:08,541 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@baf31ef{/SQL/execution,null,AVAILABLE,@Spark}
2018-10-04 17:16:08,541 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@261b3303{/SQL/execution/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:08,543 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@4bff0c9c{/static/sql,null,AVAILABLE,@Spark}
2018-10-04 17:16:13,105 - ERROR [Driver:c.m.f.MapRFileSystem@1212] - Failed to delete path maprfs:/iper/scontrini/staging/test1_write/_temporary-a5e03124-f9d3-412e-84d8-b99db023e41d, error: No such file or directory (2)
2018-10-04 17:16:13,210 - INFO [SparkDriverService:c.c.c.a.r.s.d.SparkDriverService@145] - SparkDriverService stopped.
2018-10-04 17:16:13,297 - INFO [Thread-8:o.s.j.s.ServerConnector@306] - Stopped Spark@4ea89d23{HTTP/1.1}{0.0.0.0:0}
2018-10-04 17:16:13,937 - ERROR [spark-submitter-phase-1-57769751-c7e8-11e8-a456-0200ac1822a0:c.m.f.MapRFileSystem@1212] - Failed to delete path maprfs:/user/cdap/.sparkStaging/application_1537108443055_0411, error: No such file or directory (2)
2018-10-04 17:16:13,952 - INFO [SparkExecutionService STOPPING:c.c.h.NettyHttpService@242] - Stopping HTTP Service phase-1-spark-exec-service
2018-10-04 17:16:14,083 - INFO [action-phase-1-0:c.c.c.i.a.r.w.WorkflowDriver@345] - Spark Program 'phase-1' in workflow completed
2018-10-04 17:16:15,773 - INFO [action-phase-2-0:c.c.c.i.a.r.w.WorkflowDriver@342] - Starting Spark Program 'phase-2' in workflow
2018-10-04 17:16:17,107 - INFO [SparkExecutionService STARTING:c.c.h.NettyHttpService@172] - Starting HTTP Service phase-2-spark-exec-service at address dnode1.finiper.it/172.24.34.160:0
2018-10-04 17:16:17,614 - WARN [spark-submitter-phase-2-6fda7cd2-c7e8-11e8-b625-0200ac1822a0:o.a.s.d.y.s.ConfigurableCredentialManager@66] - spark.yarn.security.tokens.hbase.enabled is deprecated, using spark.yarn.security.credentials.hbase.enabled instead
2018-10-04 17:16:17,616 - WARN [spark-submitter-phase-2-6fda7cd2-c7e8-11e8-b625-0200ac1822a0:o.a.s.d.y.s.ConfigurableCredentialManager@66] - spark.yarn.security.tokens.hive.enabled is deprecated, using spark.yarn.security.credentials.hive.enabled instead
2018-10-04 17:16:27,764 - INFO [SparkDriverHttpService STARTING:c.c.h.NettyHttpService@172] - Starting HTTP Service phase-2-http-service at address dnode1.finiper.it/172.24.34.160:0
2018-10-04 17:16:27,953 - INFO [SparkDriverService:c.c.c.a.r.s.d.SparkDriverService@98] - SparkDriverService started.
2018-10-04 17:16:27,966 - INFO [Driver:c.c.c.a.r.s.SparkMainWrapper$@77] - Launching user spark class class co.cask.cdap.etl.spark.batch.BatchSparkPipelineDriver
2018-10-04 17:16:28,453 - INFO [Driver:o.s.j.u.log@186] - Logging initialized @7055ms
2018-10-04 17:16:28,588 - INFO [Driver:o.s.j.s.Server@327] - jetty-9.2.z-SNAPSHOT
2018-10-04 17:16:28,610 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@375f0a31{/jobs,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,611 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@7246eb73{/jobs/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,611 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@5ea228e0{/jobs/job,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,611 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@1172e9ed{/jobs/job/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,611 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@5e337770{/stages,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,612 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@1b6b02b6{/stages/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,612 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@5441dc13{/stages/stage,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,612 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@527a6af7{/stages/stage/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,612 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@17b71f3f{/stages/pool,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,612 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@540ad1cd{/stages/pool/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,614 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@ef12e1b{/storage,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,615 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@f5dc0e0{/storage/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,615 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@20bc68d3{/storage/rdd,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,616 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@22ba41b0{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,616 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@253c320a{/environment,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,616 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@4487ac8d{/environment/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,616 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@73ebe3e{/executors,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,616 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@7779d8a8{/executors/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,617 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3311182f{/executors/threadDump,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,617 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@4a0b2904{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,622 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@38b6a135{/static,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,623 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@1b38213{/,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,624 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3a58226a{/api,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,624 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@29087f2c{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,624 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@243adc31{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-10-04 17:16:28,629 - INFO [Driver:o.s.j.s.ServerConnector@266] - Started Spark@194e42c4{HTTP/1.1}{0.0.0.0:40249}
2018-10-04 17:16:28,632 - INFO [Driver:o.s.j.s.Server@379] - Started @7233ms
2018-10-04 17:16:29,092 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@4904a3c{/metrics/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:41,995 - INFO [SparkDriverService:c.c.c.a.r.s.d.SparkDriverService@145] - SparkDriverService stopped.
2018-10-04 17:16:42,069 - INFO [Thread-8:o.s.j.s.ServerConnector@306] - Stopped Spark@194e42c4{HTTP/1.1}{0.0.0.0:0}
2018-10-04 17:16:42,072 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@243adc31{/stages/stage/kill,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,073 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@29087f2c{/jobs/job/kill,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,074 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@3a58226a{/api,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,076 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@1b38213{/,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,077 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@38b6a135{/static,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,078 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@4a0b2904{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,079 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@3311182f{/executors/threadDump,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,080 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@7779d8a8{/executors/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,080 - INFO [SparkDriverHttpService STOPPING:c.c.h.NettyHttpService@242] - Stopping HTTP Service phase-2-http-service
2018-10-04 17:16:42,081 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@73ebe3e{/executors,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,082 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@4487ac8d{/environment/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,083 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@253c320a{/environment,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,084 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@22ba41b0{/storage/rdd/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,085 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@20bc68d3{/storage/rdd,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,085 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@f5dc0e0{/storage/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,086 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@ef12e1b{/storage,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,087 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@540ad1cd{/stages/pool/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,088 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@17b71f3f{/stages/pool,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,088 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@527a6af7{/stages/stage/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,089 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@5441dc13{/stages/stage,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,090 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@1b6b02b6{/stages/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,091 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@5e337770{/stages,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,091 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@1172e9ed{/jobs/job/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,091 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@5ea228e0{/jobs/job,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,091 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@7246eb73{/jobs/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:42,091 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@375f0a31{/jobs,null,UNAVAILABLE,@Spark}
2018-10-04 17:16:49,439 - INFO [SparkDriverHttpService STARTING:c.c.h.NettyHttpService@172] - Starting HTTP Service phase-2-http-service at address dnode1.finiper.it/172.24.34.160:0
2018-10-04 17:16:49,600 - INFO [SparkDriverService:c.c.c.a.r.s.d.SparkDriverService@98] - SparkDriverService started.
2018-10-04 17:16:49,609 - INFO [Driver:c.c.c.a.r.s.SparkMainWrapper$@77] - Launching user spark class class co.cask.cdap.etl.spark.batch.BatchSparkPipelineDriver
2018-10-04 17:16:50,174 - INFO [Driver:o.s.j.u.log@186] - Logging initialized @6928ms
2018-10-04 17:16:50,291 - INFO [Driver:o.s.j.s.Server@327] - jetty-9.2.z-SNAPSHOT
2018-10-04 17:16:50,318 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@6f49ef86{/jobs,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,318 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@25a99427{/jobs/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,319 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3298fb2a{/jobs/job,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,319 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@6569b33c{/jobs/job/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,319 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@457c55c8{/stages,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,320 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@2c1df9ec{/stages/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,320 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@592916ee{/stages/stage,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,320 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@71f1bea4{/stages/stage/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,320 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@56847747{/stages/pool,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,321 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@159526e2{/stages/pool/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,323 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@41adcd8{/storage,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,323 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@78e24661{/storage/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,323 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@d292875{/storage/rdd,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,324 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3137bd04{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,324 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@63194be6{/environment,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,324 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@922d8aa{/environment/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,324 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@4d0c237b{/executors,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,325 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@41df7843{/executors/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,325 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@68305de5{/executors/threadDump,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,325 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@77483fc1{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,331 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@5bf3cda8{/static,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,332 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@61d4bbf5{/,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,332 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@be11d5f{/api,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,333 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@3de18dd5{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,333 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@78ba994c{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-10-04 17:16:50,339 - INFO [Driver:o.s.j.s.ServerConnector@266] - Started Spark@6269f36f{HTTP/1.1}{0.0.0.0:37131}
2018-10-04 17:16:50,339 - INFO [Driver:o.s.j.s.Server@379] - Started @7093ms
2018-10-04 17:16:50,801 - INFO [Driver:o.s.j.s.h.ContextHandler@744] - Started o.s.j.s.ServletContextHandler@31f36b2a{/metrics/json,null,AVAILABLE,@Spark}
2018-10-04 17:17:04,617 - INFO [SparkDriverService:c.c.c.a.r.s.d.SparkDriverService@145] - SparkDriverService stopped.
2018-10-04 17:17:04,678 - INFO [Thread-8:o.s.j.s.ServerConnector@306] - Stopped Spark@6269f36f{HTTP/1.1}{0.0.0.0:0}
2018-10-04 17:17:04,689 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@78ba994c{/stages/stage/kill,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,689 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@3de18dd5{/jobs/job/kill,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,690 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@be11d5f{/api,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,692 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@61d4bbf5{/,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,692 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@5bf3cda8{/static,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,692 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@77483fc1{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,693 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@68305de5{/executors/threadDump,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,693 - INFO [SparkDriverHttpService STOPPING:c.c.h.NettyHttpService@242] - Stopping HTTP Service phase-2-http-service
2018-10-04 17:17:04,693 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@41df7843{/executors/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,696 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@4d0c237b{/executors,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,696 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@922d8aa{/environment/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,697 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@63194be6{/environment,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,698 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@3137bd04{/storage/rdd/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,699 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@d292875{/storage/rdd,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,699 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@78e24661{/storage/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,700 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@41adcd8{/storage,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,700 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@159526e2{/stages/pool/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,700 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@56847747{/stages/pool,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,701 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@71f1bea4{/stages/stage/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,702 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@592916ee{/stages/stage,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,702 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@2c1df9ec{/stages/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,703 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@457c55c8{/stages,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,703 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@6569b33c{/jobs/job/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,703 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@3298fb2a{/jobs/job,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:04,703 - INFO [Thread-8:o.s.j.s.h.ContextHandler@865] - Stopped o.s.j.s.ServletContextHandler@25a99427{/jobs/json,null,UNAVAILABLE,@Spark}
2018-10-04 17:17:05,832 - ERROR [spark-submitter-phase-2-6fda7cd2-c7e8-11e8-b625-0200ac1822a0:c.m.f.MapRFileSystem@1212] - Failed to delete path maprfs:/user/cdap/.sparkStaging/application_1537108443055_0412, error: No such file or directory (2)
2018-10-04 17:17:05,842 - INFO [SparkExecutionService STOPPING:c.c.h.NettyHttpService@242] - Stopping HTTP Service phase-2-spark-exec-service
2018-10-04 17:17:05,957 - ERROR [SparkRunnerphase-2:c.c.c.i.a.r.ProgramControllerServiceAdapter@97] - Spark Program 'phase-2' failed.
java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Application application_1537108443055_0412 finished with failed status
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na]
at co.cask.cdap.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:347) ~[co.cask.cdap.cdap-spark-core2_2.11-5.0.0.jar:na]
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
at co.cask.cdap.app.runtime.spark.SparkRuntimeService$5$1.run(SparkRuntimeService.java:405) [co.cask.cdap.cdap-spark-core2_2.11-5.0.0.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: org.apache.spark.SparkException: Application application_1537108443055_0412 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167) ~[na:na]
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213) ~[na:na]
at org.apache.spark.deploy.yarn.Client.main(Client.scala) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:733) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:177) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:202) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:116) ~[na:na]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:172) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:54) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:111) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_171]
... 1 common frames omitted
2018-10-04 17:17:05,965 - ERROR [SparkRunnerphase-2:c.c.c.i.a.r.ProgramControllerServiceAdapter@98] - Spark program 'phase-2' failed with error: Application application_1537108443055_0412 finished with failed status. Please check the system logs for more details.
org.apache.spark.SparkException: Application application_1537108443055_0412 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167) ~[na:na]
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213) ~[na:na]
at org.apache.spark.deploy.yarn.Client.main(Client.scala) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:733) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:177) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:202) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:116) ~[na:na]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:172) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:54) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:111) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
2018-10-04 17:17:05,986 - ERROR [WorkflowDriver:c.c.c.d.SmartWorkflow@529] - Pipeline 'scala_debug_2_MATTEO_v2' failed.
2018-10-04 17:17:06,022 - ERROR [WorkflowDriver:c.c.c.i.a.r.w.WorkflowProgramController@89] - Workflow service 'workflow.default.scala_debug_2_MATTEO_v2.DataPipelineWorkflow.4734c1f3-c7e8-11e8-9822-0200ac1822a0' failed.
java.lang.RuntimeException: org.apache.spark.SparkException: Application application_1537108443055_0412 finished with failed status
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na]
at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver.executeAll(WorkflowDriver.java:638) ~[na:na]
at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver.run(WorkflowDriver.java:614) ~[na:na]
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: org.apache.spark.SparkException: Application application_1537108443055_0412 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167) ~[na:na]
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213) ~[na:na]
at org.apache.spark.deploy.yarn.Client.main(Client.scala) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:733) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:177) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:202) ~[na:na]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:116) ~[na:na]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:172) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:54) ~[na:na]
at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:111) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_171]
... 1 common frames omittedmarco.sa...@gmail.com
Oct 8, 2018, 9:10:01 AM10/8/18
to CDAP User
Hi everyone,
I investigated and the problem is the function toDF applied to a RDD with a case class.
val tester1DF = tester1RDD.map(a => MyClass(a)).toDF()
Following the exception captured with a try/catch
| User class threw exception: java.lang.reflect.InvocationTargetException |
It is something related to case class: any suggestions?
Regards,
Marco
Albert Shau
Oct 8, 2018, 12:35:49 PM10/8/18
to cdap...@googlegroups.com
Hi Marco,
I don't see from the logs where that InvocationTargetException is thrown. Can you attach the stack trace where you see that? I'm guessing you're running into something like https://issues.cask.co/browse/CDAP-12941.
In general, I would recommend only using these dynamic spark plugins for quickly trying things out and validating proofs of concept. For actual production quality code, I recommend write an actual plugin, where the code can be properly unit tested and version controlled.
Regards,
Albert
--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/c50d0a9b-662b-48e7-a678-1430367aa84c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
marco.sa...@gmail.com
Oct 9, 2018, 6:44:32 AM10/9/18
to CDAP User
Hi Albert,
thanks for the support. We can continue on thread "java.lang.ClassNotFoundException: Class not found in all delegated ClassLoaders" in which I think there is the same issue.
Marco
...
Reply all
Reply to author
Forward
0 new messages