Spark Version pecification for Pipelines

[Sagar Batchu]

Hi Team, 

Is it possible to set a spark version (2 or 3) in the custom Engine config for a deployed pipeline? 

--

Sagar Batchu 

LiveRamp | Engineering

m: +44 7787666112

[Albert Shau]

Hi Sagar,

There isn't a way to do this, as the version of Spark on the cluster will be the one that gets used. Are you seeing errors with certain Spark versions?

Regards,

Albert

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAEq4jBgHX5WruLkgyu6i%2BVUo1f%2BWVM%2BfxfkP73vY%3DKpeDdUMgg%40mail.gmail.com.

[Sagar Batchu]

(thanks that make sense, it's dependent on the Dataproc cluster being used)

Specifically, we are seeing some failures with using the Dynamic PySpark plugin (v2.3.3) on the 6.5.0-SNAPSHOT bleeding edge. 

Originally we were seeing 

java.lang.Exception: java.lang.NoSuchMethodError: scala.collection.JavaConversions$.deprecated$u0020propertiesAsScalaMap(Ljava/util/Properties;)Lscala/collection/mutable/Map;
at io.cdap.cdap.app.runtime.spark.SparkRuntimeService.startUp(SparkRuntimeService.java:324) ~[io.cdap.cdap.cdap-spark-core3_2.12-6.5.0-SNAPSHOT.jar:na]

causing the pipeline to fail. We upgraded the Dataproc cluster being spun up is using dataproc image "1-4-55-debian10" with Spark3. Now we are seeing an issue with SPARK_HOME not being found (full log attached)

2021-06-07 18:58:08,071 - ERROR [Driver:o.a.s.d.y.ApplicationMaster@94] - User class threw exception: java.lang.RuntimeException: org.apache.spark.SparkUserAppException: User application exited with 1
java.lang.RuntimeException: org.apache.spark.SparkUserAppException: User application exited with 1
	at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) ~[na:3.1.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_282]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_282]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_282]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_282]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732) ~[spark-yarn_2.12-3.1.1.jar:3.1.1]
Caused by: org.apache.spark.SparkUserAppException: User application exited with 1
	at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:103) ~[na:3.1.1]
	... 6 common frames omitted
2021-06-07 18:58:08,088 - ERROR [main:o.a.s.d.y.ApplicationMaster@94] - Uncaught exception: 
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) ~[spark-core_2.12-3.1.1.jar:na]
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:504) [spark-yarn_2.12-3.1.1.jar:3.1.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:268) [spark-yarn_2.12-3.1.1.jar:3.1.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:899) [spark-yarn_2.12-3.1.1.jar:3.1.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:898) [spark-yarn_2.12-3.1.1.jar:3.1.1]
	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_282]
	at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_282]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) [hadoop-common-3.2.2.jar:na]
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:898) [spark-yarn_2.12-3.1.1.jar:3.1.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) [spark-yarn_2.12-3.1.1.jar:3.1.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_282]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_282]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_282]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_282]
	at io.cdap.cdap.app.runtime.spark.distributed.SparkContainerLauncher.launch(SparkContainerLauncher.java:149) [io.cdap.cdap.cdap-spark-core3_2.12-6.5.0-SNAPSHOT.jar:na]
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(Unknown Source) [spark-yarn_2.12-3.1.1.jar:3.1.1]
java.lang.RuntimeException: org.apache.spark.SparkUserAppException: User application exited with 1
	at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) ~[na:3.1.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_282]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_282]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_282]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_282]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732) ~[spark-yarn_2.12-3.1.1.jar:3.1.1]
Caused by: org.apache.spark.SparkUserAppException: User application exited with 1
	at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:103) ~[na:3.1.1]
	... 6 common frames omitted
2021-06-07 18:58:08,093 - DEBUG [main:i.c.c.l.a.LogAppenderInitializer@137] - Stopping log appender TMSLogAppender
2021-06-07 18:58:23,850 - INFO  [main:i.c.c.a.r.s.d.SparkContainerLauncher@-2] - Launch main class org.apache.spark.deploy.yarn.ApplicationMaster.main([--class, org.apache.spark.deploy.PythonRunner, --primary-py-file, default.auchan_meas_test001.phase-1.37a97f51-c7c2-11eb-ae28-42010a0d302f.py, --properties-file, /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1623092011571_0002/container_1623092011571_0002_02_000001/__spark_conf__/__spark_conf__.properties, --dist-cache-conf, /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1623092011571_0002/container_1623092011571_0002_02_000001/__spark_conf__/__spark_dist_cache__.properties])
2021-06-07 18:58:26,213 - INFO  [SparkDriverHttpService STARTING:i.c.h.NettyHttpService@176] - Starting HTTP Service phase-1-http-service at address cdap-auchanmea-6fd1ad59-c7c1-11eb-babd-4e7e9f50e37f-w-0.c.lranalytics-eu-2912.internal/10.13.48.51:0
2021-06-07 18:58:26,443 - INFO  [SparkDriverService:i.c.c.a.r.s.d.SparkDriverService@102] - SparkDriverService started.
2021-06-07 18:58:33,794 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] - Traceback (most recent call last):
2021-06-07 18:58:33,795 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -   File "default.auchan_meas_test001.phase-1.37a97f51-c7c2-11eb-ae28-42010a0d302f.py", line 3, in <module>
2021-06-07 18:58:33,795 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -     import pyspark
2021-06-07 18:58:33,796 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -   File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1623092011571_0002/container_1623092011571_0002_02_000001/pyspark.zip/pyspark/__init__.py", line 53, in <module>
2021-06-07 18:58:33,796 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -   File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1623092011571_0002/container_1623092011571_0002_02_000001/pyspark.zip/pyspark/rdd.py", line 34, in <module>
2021-06-07 18:58:33,797 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -   File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1623092011571_0002/container_1623092011571_0002_02_000001/pyspark.zip/pyspark/java_gateway.py", line 31, in <module>
2021-06-07 18:58:33,797 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -   File "/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1623092011571_0002/container_1623092011571_0002_02_000001/pyspark.zip/pyspark/find_spark_home.py", line 68
2021-06-07 18:58:33,797 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -     print("Could not find valid SPARK_HOME while searching {0}".format(paths), file=sys.stderr)
2021-06-07 18:58:33,798 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] -                                                                                    ^
2021-06-07 18:58:33,798 - INFO  [redirect output:i.c.c.d.ExternalSparkProgram@133] - SyntaxError: invalid syntax
2021-06-07 18:58:33,910 - INFO  [SparkDriverService:i.c.c.a.r.s.d.SparkDriverService@150] - SparkDriverService stopped.
2021-06-07 18:58:33,943 - INFO  [SparkDriverHttpService STOPPING:i.c.h.NettyHttpService@253] - Stopping HTTP Service phase-1-http-service

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAOtBAghMyDwstnYgNAPstRU-voRK5XzH5wu_S2gztzPcfTsaOQ%40mail.gmail.com.

[Sagar Batchu]

Is the `SPARK_HOME` not found something we could solve by setting an environmental variable through a Dataproc initialization action? 

[Vitalii Tymchyshyn]

Hi, Sagar.

I does not looks like it's SPARK_HOME not found.

It looks like pyspark.zip is expecting pyhton 3 and it's run with python 2, so you see "SyntaxError: invalid syntax" for this line.

I will looks into it, but it seems that https://cdap.atlassian.net/browse/CDAP-17892 should be fixed for 6.5 to resolve this.