CDAP compatibility with spark 2.3.0

82 views
Skip to first unread message

ankur.gu...@gmail.com

unread,
Jun 27, 2018, 2:20:50 PM6/27/18
to CDAP User
We are trying to run realtime pipelines using CDAP. All pipelines are failing when it is being submitted to spark. Following is the exception seen

2018-06-27 09:37:46,239 - ERROR [main:c.c.c.a.r.s.d.SparkContainerLauncher@145] - Exception raised when calling org.apache.spark.deploy.yarn.ApplicationMaster.main(String[]) method

java.lang.reflect.InvocationTargetException: null

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]

        at co.cask.cdap.app.runtime.spark.distributed.SparkContainerLauncher.launch(SparkContainerLauncher.java:114) ~[co.cask.cdap.cdap-spark-core2_2.11-4.3.4.jar:na]

        at org.apache.spark.deploy.yarn.ApplicationMaster.main(Unknown Source) [cdap-spark-launcher.jar:na]

Caused by: java.lang.IllegalStateException: SparkContextProvider.getSparkContext should only be called in Spark executor process.

        at com.google.common.base.Preconditions.checkState(Preconditions.java:149) ~[com.google.guava.guava-13.0.1.jar:na]

        at co.cask.cdap.app.runtime.spark.SparkRuntimeContextProvider.createIfNotExists(SparkRuntimeContextProvider.java:155) ~[co.cask.cdap.cdap-spark-core2_2.11-4.3.4.jar:na]

        at co.cask.cdap.app.runtime.spark.SparkRuntimeContextProvider.get(SparkRuntimeContextProvider.java:136) ~[co.cask.cdap.cdap-spark-core2_2.11-4.3.4.jar:na]

        ... 6 common frames omitted



Matthew Wuenschel

unread,
Jun 28, 2018, 4:08:07 PM6/28/18
to cdap...@googlegroups.com
Hi,

Spark 2.3.0 isn't supported at this time. We plan to support spark 2.3 in a future release. 

Thanks,
Matt

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/083c262e-3282-498a-9d25-e49b6d2ffc97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ankur Gupta

unread,
Jun 29, 2018, 12:59:41 AM6/29/18
to cdap...@googlegroups.com
Thanks Matt

On Fri, 29 Jun 2018 at 01:38, Matthew Wuenschel <ma...@cask.co> wrote:
Hi,

Spark 2.3.0 isn't supported at this time. We plan to support spark 2.3 in a future release. 

Thanks,
Matt

On Wed, Jun 27, 2018 at 11:20 AM, <ankur.gu...@gmail.com> wrote:
We are trying to run realtime pipelines using CDAP. All pipelines are failing when it is being submitted to spark. Following is the exception seen

2018-06-27 09:37:46,239 - ERROR [main:c.c.c.a.r.s.d.SparkContainerLauncher@145] - Exception raised when calling org.apache.spark.deploy.yarn.ApplicationMaster.main(String[]) method

java.lang.reflect.InvocationTargetException: null

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]

        at co.cask.cdap.app.runtime.spark.distributed.SparkContainerLauncher.launch(SparkContainerLauncher.java:114) ~[co.cask.cdap.cdap-spark-core2_2.11-4.3.4.jar:na]

        at org.apache.spark.deploy.yarn.ApplicationMaster.main(Unknown Source) [cdap-spark-launcher.jar:na]

Caused by: java.lang.IllegalStateException: SparkContextProvider.getSparkContext should only be called in Spark executor process.

        at com.google.common.base.Preconditions.checkState(Preconditions.java:149) ~[com.google.guava.guava-13.0.1.jar:na]

        at co.cask.cdap.app.runtime.spark.SparkRuntimeContextProvider.createIfNotExists(SparkRuntimeContextProvider.java:155) ~[co.cask.cdap.cdap-spark-core2_2.11-4.3.4.jar:na]

        at co.cask.cdap.app.runtime.spark.SparkRuntimeContextProvider.get(SparkRuntimeContextProvider.java:136) ~[co.cask.cdap.cdap-spark-core2_2.11-4.3.4.jar:na]

        ... 6 common frames omitted



--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/083c262e-3282-498a-9d25-e49b6d2ffc97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "CDAP User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cdap-user/DuIt8S73PHM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cdap-user+...@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Sent from My Iphone

yes...@gmail.com

unread,
Dec 14, 2018, 6:40:58 AM12/14/18
to CDAP User
What if you add system.env Option "-DSPARK_YARN_MODE=true" when you run CDAP deamons.

I found this line at CDAP source.

reconditions.checkState(!contextConfig.isLocal() && Boolean.parseBoolean(System.getenv("SPARK_YARN_MODE")), 
                               "SparkContextProvider.getSparkContext should only be called in Spark executor process."); 

This line means contextConfig.isLocal() be true and SPARK_YARN_MODE system env should be true.





2018년 6월 28일 목요일 오전 3시 20분 50초 UTC+9, Ankur Gupta 님의 말:
Reply all
Reply to author
Forward
0 new messages