Hadoop configurations to Spark pods for pipeline runs

[Tuomas D]

Hi,

I'm working in moving my system to Kubernetes based setup and having some challenges with pipeline steps running in Spark Kubernetes driver and executor pods wrt. accessing hadoop configurations.

Is there existing way, or guidance on how others have solved this, to pass properties, such as fs.defaultFS, from CDAP config YAML and have those available in Hadoop Configuration instance in Spark driver and executor pods?

In workflow pods I can see /etc/hadoop/conf/core-site.xml populated with properties having "hadoop:" prefix in CDAP config in instance YAML that I'm using to install CDAP.

There is no similar file created in Spark driver or excutor pods, I have noticed that hConf.xml and cConf.xml do get installed on those pods but automatically they are not picked by default Hadoop Configuration instances as they look at specific locations and files (like core-site.xml).

Help appreciated in finding way to have configs populated in pods (workflow, spark driver and executor, etc) to have limited need to change code (e.g. FileSystem.get(Configuration) with Configuration loaded by pipelines using default locations).

Thanks, Tuomas

[Prashant Jaikumar]

Hi Tuomas,

fs.defaultFS is set by the cdap operator to the value of the locationURI config in the CDAP yaml. 

https://github.com/cdapio/cdap-operator/blob/develop/templates/core-site.xml#L20C34-L20C34

https://cdap.atlassian.net/wiki/spaces/DOCS/pages/911179793/Installing+CDAP+on+Kubernetes#Deploy-CDAP

CDAP, hadoop configs are localized onto Spark driver and executor pods as cConf.xml, hConf.xml -- these configs will be applied on Spark pods.

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/cdee4c27-2609-460c-8c0d-8c00f24c41ean%40googlegroups.com.