Connecting to External Hive Metastore within Serverless Spark Jobs

[Deepanshu Bhateja]

Hello Everyone,

I hope this message finds you well. I find myself in a situation where I need to connect to external Hive-Metastore(Cloud SQL) through Serverless spark jobs. 

I read somewhere that HMS is not bundled with Dataproc Serverless and rather use " 1-node Dataproc 2.x cluster to act as a Thrift proxy for Serverless Spark jobs".

Can anyone provide clear understanding of the above. Any documentation or resources that delves into the details of configuration options that are needed to be modified while creating a cluster to act as thrift proxy.

Thanks for your time and assistance, I look forward to hearing from you soon.

Best Regards
Deepanshu

[amancuso...@gmail.com]

Generally, see the gcloud example: https://cloud.google.com/dataproc-serverless/docs/quickstarts/spark-batch#dataproc_serverless_create_batch_workload-gcloud.