How to use spylon-kernel (scala) in Jupyter Notebook of dataproc to connect to Hive metastore?

[Desiree Zhu]

I install Jupyter Notebook in my dataproc, but how can I make to connect to the Hive in another dataproc? I have put hive-site.xml in /usr/lib/spark/conf therefore spark-sql could connect there but not work with Jupyter Notebook, seem like the spark.sql(show databases) connect to local metastore instead of Hive one. Thanks!

[karth...@google.com]

Looking in /etc/spark/conf/spark-env.sh, Spark is already configured to read Hive configuration through /etc/hive/conf/*: `SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/etc/hive/conf"`

You can configure Hive properties in Dataproc through --properties: https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/cluster-properties.

So I suspect something like `gcloud dataproc clusters create --optional-components=ANACONDA,JUPYTER --properties hive:some.hive.property=some.value,hive:another.hive.property=another.value` will work.

[Desiree Zhu]

Thank you so much! I put the hive-site.xml under /etc/hive/conf. Now the Jupyter Notebook kernal 'PySpark' could connect to the hive, but 'spylon-kernal' not, there must be some configure for the kernal...but could not find it. 

[hg...@google.com]

Tested as much as I understand the scenarios here, spylon-kernal connects with remove Hive metastore and conduct read/write successfully after config spark-sql with spark.sql.catalogImplementation=hive by setting following properties when creating cluster:

--properties hive:hive.metastore.warehouse.dir=gs://<warehouse-path>,hive:hive.metastore.uris=thrift://<remote-hive-host>:9083,spark:spark.sql.catalogImplementation=hive