hive.aux.jars.path

[David Engel]

Some of our analysts have expressed the desire to use superset without
SQL. To make everything they want to do possible, I need to add
virtual columns to some tables using our UDFs. Unfortunately, there
doesn't seem to be any way to force superset to automatically add the
appropriate jars when a Hive conection is made.

Because of this, I'm now attempting to specify a jar in
hive.aux.jars.path to make Hive always load it at startup. This too
doesn't seem to work. This happens whether I put the jar in a locally
mounted directory accessible to all pods or in HDFS. Hiveserver2
doesn't start mr3master because it apparently wants the jar to also be
listed in mr3.cluster.additional.classpath. Please see the two
attached logs for the exact errors.

David
--
David Engel
da...@istwok.net

[Sungwoo Park]

In the current implementation, Hive-MR3-K8s does not use hive.aux.jars.path (disallowed) and hive.mr3.aux.jars (ignored), and thus MR3 master does not take additional local resources. This is not a bug, but just a design decision. It requires an extension of MR3 to be able to take additional local resources.

On Kubernetes, in order to load a Jar file automatically, one can just build a new Docker image and place the Jar file in one of the following paths:

MR3_ADD_CLASSPATH_OPTS="-Dmr3.cluster.additional.classpath=$REMOTE_BASE_DIR/mr3/mr3lib/*:$REMOTE_BASE_DIR/hive/apache-hive/lib/*:$REMOTE_BASE_DIR/tez/tezjar/*:$REMOTE_BASE_DIR/tez/tezjar/lib/*"

(On Kubernetes,  mr3.cluster.additional.classpath is hard-coded.)

Sungwoo