Schema tool failing when running run-merastore.sh

[Mubashir Chamanshaikh]

Hello,
We are running 2 none k8s cluster (1master and 1 worker).
On master node we have configured MySQL image.
All hibeimr3 code is on master.
When running run-merastore.sh schema tool failed error and communication link error er are facing.
Persisi vlm is getting created in worker node. Also put the connector jar thr.

Thank you in advance.

[Sungwoo Park]

Hello,

If the Metastore log reports something like "communication link failure", it means that Metastore Pod cannot connect to your MySQL server. You can test the connection by logging on to the Metastore Pod and check the connection. Check if HIVE_DATABASE_HOST is set properly in kubernetes/env.sh.

e.g.)

HIVE_DATABASE_HOST=192.168.10.1

e.g.)

HIVE_DATABASE_HOST=mysql.hivemr3.svc.cluster.local

For more details on configuring HiveServer2 and Metastore, please see:

https://mr3docs.datamonad.com/docs/k8s/guide/configure-hiveserver2/ 

https://mr3docs.datamonad.com/docs/k8s/advanced/run-metastore/

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

Thanks for the quick reply,
In enh.sh there are two fields one is
HIVE_DATABASE_HOST AND METASTORE_HOST
Do we assign MySQL pod ip or name to METASTORE HOST? And how about hive database host?

Thank you.

[Sungwoo Park]

If you create a Metastore Pod, it should be (where 9850 is set in yaml/metastore-service.yaml):

HIVE_METASTORE_HOST=hivemr3-metastore-0.metastore.hivemr3.svc.cluster.local

HIVE_METASTORE_PORT=9850

HIVE_DATABASE_HOST specifies the host where the MySQL database for Metastore is running. If MySQL is running on the host node, then it should be set to the address of the host node.

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

We don't have MySQL pod running. We are running MySQL on docker container and client on same local machine.(worker node)
Same IP we are passing to hive-databasehost values but no luck.. we are facing that communication link error..
Will Value for metastore host constant as you define above??

Thanks.

[Sungwoo Park]

Running MySQL in a Docker container is okay. For example, the following quick start guide uses a Docker container running MySQL for the database for Metastore:

https://mr3docs.datamonad.com/docs/quick/k8s/run-k8s-dockerhub/ 

In such a case, HIVE_DATABASE_HOST should be set to the IP address of the host where the Docker container is running. Before running Hive on MR3, you can try to connect to MySQL from the same host, e.g., by executing 'mysql -h 192.168.77.14 -u root -p' on console. In this way, you can check if MySQL is running and reachable. Metastore contacts MySQL running at HIVE_DATABASE_HOST.

Before you run Hive on MR3, make sure that all K8s objects from the previous run do not exist, e.g., by executing:

kubectl -n hivemr3 delete statefulsets --all; kubectl -n hivemr3 delete replicationcontrollers --all; kubectl -n hivemr3 delete pod --all; kubectl -n hivemr3 delete configmap --all; kubectl -n hivemr3 delete svc --all; kubectl -n hivemr3 delete secret --all; kubectl -n hivemr3 delete serviceaccount hive-service-account; kubectl -n hivemr3 delete role --all; kubectl -n hivemr3 delete rolebinding --all; kubectl delete clusterrole node-reader; kubectl delete clusterrolebinding hive-clusterrole-binding; kubectl -n hivemr3 delete persistentvolumeclaims workdir-pvc; kubectl delete persistentvolumes workdir-pv; kubectl -n hivemr3 delete persistentvolumeclaims workdir-pvc-ranger; kubectl delete persistentvolumes workdir-pv-ranger;

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

Thanks for quick reply,
Yeah MySQL is reachable. I am flowing the same link which you sent above.. but that link is for minikube(single node)setup and we have 2 node k8s cluster.
I tried same link on minikube that's working perfectly.
Yes I am deleting old set-up before running run-metastore.sh.
Logs are showing schema tool failed, Communication link error and sometime no route to HOST exception.
Kindly help me on this.

Thanks
Mubashir

[Sungwoo Park]

Could you try ping inside the Metastore Pod to make sure that Pods running on different nodes can reach each other?  It looks to me like there is a problem with network configuration in K8s.

gitlab-runner@orange1:~/mr3-run/kubernetes$ grep HIVE_DATABASE_HOST env.sh

# HIVE_DATABASE_HOST = host for Metastore database 

HIVE_DATABASE_HOST=192.168.10.1

gitlab-runner@orange1:~/mr3-run/kubernetes$ kubectl exec -it -n hivemr3 hivemr3-metastore-0 /bin/bash;

bash-4.2$ ping 192.168.10.1

PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.

64 bytes from 192.168.10.1: icmp_seq=1 ttl=63 time=0.303 ms

64 bytes from 192.168.10.1: icmp_seq=2 ttl=63 time=0.272 ms

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

I tried pinging from metastore pod, it's saying host unreachable. Ping to localhost is happening.
When I am running mysql --user=root --password=passwd --host=127.0.0.1 -e 'show databases;' getting expected output (with hostip and 127.0.0.1 as well)
I tried changing hivedbhost to 127.0.0.1 this time did not get no route to host error.but again when I changed to ip got the same.
When ran with local host got connection refused exception.

Thanks.

[Sungwoo Park]

1.

Inside Metastore Pod, 127.0.0.1 points to the Metastore Pod itself. So, you don't want to set  HIVE_DATABASE_HOST to  127.0.0.1. The simplest solution is to set it to the IP address (not host name) of the host where MySQL is running.

2. 
Setting HIVE_DATABASE_HOST to a host name (instead of an IP address) may require adding the host name to spec/template/spec/hostAliases in metastore.yaml. Please see 'In the spec/template/spec/hostAliases section' of the following page:

https://mr3docs.datamonad.com/docs/k8s/advanced/run-metastore/ 

3. 

Check if network is properly configured in K8s. For example, you can check if a Pod for managing network connection is running okay on each K8s node. In the following example, weave-net-stcd9 is the Pod for managing network connection on a node called blue1.

[2020-08-09 13:41:47] gitlab-runner@blue0:~$ kubectl describe node blue1 | grep kube-system

  kube-system                 kube-proxy-vfbcj             0 (0%)        0 (0%)      0 (0%)           0 (0%)         17d

  kube-system                 weave-net-stcd9              20m (0%)      0 (0%)      0 (0%)           0 (0%)         17d

If none of the above works, could you attach part of the Metastore log?

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

Thanks alot, issue was with network part.
Now I am getting below error when starting metastore pod(run-metastore.sh)
Error:-version information not found in metastore.
data nucleus metadata has jdbc-type optional at this is not valid.

Thanks

[Mubashir Chamanshaikh]

Thanks, that got resolved.
Thanks for your support.

[Sungwoo Park]

Did you fix 'version information not found in metastore'  by initializing Metastore (with "--init-schema")? (I would like to leave a record of how this problem was fixed.)

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

Correct! I fixed it by adding init-schema in metastore.yaml

Thanks 😊

[Mubashir Chamanshaikh]

Hello,
I am able to run metastore pod successfully.
Now I am facing issue in staring hiveserrver2 pod.
ERROR: sockettimeoutexception: connect time out.
ERROR: Failed to initialize config configmap mr3conf-master..check already exists.

We have only one data node. With 8GB memory and 4 vcore

Thanks.

[Sungwoo Park]

sockettimeoutexception: connect time out 

ERROR: Failed to initialize config configmap mr3conf-master..check already exists.

It looks like this is a configuration problem, e.g., the K8s cluster does not have room for HiveServer2 Pod. Could you check the progress of individual steps when executing run-hive.sh, and find out which step produces the first error?

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

First error:- failed to start mr3 session
failed to initialise configmap mr3configmap-master master check if it is already exist
Next
Sockettimeoutexception connect timed out

Thanks.

[Sungwoo Park]

For some reason, HiveServer2 Pod cannot create mr3conf-configmap-master ConfigMap.

The best case is that  mr3conf-configmap-master ConfigMap already exists in the namespace hivemr3. In this case, delete the ConfigMap and restart. In fact, you would better delete all objects in the namespace hivemr3, and start over.

If this is not the case, MR3 does not report why it fails to create the ConfigMap because it does not know why. Then, unfortunately, the user should find out why the HiveServer2 Pod fails to create a new ConfigMap.

Please let me know if you make progress on this problelm.

Cheers,

--- Sungwoo

[Mubashir Chamanshaikh]

Yes we have deleted the namespace and all the object inside it.
But the configmap is getting created when we run metastore.sh.
and when we run Hive.sh it is saying it is already exist like namespace already exist configmap already exist and the other cluster role already exist pv etc.

Thanks

[Sungwoo Park]

mr3conf-configmap-master ConfigMap is created by HiveServer2 Pod at runtime, and it is not created by the script run-metastore.sh or run-hive.sh. 

Could you get the full stack trace when the Exception is raised (to make sure that it is not an IOException)?

Cheers,

--- Sungwoo

[Ahmed]

 INFO [main] AMReplicationControllerContainerExtender: Trying to create ConfigMap mr3conf-configmap-master

2020-08-10T13:02:37,538 ERROR [main] session.MR3Session: Failed to start MR3 Session

com.datamonad.mr3.api.common.MR3Exception: Failed to initialize ConfigMap mr3conf-configmap-master - check if it already exists

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMK8sClient.start(AMK8sClient.scala:55) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3ClientBase.start(MR3ClientBase.scala:52) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.start(MR3SessionClientImpl.scala:70) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.start(HiveMR3ClientImpl.java:80) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:134) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.createSession(MR3SessionManagerImpl.java:393) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.setup(MR3SessionManagerImpl.java:162) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:400) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:942) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:132) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1208) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1052) ~[hive-service-3.1.2.jar:3.1.2]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]

        at org.apache.hadoop.util.RunJar.run(RunJar.java:318) ~[hadoop-common-3.1.2.jar:?]

        at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.2.jar:?]

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [ConfigMap]  with name: [mr3conf-configmap-master]  in namespace: [hivemr3]  failed.

        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:229) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:384) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:382) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:118) ~[mr3-tez-1.0-assembly.jar:1.0]

        ... 18 more

Caused by: java.net.SocketTimeoutException: connect timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_232]

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_232]

        at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_232]

        at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:110) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.RealCall.execute(RealCall.java:92) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:330) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:311) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:810) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:218) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:384) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:382) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:118) ~[mr3-tez-1.0-assembly.jar:1.0]

        ... 18 more

2020-08-10T13:02:37,538  INFO [main] mr3.HiveMR3ClientImpl: HiveMR3Client.close() terminates the current Application

2020-08-10T13:02:37,538  WARN [main] mr3.HiveMR3ClientImpl: Failed to close MR3Client

java.lang.IllegalArgumentException: requirement failed: MR3SessionClient c8fe85b1-d7aa-4e56-93d4-95b961e82cb9 not running

        at com.datamonad.mr3.client.MR3SessionClientImpl.getCurrentDagClientHandlerAppId(MR3SessionClientImpl.scala:164) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.shutdownAppMasterToTerminateApplication(MR3SessionClientImpl.scala:143) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.close(HiveMR3ClientImpl.java:128) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.close(MR3SessionImpl.java:266) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:142) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.createSession(MR3SessionManagerImpl.java:393) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.setup(MR3SessionManagerImpl.java:162) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:400) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:942) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:132) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1208) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1052) ~[hive-service-3.1.2.jar:3.1.2]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]

        at org.apache.hadoop.util.RunJar.run(RunJar.java:318) ~[hadoop-common-3.1.2.jar:?]

        at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.2.jar:?]

2020-08-10T13:02:37,576 ERROR [main] server.HiveServer2: Error in setting up MR3SessionManager

org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create or start MR3Client

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:143) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.createSession(MR3SessionManagerImpl.java:393) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.setup(MR3SessionManagerImpl.java:162) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:400) [hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:942) [hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:132) [hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1208) [hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1052) [hive-service-3.1.2.jar:3.1.2]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]

        at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.2.jar:?]

        at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.2.jar:?]

Caused by: com.datamonad.mr3.api.common.MR3Exception: Failed to initialize ConfigMap mr3conf-configmap-master - check if it already exists

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMK8sClient.start(AMK8sClient.scala:55) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3ClientBase.start(MR3ClientBase.scala:52) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.start(MR3SessionClientImpl.scala:70) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.start(HiveMR3ClientImpl.java:80) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:134) ~[hive-exec-3.1.2.jar:3.1.2]

        ... 13 more

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [ConfigMap]  with name: [mr3conf-configmap-master]  in namespace: [hivemr3]  failed.

        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:229) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:384) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:382) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:118) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMK8sClient.start(AMK8sClient.scala:55) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3ClientBase.start(MR3ClientBase.scala:52) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.start(MR3SessionClientImpl.scala:70) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.start(HiveMR3ClientImpl.java:80) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:134) ~[hive-exec-3.1.2.jar:3.1.2]

        ... 13 more

Caused by: java.net.SocketTimeoutException: connect timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_232]

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_232]

        at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_232]

        at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:110) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.RealCall.execute(RealCall.java:92) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:330) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:311) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:810) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:218) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:384) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:382) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:118) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMK8sClient.start(AMK8sClient.scala:55) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3ClientBase.start(MR3ClientBase.scala:52) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.start(MR3SessionClientImpl.scala:70) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.start(HiveMR3ClientImpl.java:80) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:134) ~[hive-exec-3.1.2.jar:3.1.2]

        ... 13 more

2020-08-10T13:02:37,580  INFO [main] server.HiveServer2: Shutting down HiveServer2

2020-08-10T13:02:37,580  INFO [main] metastore.HiveMetaStoreClient: Closed a connection to metastore, current connections: 0

2020-08-10T13:02:37,580  INFO [main] session.MR3SessionManagerImpl: Closing MR3SessionManager

2020-08-10T13:02:37,580  WARN [main] server.HiveServer2: Error starting HiveServer2 on attempt 1, will retry in 60000ms

org.apache.hive.service.ServiceException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create or start MR3Client

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:403) ~[hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:942) [hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:132) [hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1208) [hive-service-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1052) [hive-service-3.1.2.jar:3.1.2]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]

        at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.2.jar:?]

        at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.2.jar:?]

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create or start MR3Client

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:143) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.createSession(MR3SessionManagerImpl.java:393) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.setup(MR3SessionManagerImpl.java:162) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:400) ~[hive-service-3.1.2.jar:3.1.2]

        ... 10 more

Caused by: com.datamonad.mr3.api.common.MR3Exception: Failed to initialize ConfigMap mr3conf-configmap-master - check if it already exists

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMK8sClient.start(AMK8sClient.scala:55) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3ClientBase.start(MR3ClientBase.scala:52) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.start(MR3SessionClientImpl.scala:70) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.start(HiveMR3ClientImpl.java:80) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:134) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.createSession(MR3SessionManagerImpl.java:393) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.setup(MR3SessionManagerImpl.java:162) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:400) ~[hive-service-3.1.2.jar:3.1.2]

        ... 10 more

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [ConfigMap]  with name: [mr3conf-configmap-master]  in namespace: [hivemr3]  failed.

        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:229) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:384) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:382) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:118) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMK8sClient.start(AMK8sClient.scala:55) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3ClientBase.start(MR3ClientBase.scala:52) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.start(MR3SessionClientImpl.scala:70) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.start(HiveMR3ClientImpl.java:80) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:134) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.createSession(MR3SessionManagerImpl.java:393) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.setup(MR3SessionManagerImpl.java:162) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:400) ~[hive-service-3.1.2.jar:3.1.2]

        ... 10 more

Caused by: java.net.SocketTimeoutException: connect timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_232]

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_232]

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_232]

        at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_232]

        at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:110) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254) ~[mr3-tez-1.0-assembly.jar:1.0]

        at okhttp3.RealCall.execute(RealCall.java:92) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:330) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:311) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:810) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:218) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:384) ~[mr3-tez-1.0-assembly.jar:1.0]

        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:382) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMReplicationControllerContainerExtender.initialize(AMReplicationControllerContainerExtender.scala:118) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.AMK8sClient.start(AMK8sClient.scala:55) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3ClientBase.start(MR3ClientBase.scala:52) ~[mr3-tez-1.0-assembly.jar:1.0]

        at com.datamonad.mr3.client.MR3SessionClientImpl.start(MR3SessionClientImpl.scala:70) ~[mr3-tez-1.0-assembly.jar:1.0]

        at org.apache.hadoop.hive.ql.exec.mr3.HiveMR3ClientImpl.start(HiveMR3ClientImpl.java:80) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:134) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.createSession(MR3SessionManagerImpl.java:393) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionManagerImpl.setup(MR3SessionManagerImpl.java:162) ~[hive-exec-3.1.2.jar:3.1.2]

        at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:400) ~[hive-service-3.1.2.jar:3.1.2]

[Sungwoo Park]

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [ConfigMap]  with name: [mr3conf-configmap-master]  in namespace: [hivemr3]  failed. 

Caused by: java.net.SocketTimeoutException: connect timed out 

From the above messages, my guess is that HiveServer2 Pod fails to contact the K8s API server -- it tries to ask the K8s API server if there already exists a ConfigMap mr3-conf-configmap-master, but the connection itself is not established. Not a solution to your problem, but I would suggest you check the health of your K8s cluster.

Cheers,

--- Sungwoo

[Ahmed]

K8s Health is green.
If that is the case then metastore pod should also throw the same exception, but it is running.

Thanks

[Ahmed]

I have recreated the cluster, still same issue.
Kindly help on this.

Thanks

[Ahmed]

In the pods description we are getting below ERROR.
Liveness probe failed dial TCP 10.2 41.3:9852 connection refused
readiness probe failed: dial TCP 10.2 14.1.3:9852 to connect: connection refused

Thanks

[Sungwoo Park]

Metastore Pod does not create a new ConfigMap, but HiveServer2 Pod creates a new ConfigMap before creating DAGAppMaster Pod. So, Metastore Pod does not contact the K8s API server, whereas HiveServer2 Pod should contact the K8s API server. 

In run-hive.sh, you can see:

kubectl create -f $YAML_DIR/hive-role.yaml

kubectl create rolebinding hive-role-binding --role=hive-role --serviceaccount=$MR3_NAMESPACE:$MR3_SERVICE_ACCOUNT -n $MR3_NAMESPACE

In yaml/hive-role.yaml:

- apiGroups: [""]

  resources: ["pods"]

  verbs: ["get", "watch", "list", "create", "delete"]

- apiGroups: [""]

  resources: ["configmaps"]

  verbs: ["get", "create", "update", "delete"]

So, HiveServer2 Pod can create new ConfigMaps and Pods. In your setting, this does not work as intended because HiveServer2 Pod cannot reach the API server. So, I guess there is a problem either with K8s or with HiveServer2 Pod. Can you ssh into the HiveServer2 Pod and see if you can reach outside, e.g., by running ping?

Cheers,

--- Sungwoo

[Sungwoo Park]

This is as expected because HiveServer2 becomes live only after MR3 DAGAppMaster starts.

Cheers,

--- Sungwoo

[Ahmed]

Outside ping is not working..
I tried pinging master but no luck,
But ping to worker is working. That may be because pods are running on the same node.

Thanks

[Sungwoo Park]

Then, I guess it is a problem with network configuration, either in the cluster itself or in the K8s cluster.

Can you check the log of the Pod on the worker node that is responsible for network in the K8s cluster? For example, in our test cluster, the Pod is weave-net-mvsrd.

gitlab-runner@orange1:~/mr3-run/kubernetes$ kubectl describe node orange2 | grep "kube-system"

  kube-system                 kube-proxy-4dbps                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         231d

  kube-system                 tiller-deploy-8488d98b4c-njzss    0 (0%)        0 (0%)      0 (0%)           0 (0%)         27d

  kube-system                 weave-net-mvsrd                   20m (0%)      0 (0%)      0 (0%)           0 (0%)         231d

Cheers,

--- Sungwoo

[Ahmed]

Hi sungwoo,

There was issue with network and that has been taken care.
Now still we are getting few error in hiveserrver2 pod.
ERROR: fail to start mr3 session
Matric line failed to start for transition to ready
Mr3master pod is in pending state.

[Sungwoo Park]

Hi,

You could try to find out why MR3 master Pod fails to start by checking its state using 'kubectl describe' or its log. It's hard to tell the source of the problem from your message, which just says that MR3 master Pod does not start.

Cheers,

--- Sungwoo

[Ahmed]

Hi Sungwoo,

That issue was due to resources, when i changed number of worker core  from 2 to 1 it got resolved.

Since we are not using distribute storage for our 3 node cluster, How can we use minio object storage???

Any link for that.?

Kindly help on this..

Thanks, 

Ahmed

[Sungwoo Park]

Hello,

MinIO is okay to use with MR3. (One of our internal K8s clusters uses MinIO and everything works okay.) As it simulates S3, all the instructions for S3 equally apply to MinIO. For using MinIO in Hive on MR3, please see:

https://mr3docs.datamonad.com/docs/k8s/advanced/access-s3/  

You can also use S3 in place of PersistentVolumes. For more details, please see:

https://mr3docs.datamonad.com/docs/k8s/advanced/use-s3/  

Cheers,

--- Sungwoo
 

[Ahmed]

Thanks for the quick reply.

We have minio running in distributed mode not with s3.

how we can use minio as PV for hivemr3

Could you please help with an example ..??

Thanks.

[Sungwoo Park]

1. 

If your MinIO installation does not support S3 API, we don't know how to use Hive on MR3, which has been tested only with HDFS and S3 for distributed storage. Please check if your MinIO installation supports S3 API.

2.

If your MinIO installation supports S3 API, you can use S3 instead of PersistentVolumes. Note that S3 does not implement PersistentVolumes.

Cheers,

--- Sungwoo

[Ahmed]

Hi Sungwoo,

I am getting below error,

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)

 at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50) ~[aws-java-sdk-bundle-1.11.271.jar:?]

i have set env variable before starting hive

since minio is s3 compatible storage i am using s3a protocol i am referring link which you have provided above.

Thanks.

[Sungwoo Park]

Can you ssh to DAGAppMaster Pod, and see if the two env variables are set with key and password?

gitlab-runner@orange1:~/mr3-run/kubernetes$ kubectl exec -it -n hivemr3 mr3master-5732-0-fjfhx -- /bin/bash

root@mr3master-5732-0:/opt/mr3-run/hive# printenv | grep AWS

AWS_SECRET_ACCESS_KEY=yourawssecret

AWS_ACCESS_KEY_ID=youraccesskey

1. If these environment variables are properly set, then I guess it's a configuration problem with MinIO or S3 in kubernetes/conf/core-site.xml.

2. If not, it's a configuration problem with MR3.

Cheers,

--- Sungwoo

[Ahmed]

 DAGAppMaster Pod is not running.. below are hiveserve2 logs.

WARN [main] server.HiveServer2: Error starting HiveServer2 on attempt 1, will retry in 60000ms

java.lang.RuntimeException: Error applying authorization policy on hive configuration: org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on hivemr3-warehouse-dir: com.amazonaws.AmazonClientException: No AWS Credentials provided by EnvironmentVariableCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)): No AWS Credentials provided by EnvironmentVariableCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))

i am referring below link.

https://mr3docs.datamonad.com/docs/k8s/advanced/access-s3/  

Thanks

[Sungwoo Park]

Do you see env-secret.sh mounted inside HiveServer2 Pod?

gitlab-runner@orange1:~/mr3-run$ kubectl exec -it -n hivemr3 hivemr3-hiveserver2-8mv85 /bin/bash;

root@hivemr3-hiveserver2-8mv85:/opt/mr3-run/hive# grep AWS /opt/mr3-run/env-secret.sh 

export AWS_ACCESS_KEY_ID=accesskey

export AWS_SECRET_ACCESS_KEY=awesomesecret

Cheers,

--- Sungwoo

[Ahmed]

Hi  Sungwoo,

There is no such file in the hiveserver2 pod.

Thanks. 

[Ahmed]

I mean  env-secret.sh  is present in the pod. 

[Sungwoo Park]

Do you find env.sh in the HiveServer2 Pod? If it does, does it export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?

--- Sungwoo

[Ahmed]

yes env.sh we have in the pod. but that does not export key and ID.

Do we have to export this in env.sh

Thanks.

[Sungwoo Park]

Yes, (if you don't use Helm), you should export the variables in env.sh (https://mr3docs.datamonad.com/docs/k8s/advanced/access-s3/).

Cheers,

--- Sungwoo

[Praveen Kumar]

Got it. Thank you, Sungwoo.

--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/74561f41-3edb-4e27-82a7-9822c95e8dafn%40googlegroups.com.

[Ahmed]

Hi  Sungwoo Park,

Now we are using hdfs as a persistence volume. if distributed fs is hdfs what would be contents of workdir-pvc.yaml    workdir-pv.yaml .

and what all changes we need to do in hive configuration.

how we can put mysql driver in hdfs and fetch..?

kindly help on this.

Thanks

[Sungwoo Park]

Hello,

PersistentVolumes are not something like HDFS or S3. In the case of MR3, PersistentVolumes are not used as the source of input data. That is, the user would place his input data on HDFS or S3, while PersistentVolumes are a distributed storage that is used to pass temporary data (e.g., result of a query to be displayed in Beeline) between HiveServer2, DAGAppMaster, and ContainerWorkers. Thus, you cannot use HDFS as PersistentVolumes.

Normally the user would need:

1. HDFS or S3 for the source of data

2. PersistentVolume for passing temporary data between HiveServer2, DAGAppMaster, and ContainerWorkers

If you take this option, you should create a PersistentVolume, e.g., with NFS, and set workdir-pv.yaml and workdir-pvc.yaml.

However, the user can run MR3 without using PersistentVolumes. This is explained in:

https://mr3docs.datamonad.com/docs/k8s/advanced/use-hdfs/ 

In this case, the Docker image should contain a MySQL connector jar file, or the user should manually mount using a hostPath volume.

Cheers, 

--- Sungwoo

[Ahmed]

Got you.

when we fire select avg(HP) from pokemon; we are getting below error. Is it because we are not using distributed storage.??

Caused by: java.io.FileNotFoundException: File file:/opt/mr3-run/work-dir/warehouse/pokemon/pokemon_copy_1.csv does not exist

Thanks

[Ahmed]

Hi,

Also how we can create hive warehouse directory on hdfs..???

Thanks.

[Sungwoo Park]

You create a directory on HDFS, and designate it as your warehouse directory in kubernetes/env.sh (see  https://mr3docs.datamonad.com/docs/k8s/guide/configure-hiveserver2/):

HIVE_WAREHOUSE_DIR=/tmp/hivemr3/warehouse

Then, all the databases and tables are created under this directory. You want to check the permission so that Hive can read/write to the directory.

Cheers,

--- Sungwoo

[Ahmed]

Hi Sungwoo,

We refer the link sent above  and create warehouse dir on hdfs allow hive to read write and define warehouse path "HIVE_WAREHOUSE_DIR=/tmp/hivemr3/warehouse" in env.sh

but still its pointing to below location in the pod

/opt/mr3-run/work-dir/warehouse/

when we ran sample query i am getting below error (select avg(HP) from pokemon;)

Input path does not exist: file:/opt/mr3-run/work-dir/warehouse/pokemon

also we tried changing warehouse path to hdfs://ip:8020/tmp/hivemr3/warehouse but no luck.

kindly help on this.

Thanks.

[Sungwoo Park]

Hello,

I expanded the page (https://mr3docs.datamonad.com/docs/k8s/guide/configure-hiveserver2/) on  HIVE_WAREHOUSE_DIR, so please check it out.

If you created pokemon previously, Metastore still thinks the database is in the local filesystem. So, I'd suggest you create new databases in the warehouse on HDFS.

Cheers,

--- Sungwoo

[Ahmed]

Thanks.

Using above link we were able to resolved the issue.

any link to run benchmark jobs.?

Thanks.

[Sungwoo Park]

https://mr3docs.datamonad.com/docs/hadoop/advanced/run-tpcds/ 

If you have a Hadoop installation (with MapReduce), you can generate a TPC-DS dataset in text format. Then, you can load the dataset in ORC by running Hive on MR3 on Kubernetes. So, see if you can generate TPC-DS datasets.

Cheers,

--- Sungwoo

[Ahmed]

Hi  Sungwoo,

We have uploaded movielense data set and ran a complex query. but every thing is running on single worker. Multiple workers are not getting launched.  

Also getting below error in hiveserver2 logs from the beginning only.

2020-08-26T15:24:31,758 ERROR [HiveServer2-Handler-Pool: Thread-37] server.TThreadPoolServer: Error occurred during processing of message.

java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.2.jar:3.1.2]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

Thanks

[Sungwoo Park]

We have uploaded movielense data set and ran a complex query. but every thing is running on single worker. Multiple workers are not getting launched.  

Either a single ContainerWorker is enough for running the query, or you should change the configuration. Please see 'Resources for mappers (Map Tasks), reducers (Reduce Tasks), and ContainerWorkers' of Performance Tuning (https://mr3docs.datamonad.com/docs/hadoop/advanced/performance-tuning/). Also check if K8s nodes have enough resources to accommodate ContainerWorker Pods.

 

Also getting below error in hiveserver2 logs from the beginning only.

2020-08-26T15:24:31,758 ERROR [HiveServer2-Handler-Pool: Thread-37] server.TThreadPoolServer: Error occurred during processing of message.

java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.2.jar:3.1.2]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

It's not a bug. Please see Troubleshooting (https://mr3docs.datamonad.com/docs/k8s/troubleshoot/).

Cheers,

--- Sungwoo

[Sungwoo Park]

Please see another thread 'ping related warning/error in the log - no data or no SASL in the stream' in this mailing list for more details on this issue. The docker image 1.2-SNAPSHOT does not show this problem.

--- Sungwoo

[Ahmed]

Hi Sungwoo,

On Running below query we getting error. When we increase this memory error resolve, did not get how memory it requires.?

select COUNT(name), power_rate from pokemon1 group by power_rate;

hive.mr3.map.task.memory.mb=512

hive.mr3.reduce.task.memory.mb=512

hive.mr3.all-in-one.containergroup.memory.mb=1024

ERROR:-

Caused by: java.lang.IllegalArgumentException: tez.runtime.io.sort.mb 1040 should be larger than 0 and should be less than the available task memory 

(MB):358

Also how we can configure hive WebUI..???

Thanks.

[Ahmed]

Hi 

How we can user hdfs for workers data

What could be value for below property.?

<property> <name>mr3.k8s.pod.worker.hostpaths</name> 

  <value>/data1/gla/k8s</value> </property> 

Kindly help on this.

Thanks

 

[Sungwoo Park]

1.

tez.runtime.io.sort.mb  should be adjusted so that the sort buffer can fit in the memory of mapper/reducer. In your case, you allocate 1GB for sort buffer in a mapper/reducer whose memory is 512MB.

 

2.

hive.server2.webui.port specifies the port for web UI. However, web UI does not show useful information on HiveServer2.

Cheers,

--- Sungwoo

[Sungwoo Park]

You can't use HDFS for mr3.k8s.pod.worker.hostpaths because it specifies local directories to be used as hostPath volumes. For details, please see: https://mr3docs.datamonad.com/docs/k8s/guide/configure-hivek8s/

Cheers.

--- Sungwoo 

[Ahmed]

You can't use HDFS for mr3.k8s.pod.worker.hostpaths because it specifies local directories to be used as hostPath volumes 

This means we need to create local dir on all nodes..??

or it will create automatically .???

or it will creating dir inside the pods..???

Thanks. 

[Sungwoo Park]

Yes, you have to create the same set of working directories on every node, if you would like to use local disks for writing intermediate data.

Cheers,

--- Sungwoo

[Ahmed]

Hi,

What are the other options to store intermediate data.??

Thanks.

[Sungwoo Park]

You can use emptyDir volumes (mr3.k8s.pod.worker.emptydirs). emptyDir volumes are okay for running small queries that do not generate a lot of intermediate data. For details, see https://mr3docs.datamonad.com/docs/k8s/guide/configure-hivek8s/.

Cheers,

--- Sungwoo

[Ahmed]

Thanks, above link helped me.

But still we are not able to launch multiple worker pods=>we have memory available around 4 gb on each node.

we tried opening two terminal and ran query, it got stuck but not creating on more worker pod.

here is the configurations.

hive.mr3.map.task.memory.mb=1024

hive.mr3.reduce.task.memory.mb=1024

hive.mr3.all-in-one.containergroup.memory.mb=2048

Kindly give your suggestion on this.

Thanks.

[Sungwoo Park]

If a worker node has 4GB of physical memory, the size of memory available to K8s is slightly smaller. As a result, you cannot fit 2 ContainerWorker Pods in a single worker node. So, check the size of memory available in each worker node.

Cheers,

--- Sungwoo

[Ahmed]

will it try to run worker pod on same node.???

Thanks

[Ahmed]

Hi 

Got confused with work-dir-volume=> Do we need to create PV and PVC for work-dir even if we are using hdfs..??

currently we are creating pv pvc for mysql connector only.

as per below link we dont need pv for worker pod when we are using hdfs, But again we dont want to use emtydir for intermediate data.

https://mr3docs.datamonad.com/docs/k8s/advanced/use-hdfs/ 

we have create /data1/gla/k8s on all the nodes and  set in mr3-site.xml

<property>

  <name>mr3.k8s.pod.worker.hostpaths</name>

 <value>/data1/gla/k8s</value>

</property>

but we are getting this below error

org.apache.hadoop.util.DiskChecker$DiskErrorException: No space available in any of the local directories.

        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:400)

Thanks.

[Sungwoo Park]

1. PVs are for passing data such as the result of running a query. You can either create PVs or use HDFS instead if you don't want to create PVs.

If you create PVs, then you can 1) use HDFS for data source; 2) use PVs for passing data such as the result of running queries; 3) mr3.k8s.pod.worker.hostpaths for local directories.  

2. For 'No space available in any of the local directories', please see https://mr3docs.datamonad.com/docs/k8s/troubleshoot/. In most cases, it happens because ContainerWorker Pods do not have write permission on the local directories. 

There is another active thread 'No space available in any of the local directories', so it might help you.

Cheers,

--- Sungwoo

[Ahmed]

Thanks that got fix.

if we define volume type to hostpath it will create /data1/gla/k8s if not exist as well.

Now Next step is to configure ranger on it. Not able to get ranger build file in current mr3 folder.

do we need to download other code.? Or Any best link to configure ranger.?

Thanks.

[Sungwoo Park]

You can download a pre-built MR3 release and find build-k8s-ranger.sh. 

We have two pages on Ranger:

https://mr3docs.datamonad.com/docs/k8s/guide/run-ranger/

https://mr3docs.datamonad.com/docs/hadoop/advanced/integrate-ranger/

Check the source code of build-k8s-ranger.sh for details of building a Docker image. For configuring Ranger, you could visit Apache Ranger webpage.

Cheers,

--- Sungwoo

[Ahmed]

Hi  Sungwoo ,

I have downloaded hivemr3-1.1-hive3.1.2.tar.gz and ran build-k8s-ranger.sh but getting below error.

downloading and building Ranger at /tmp/ranger

Cloning into '/tmp/ranger'...

remote: Enumerating objects: 86558, done.

remote: Total 86558 (delta 0), reused 0 (delta 0), pack-reused 86558

Receiving objects: 100% (86558/86558), 95.28 MiB | 5.78 MiB/s, done.

Resolving deltas: 100% (44060/44060), done.

/tmp/ranger ~/hivemr3-1.1-hive3.1.2

Note: checking out 'f57022645db36054d2c63f1af503b82363999b47'.

You are in 'detached HEAD' state. You can look around, make experimental

changes and commit them, and you can discard any commits you make in this

state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may

do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at f570226... RANGER-2556:RangerHivePlugin Row filtering and Column Masking auditing gives inconsistent audit information - addon

/root/hivemr3-1.1-hive3.1.2/common-build-setup.sh: line 127: mvn: command not found

~/hivemr3-1.1-hive3.1.2

tar (child): /tmp/ranger/target/ranger-2.1.0-SNAPSHOT-admin.tar.gz: Cannot open: No such file or directory

tar (child): Error is not recoverable: exiting now

tar: Child returned status 2

tar: Error is not recoverable: exiting now

mv: cannot stat ‘/root/hivemr3-1.1-hive3.1.2/kubernetes/ranger/ranger-*-admin’: No such file or directory

tar (child): /tmp/ranger/target/ranger-2.1.0-SNAPSHOT-hive-plugin.tar.gz: Cannot open: No such file or directory

tar (child): Error is not recoverable: exiting now

tar: Child returned status 2

tar: Error is not recoverable: exiting now

kindly help on this.

Thanks.

[Sungwoo Park]

/root/hivemr3-1.1-hive3.1.2/common-build-setup.sh: line 127: mvn: command not found

You should install mvn to build Ranger.

For your earlier question on setting up hostPath volumes, I have updated the page on User Guide:

https://mr3docs.datamonad.com/docs/k8s/guide/

Cheers,

--- Sungwoo

[Ahmed]

Thanks  Sungwoo . 

[Ahmed]

Hi Sungwoo ,

Can we run ranger without kerberos..???

if so  can we skip below steps.?

The configuration keys ranger.spnego.kerberos.principal, ranger.admin.kerberos.principal, and ranger.lookup.kerberos.principal should use 

service principals whose keytab files are in the directory kubernetes/ranger-key.  

Thanks.

[Ahmed]

Got it. please ignore above email.

[Ahmed]

Hi

getting below error running hive quires other that hive user.

its only working with hive user.

show databses;

Error: Error while compiling statement: FAILED: ParseException line 1:5 cannot recognize input near 'show' 'databses' '<EOF>' in ddl statement (state=42000,code=40000)

kindly help on this.

Thanks.

[Ahmed]

after enabling ranger getting same error for hive user as well.

also not able to add hive service from ranger, getting below error

org.apache.ranger.plugin.client.HadoopException: Unable to execute SQL [show databases like "*"]..
Unable to execute SQL [show databases like "*"]..
Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [hive] does not have [USE] privilege on [Unknown resource!!].
Permission denied: user [hive] does not have [USE] privilege on [Unknown resource!!]  

Thanks.

[Sungwoo Park]

Hi,

It takes a while for HiveServer2 to download Ranger policy. Check the log of HiveServer2. For example, you might see something like:

2020-08-12T04:03:14,477 ERROR [main] util.PolicyRefresher: PolicyRefresher(serviceName=BLUE_hive): failed to refresh policies. Will continue to use last known version of policies (-1)

...

2020-08-12T04:04:13,759  INFO [Thread-7] util.PolicyRefresher: PolicyRefresher(serviceName=BLUE_hive): found updated version. lastKnownVersion=-1; newVersion=25

Until HiveServer2 retrieves the policy from Ranger, no user can execute any query.

If you still see the same problem after HiveServer2 retrieves the policy from Ranger, it's a problem of configuring the policy in Ranger.

Cheers,

--- Sungwoo

[Ahmed]

Hi  Sungwoo ,

We have not done any config related to Solr Is that mandatory ???

We dont need Solr actually.

Thanks 

[Sungwoo Park]

Ranger internally uses Solr. The Ranger in MR3docs configures Solr.

Cheers,

--- Sungwoo

[Ahmed]

Hi,

But when i ran the ranger.yaml i am getting an error which is saying (You need to choose one of solr or ranger)

do we need to create separate yaml for solr and ranger.??

Thanks.

[Sungwoo Park]

Hello,

Please follow the instruction in https://mr3docs.datamonad.com/docs/k8s/guide/run-ranger/, and execute kubernetes/run-ranger.sh. Internally it runs two containers (ranger and solr).

Cheers,

--- Sungwoo

[Ahmed]

Hi,

when we running quires from hue we are getting below errors, any idea on this.??

• database is locked
• Cannot parse job IDs for execution engine mr3
• Cannot parse job IDs for execution engine mr3

Thanks

[Ahmed]

Hi 

Getting below container when using two container.

error: a container name must be specified for pod hivemr3-ranger, choose one of: [solr ranger] 

Thanks

[Ahmed]

Hi  Sungwoo,

above issue got resolved but below issue persist.

• database is locked
• Cannot parse job IDs for execution engine mr3
• Cannot parse job IDs for execution engine mr3

Thanks

[Sungwoo Park]

Not sure about the source of the problem. Please check out: https://mr3docs.datamonad.com/docs/hadoop/advanced/access-hue/

(You might want to provide more information on how to produce errors. We assume that you have read relevant pages in MR3docs, so with no information on how to produce errors, we would have no clue on how to fix them and would not be able to reply.)

Cheers,

--- Sungwoo

[Ahmed]

Hi  Sungwoo ,

Thanks a lot, Above link worked. Issue has been resolve.

 we set  hive.execution.engine  to tez, it was mr3 before.

does that make any changes in the performance ..???

Thanks. 

[Sungwoo Park]

No difference. Internally it uses MR3.

--- Sungwoo

[Ahmed]

That's great.

Thanks 

[Ahmed]

Hi,

We need some jars to support s3a protocol, So we are building custom hive image using below docker file.

getting error when running metastore.

Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path

+ hive --service metastore -p 9850

Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path

Dockerfile:

FROM centos:7.6.1810

MAINTAINER **** "*****"

USER root

RUN yum install -y java-1.8.0-openjdk \

    && echo "securerandom.source=file:/dev/urandom" >> /usr/lib/jvm/jre/lib/security/java.security \

    && yum install -y which \

    && yum clean all \

    && rm -rf \

          /usr/lib/jvm/jre/plugin \

          /usr/lib/jvm/jre/bin/javaws \

          /usr/lib/jvm/jre/bin/jjs \

          /usr/lib/jvm/jre/bin/orbd \

          /usr/lib/jvm/jre/bin/pack200 \

          /usr/lib/jvm/jre/bin/policytool \

          /usr/lib/jvm/jre/bin/rmid \

          /usr/lib/jvm/jre/bin/rmiregistry \

          /usr/lib/jvm/jre/bin/servertool \

          /usr/lib/jvm/jre/bin/tnameserv \

          /usr/lib/jvm/jre/bin/unpack200 \

          /usr/lib/jvm/jre/lib/javaws.jar \

          /usr/lib/jvm/jre/lib/deploy* \

          /usr/lib/jvm/jre/lib/desktop \

          /usr/lib/jvm/jre/lib/*javafx* \

          /usr/lib/jvm/jre/lib/*jfx* \

          /usr/lib/jvm/jre/lib/amd64/libdecora_sse.so \

          /usr/lib/jvm/jre/lib/amd64/libprism_*.so \

          /usr/lib/jvm/jre/lib/amd64/libfxplugins.so \

          /usr/lib/jvm/jre/lib/amd64/libglass.so \

          /usr/lib/jvm/jre/lib/amd64/libgstreamer-lite.so \

          /usr/lib/jvm/jre/lib/amd64/libjavafx*.so \

          /usr/lib/jvm/jre/lib/amd64/libjfx*.so \

          /usr/lib/jvm/jre/lib/ext/jfxrt.jar \

          /usr/lib/jvm/jre/lib/ext/nashorn.jar \

          /usr/lib/jvm/jre/lib/oblique-fonts \

          /usr/lib/jvm/jre/lib/plugin.jar \

          /usr/lib/jvm/jre/man/

ENV JAVA_HOME /etc/alternatives/jre

ENV PATH="/etc/alternatives/jre/bin:${PATH}"

# create directories under /opt/mr3-run

RUN mkdir -p /opt/mr3-run && \

    mkdir -p /opt/mr3-run/scratch-dir && \

    mkdir -p /opt/mr3-run/work-dir && \

    mkdir -p /opt/mr3-run/work-local-dir && \

    mkdir -p /opt/mr3-run/key && \

    mkdir -p /opt/mr3-run/conf && \

    mkdir -p /opt/mr3-run/hive/tmp && \

    mkdir -p /opt/mr3-run/hive/run-result && \

    mkdir -p /opt/mr3-run/host-lib

# copy files

COPY common-setup.sh /opt/mr3-run/

COPY hadoop/hadoop-setup.sh /opt/mr3-run/hadoop/

COPY hive/*.sh /opt/mr3-run/hive/

COPY mr3/mr3-setup.sh /opt/mr3-run/mr3/

COPY tez/tez-setup.sh /opt/mr3-run/tez/

#COPY hadoop/apache-hadoop /opt/mr3-run/hadoop/apache-hadoop

COPY hive/apache-hive /opt/mr3-run/hive/apache-hive

COPY lib /opt/mr3-run/lib

COPY mr3/mr3lib /opt/mr3-run/mr3/mr3lib

COPY tez/tezjar /opt/mr3-run/tez/tezjar

# set environments

WORKDIR /opt/mr3-run/hive

ARG UID=1000

RUN adduser --no-create-home -s /sbin/nologin -U -u $UID hive && \

    chown hive /opt/mr3-run/scratch-dir && \

    chown hive /opt/mr3-run/work-dir && \

    chown hive /opt/mr3-run/work-local-dir && \

    chown hive /opt/mr3-run/hive && \

    chown hive /opt/mr3-run/hive/tmp && \

    chown hive /opt/mr3-run/hive/run-result

USER hive

Kindly help on this.

Thanks

[Sungwoo Park]

You need a Hadoop installation (but not a working Hadoop cluster) in order to build a Docker image. See:

https://mr3docs.datamonad.com/docs/install/install-hivek8s/

The Docker image mr3project/hive3 already supports s3a, so you don't need a new image if your goal is to support s3a.

Cheers,

--- Sungwoo

[Ahmed]

Hi Sungwoo,

Correct its supporting s3a.

We are using minio object storage as hive warehouse dir and hive scratch dir, We are facing issue while running queries,

Caused by: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint

we followed this two links 

https://mr3docs.datamonad.com/docs/k8s/advanced/use-s3/

https://mr3docs.datamonad.com/docs/k8s/advanced/access-s3/

we are not using S3, But S3 compatible storage (minio)

env.sh

export AWS_ACCESS_KEY_ID=***            (our minio keys which we get when we start minio)

export AWS_SECRET_ACCESS_KEY=***

Putting below in hive-site.xml

<property>

  <name>hive.acid.direct.insert.enabled</name>

  <value>true</value>

</property>

<property>

<name>fs.s3a.endpoint</name>

<description>AWS S3 endpoint to connect to.</description>

<value>http://ip:9000</value>

</property>

<property>

<name>fs.s3a.access.key</name>

<description>AWS access key ID.</description>

<value>****</value>

</property>

<property>

<name>fs.s3a.secret.key</name>

<description>AWS secret key.</description>

<value>****</value>

</property>

<property>

<name>fs.s3a.path.style.access</name>

<value>true</value>

<description>Enable S3 path style access.</description>

</property>

<property>

<name>hive.metastore.warehouse.dir</name>

<value>s3a://hive/warehouse</value>

</property>

Kindly help on this.

Thanks.

[Ahmed]

resolved above issue.

Thanks