Hive always loses some permissions created in Ranger
67 views
Skip to first unread message
Carol Chapman
Jul 25, 2022, 10:26:05 PM7/25/22
to MR3
At present, I find that online mr3 hive always loses the permissions of some tables given by Ranger after running for a period of time.
Here is the error log:
2022-07-13T14:59:45,745 WARN [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Error executing statement:
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.executeStatementAsync(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:567) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.1.3.jar:3.1.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.submitBackgroundOperation(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:280) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:568) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:554) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
... 18 more
Caused by: java.lang.NullPointerException
at org.apache.hive.service.cli.session.SessionManager.submitBackgroundOperation(SessionManager.java:665) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.submitBackgroundOperation(HiveSessionImpl.java:590) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.submitBackgroundOperation(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:280) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:568) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:554) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
... 18 more
2022-07-13T14:59:45,746 INFO [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Session disconnected without closing properly.
2022-07-13T14:59:45,746 INFO [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Closing the session: SessionHandle [f386c73e-8995-4638-9b68-7165aead878e]
2022-07-13T14:59:45,746 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] operation.OperationManager: Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=433eb954-118d-41ae-86eb-952bae21f09b]
2022-07-13T14:59:45,746 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] operation.OperationManager: Removed queryId: hue_20220713145945_31000b4b-09ff-455f-b79f-45ef4eb1cf14 corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=433eb954-118d-41ae-86eb-952bae21f09b] with tag: null
2022-07-13T14:59:45,758 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] exec.TableScanOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_TS_0:0,
2022-07-13T14:59:45,759 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] exec.SelectOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_SEL_1:0,
2022-07-13T14:59:45,759 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] exec.ListSinkOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_LIST_SINK_3:0,
2022-07-13T14:59:45,774 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] operation.Operation: Closing operation log /tmp/hue/operation_logs/f386c73e-8995-4638-9b68-7165aead878e/hue_20220713145945_31000b4b-09ff-455f-b79f-45ef4eb1cf14 without delay
2022-07-13T14:59:45,774 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] conf.HiveConf: Using the default value passed in for log id: f386c73e-8995-4638-9b68-7165aead878e
2022-07-13T14:59:45,774 ERROR [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] session.HiveSessionImpl: Failed to cleanup session log dir: SessionHandle [f386c73e-8995-4638-9b68-7165aead878e]
java.io.FileNotFoundException: File does not exist: /tmp/hue/operation_logs/f386c73e-8995-4638-9b68-7165aead878e
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2396) ~[commons-io-2.6.jar:2.6]
at org.apache.hive.service.cli.session.HiveSessionImpl.cleanupSessionLogDir(HiveSessionImpl.java:828) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:781) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.close(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:552) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:241) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService$1.deleteContext(ThriftBinaryCLIService.java:147) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300) ~[hive-exec-3.1.3.jar:3.1.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2022-07-13T14:59:45,774 INFO [HiveServer2-Handler-Pool: Thread-49] lockmgr.DbTxnManager: Stopped heartbeat for query: null
2022-07-13T14:59:45,814 INFO [HiveServer2-Handler-Pool: Thread-49] session.SessionState: Deleted directory: /tmp/hue/databank01/f386c73e-8995-4638-9b68-7165aead878e on fs with scheme hdfs
2022-07-13T14:59:45,819 INFO [HiveServer2-Handler-Pool: Thread-49] session.SessionState: Deleted directory: /tmp/hue/f386c73e-8995-4638-9b68-7165aead878e on fs with scheme file
2022-07-13T14:59:45,819 INFO [HiveServer2-Handler-Pool: Thread-49] metastore.HiveMetaStoreClient: Closed a connection to metastore, current connections: 7
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] service.CompositeService: Session closed, SessionHandle [f8c2652a-bd18-4b89-bf22-8e88b967bc10], current sessions:0
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,684 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] thrift.ThriftCLIService: Session disconnected without closing properly.
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] thrift.ThriftCLIService: Closing the session: SessionHandle [f8c2652a-bd18-4b89-bf22-8e88b967bc10]
2022-07-13T15:00:01,684 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] operation.OperationManager: Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=6b724d80-d308-415a-a1cf-c6fdd10ef4ed]
2022-07-13T15:00:01,684 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] operation.OperationManager: Removed queryId: hue_20220713110546_73a5ee6c-b01c-4ed0-96ed-dff58ed77fe4 corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=6b724d80-d308-415a-a1cf-c6fdd10ef4ed] with tag: null
2022-07-13T15:00:01,713 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] operation.Operation: Closing operation log /tmp/hue/operation_logs/f8c2652a-bd18-4b89-bf22-8e88b967bc10/hue_20220713110546_73a5ee6c-b01c-4ed0-96ed-dff58ed77fe4 without delay
2022-07-13T15:00:01,713 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,713 ERROR [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] session.HiveSessionImpl: Failed to cleanup session log dir: SessionHandle [f8c2652a-bd18-4b89-bf22-8e88b967bc10]
java.io.FileNotFoundException: File does not exist: /tmp/hue/operation_logs/f8c2652a-bd18-4b89-bf22-8e88b967bc10
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2396) ~[commons-io-2.6.jar:2.6]
at org.apache.hive.service.cli.session.HiveSessionImpl.cleanupSessionLogDir(HiveSessionImpl.java:828) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:781) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.close(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:552) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:241) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService$1.deleteContext(ThriftBinaryCLIService.java:147) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300) ~[hive-exec-3.1.3.jar:3.1.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.executeStatementAsync(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:567) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.1.3.jar:3.1.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.submitBackgroundOperation(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:280) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:568) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:554) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
... 18 more
Caused by: java.lang.NullPointerException
at org.apache.hive.service.cli.session.SessionManager.submitBackgroundOperation(SessionManager.java:665) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.submitBackgroundOperation(HiveSessionImpl.java:590) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.submitBackgroundOperation(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:280) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:568) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:554) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
... 18 more
2022-07-13T14:59:45,746 INFO [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Session disconnected without closing properly.
2022-07-13T14:59:45,746 INFO [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Closing the session: SessionHandle [f386c73e-8995-4638-9b68-7165aead878e]
2022-07-13T14:59:45,746 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] operation.OperationManager: Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=433eb954-118d-41ae-86eb-952bae21f09b]
2022-07-13T14:59:45,746 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] operation.OperationManager: Removed queryId: hue_20220713145945_31000b4b-09ff-455f-b79f-45ef4eb1cf14 corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=433eb954-118d-41ae-86eb-952bae21f09b] with tag: null
2022-07-13T14:59:45,758 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] exec.TableScanOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_TS_0:0,
2022-07-13T14:59:45,759 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] exec.SelectOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_SEL_1:0,
2022-07-13T14:59:45,759 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] exec.ListSinkOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_LIST_SINK_3:0,
2022-07-13T14:59:45,774 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] operation.Operation: Closing operation log /tmp/hue/operation_logs/f386c73e-8995-4638-9b68-7165aead878e/hue_20220713145945_31000b4b-09ff-455f-b79f-45ef4eb1cf14 without delay
2022-07-13T14:59:45,774 INFO [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] conf.HiveConf: Using the default value passed in for log id: f386c73e-8995-4638-9b68-7165aead878e
2022-07-13T14:59:45,774 ERROR [f386c73e-8995-4638-9b68-7165aead878e HiveServer2-Handler-Pool: Thread-49] session.HiveSessionImpl: Failed to cleanup session log dir: SessionHandle [f386c73e-8995-4638-9b68-7165aead878e]
java.io.FileNotFoundException: File does not exist: /tmp/hue/operation_logs/f386c73e-8995-4638-9b68-7165aead878e
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2396) ~[commons-io-2.6.jar:2.6]
at org.apache.hive.service.cli.session.HiveSessionImpl.cleanupSessionLogDir(HiveSessionImpl.java:828) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:781) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.close(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:552) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:241) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService$1.deleteContext(ThriftBinaryCLIService.java:147) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300) ~[hive-exec-3.1.3.jar:3.1.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2022-07-13T14:59:45,774 INFO [HiveServer2-Handler-Pool: Thread-49] lockmgr.DbTxnManager: Stopped heartbeat for query: null
2022-07-13T14:59:45,814 INFO [HiveServer2-Handler-Pool: Thread-49] session.SessionState: Deleted directory: /tmp/hue/databank01/f386c73e-8995-4638-9b68-7165aead878e on fs with scheme hdfs
2022-07-13T14:59:45,819 INFO [HiveServer2-Handler-Pool: Thread-49] session.SessionState: Deleted directory: /tmp/hue/f386c73e-8995-4638-9b68-7165aead878e on fs with scheme file
2022-07-13T14:59:45,819 INFO [HiveServer2-Handler-Pool: Thread-49] metastore.HiveMetaStoreClient: Closed a connection to metastore, current connections: 7
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] service.CompositeService: Session closed, SessionHandle [f8c2652a-bd18-4b89-bf22-8e88b967bc10], current sessions:0
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,684 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] thrift.ThriftCLIService: Session disconnected without closing properly.
2022-07-13T15:00:01,684 INFO [HiveServer2-Handler-Pool: Thread-772] thrift.ThriftCLIService: Closing the session: SessionHandle [f8c2652a-bd18-4b89-bf22-8e88b967bc10]
2022-07-13T15:00:01,684 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] operation.OperationManager: Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=6b724d80-d308-415a-a1cf-c6fdd10ef4ed]
2022-07-13T15:00:01,684 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] operation.OperationManager: Removed queryId: hue_20220713110546_73a5ee6c-b01c-4ed0-96ed-dff58ed77fe4 corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=6b724d80-d308-415a-a1cf-c6fdd10ef4ed] with tag: null
2022-07-13T15:00:01,713 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] operation.Operation: Closing operation log /tmp/hue/operation_logs/f8c2652a-bd18-4b89-bf22-8e88b967bc10/hue_20220713110546_73a5ee6c-b01c-4ed0-96ed-dff58ed77fe4 without delay
2022-07-13T15:00:01,713 INFO [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] conf.HiveConf: Using the default value passed in for log id: f8c2652a-bd18-4b89-bf22-8e88b967bc10
2022-07-13T15:00:01,713 ERROR [f8c2652a-bd18-4b89-bf22-8e88b967bc10 HiveServer2-Handler-Pool: Thread-772] session.HiveSessionImpl: Failed to cleanup session log dir: SessionHandle [f8c2652a-bd18-4b89-bf22-8e88b967bc10]
java.io.FileNotFoundException: File does not exist: /tmp/hue/operation_logs/f8c2652a-bd18-4b89-bf22-8e88b967bc10
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2396) ~[commons-io-2.6.jar:2.6]
at org.apache.hive.service.cli.session.HiveSessionImpl.cleanupSessionLogDir(HiveSessionImpl.java:828) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:781) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.3.jar:3.1.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_201]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at com.sun.proxy.$Proxy72.close(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:552) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:241) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService$1.deleteContext(ThriftBinaryCLIService.java:147) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300) ~[hive-exec-3.1.3.jar:3.1.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Carol Chapman
Jul 25, 2022, 10:26:50 PM7/25/22
to MR3
If I need to provide any additional information, I will provide it at the first time
Sungwoo Park
Jul 26, 2022, 12:27:33 AM7/26/22
to MR3
Do you see the same problem when using Beeline to connect to HiveServer2?
Or, could you share how to reproduce the error? This is probably an issue with Hive because Hive-MR3 replaces only the backend of Hive.
Cheers,
--- Sungwoo
Message has been deleted
Carol Chapman
Jul 26, 2022, 3:22:23 AM7/26/22
to MR3
This problem is not easy to recur. Because every time you restart it, it is always OK, but it will always go wrong with a long running time.
I have assigned multiple permission accounts to hive in Ranger, and the accounts with low permission often cause problems.
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
I have assigned multiple permission accounts to hive in Ranger, and the accounts with low permission often cause problems.
But I found this error when I started hive:
2022-07-19T14:24:46,255 ERROR [main] util.PolicyRefresher: failed to save policies to cache file '/etc/ranger/xxxx/policycache/hiveServer2_xxxxx_hive.json'
java.io.FileNotFoundException: /etc/ranger/xxxx/policycache/hiveServer2_xxxxx_hive.json (Permission denied)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_201]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_201]
at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[?:1.8.0_201]
at java.io.FileOutputStream.<init>(FileOutputStream.java:162) ~[?:1.8.0_201]
at java.io.FileWriter.<init>(FileWriter.java:90) ~[?:1.8.0_201]
at org.apache.ranger.plugin.util.PolicyRefresher.saveToCache(PolicyRefresher.java:392) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:210) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:149) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:222) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.authorization.hive.authorizer.RangerHivePlugin.init(RangerHiveAuthorizer.java:1933) ~[ranger-hive-plugin-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.<init>(RangerHiveAuthorizer.java:124) ~[ranger-hive-plugin-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory.createHiveAuthorizer(RangerHiveAuthorizerFactory.java:37) ~[ranger-hive-plugin-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:893) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1843) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.init(CLIService.java:115) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.CompositeService.init(CompositeService.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:240) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:990) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1256) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1100) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_201]
java.io.FileNotFoundException: /etc/ranger/xxxx/policycache/hiveServer2_xxxxx_hive.json (Permission denied)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_201]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_201]
at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[?:1.8.0_201]
at java.io.FileOutputStream.<init>(FileOutputStream.java:162) ~[?:1.8.0_201]
at java.io.FileWriter.<init>(FileWriter.java:90) ~[?:1.8.0_201]
at org.apache.ranger.plugin.util.PolicyRefresher.saveToCache(PolicyRefresher.java:392) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:210) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:149) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:222) ~[ranger-plugins-common-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.authorization.hive.authorizer.RangerHivePlugin.init(RangerHiveAuthorizer.java:1933) ~[ranger-hive-plugin-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.<init>(RangerHiveAuthorizer.java:124) ~[ranger-hive-plugin-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory.createHiveAuthorizer(RangerHiveAuthorizerFactory.java:37) ~[ranger-hive-plugin-1.2.0.3.1.0.0-78.jar:1.2.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:893) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1843) ~[hive-exec-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.cli.CLIService.init(CLIService.java:115) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.CompositeService.init(CompositeService.java:59) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:240) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:990) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1256) ~[hive-service-3.1.3.jar:3.1.3]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1100) ~[hive-service-3.1.3.jar:3.1.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
at org.apache.hadoop.util.RunJar.run(RunJar.java:318) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.2.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.2.jar:?]
Is this the reason? (I didn't have time to read the code in detail. I just guessed that the cache expired and I couldn't read the file.)
Sungwoo Park
Jul 26, 2022, 4:32:11 AM7/26/22
to MR3
Not sure why you get the error. Normally you should find these files under Ranger policy cache directory:
hive@hiveserver2-59bd84cf58-cg6dl:/opt/mr3-run/hiveserver2-ranger-policycache$ ls
hiveServer2_BLUE_hive.json hiveServer2_BLUE_hive_roles.json hive_audit_solr_spool
hiveServer2_BLUE_hive.json hiveServer2_BLUE_hive_roles.json hive_audit_solr_spool
The policy cache directory is specified with ranger.plugin.hive.policy.cache.dir in ranger-hive-security.xml, e.g.:
<property>
<name>ranger.plugin.hive.policy.cache.dir</name>
<value>/opt/mr3-run/hiveserver2-ranger-policycache</value>
</property>
<name>ranger.plugin.hive.policy.cache.dir</name>
<value>/opt/mr3-run/hiveserver2-ranger-policycache</value>
</property>
Configuring Ranger can be a tough task because there is not much documentation about it. For this reason, we maintain Ranger configurations only for Hive-MR3 on Kubernetes.
Cheers,
--- Sungwoo
Carol Chapman
Jul 27, 2022, 1:20:44 AM7/27/22
to MR3
Okay,
I'll first solve the error I encountered in my startup, and then observe it for a period of time
Reply all
Reply to author
Forward
0 new messages