Hue and Hiveserver2 running on different machines: NullPointerException on performing view Metastore Table

[vvi...@gmail.com]

Hello,

  I have installed Hue 3.5 on machine which is not part of Hadoop cluster.  I have configured Hue to point to the hiveserver2 on the cluster in hue.ini.

Metastore tables are listed but on clicking one of the table to view data, i get the following exception:

Hive Error

Bad status for request TFetchResultsReq(operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\x89\xc1\xc6\xfe\xd1f@4\x9b\x8c\xfef\xa6\x96\xba\xe3', guid='\x16\xd1\x12d\x10B@\xc6\x95\xd1\x15\xf1\xef\xeeU\xed')), orientation=0, maxRows=1000): TFetchResultsResp(status=TStatus(errorCode=None, errorMessage=None, sqlState=None, infoMessages=['*java.lang.NullPointerException:null:16:15', 'org.apache.hive.service.cli.RowBasedSet:<init>:RowBasedSet.java:40', 'org.apache.hive.service.cli.RowSetFactory:create:RowSetFactory.java:32', 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:336', 'org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:220', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:685', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:454', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:672', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:745'], statusCode=3), results=None, hasMoreRows=None)

I have done some searching on the web and found that HiveServer2 Impersonation should be enabled using the following config.

It still gives me the same error:

<property>

  <name>hive.server2.enable.impersonation</name>

  <description>Enable user impersonation for HiveServer2</description>

  <value>true</value>

</property>

The following is the extract of hiveserver2.log:

2015-09-24 17:38:21,056 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:21,226 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:21,247 INFO  [HiveServer2-Handler-Pool: Thread-41]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: USE default

2015-09-24 17:38:23,103 INFO  [HiveServer2-Handler-Pool: Thread-41]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed

2015-09-24 17:38:23,105 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=parse start=1443116301226 end=1443116303105 duration=1879 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,283 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,411 INFO  [HiveServer2-Handler-Pool: Thread-41]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: get_database: default

2015-09-24 17:38:23,412 INFO  [HiveServer2-Handler-Pool: Thread-41]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hive ip=unknown-ip-addr cmd=get_database: default

2015-09-24 17:38:23,441 INFO  [HiveServer2-Handler-Pool: Thread-41]: metadata.HiveUtils (HiveUtils.java:getMetaStoreAuthorizeProviderManagers(353)) - Adding metastore authorization provider: org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider

2015-09-24 17:38:23,607 INFO  [HiveServer2-Handler-Pool: Thread-41]: ql.Driver (Driver.java:compile(436)) - Semantic Analysis Completed

2015-09-24 17:38:23,607 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=semanticAnalyze start=1443116303283 end=1443116303607 duration=324 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,643 INFO  [HiveServer2-Handler-Pool: Thread-41]: ql.Driver (Driver.java:getSchema(240)) - Returning Hive schema: Schema(fieldSchemas:null, properties:null)

2015-09-24 17:38:23,644 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=doAuthorization from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,663 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=doAuthorization start=1443116303644 end=1443116303663 duration=19 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,664 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=compile start=1443116301056 end=1443116303664 duration=2608 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,672 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,673 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,673 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=acquireReadWriteLocks from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,687 INFO  [HiveServer2-Handler-Pool: Thread-41]: lockmgr.DbTxnManager (DbTxnManager.java:acquireLocks(108)) - Setting lock request transaction to 0 for queryId=hive_20150924173821_6f14ac16-5082-4350-9aad-b8b1dccee032

2015-09-24 17:38:23,688 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=acquireReadWriteLocks start=1443116303673 end=1443116303688 duration=15 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,690 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:23,691 INFO  [HiveServer2-Handler-Pool: Thread-41]: ql.Driver (Driver.java:execute(1328)) - Starting command(queryId=hive_20150924173821_6f14ac16-5082-4350-9aad-b8b1dccee032): USE default

2015-09-24 17:38:24,317 INFO  [HiveServer2-Handler-Pool: Thread-41]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(296)) - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/

2015-09-24 17:38:24,319 INFO  [HiveServer2-Handler-Pool: Thread-41]: hooks.ATSHook (ATSHook.java:<init>(84)) - Created ATS Hook

2015-09-24 17:38:24,320 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,327 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1443116304320 end=1443116304327 duration=7 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,327 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,328 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.security.authorization.plugin.DisallowTransformHook start=1443116304327 end=1443116304328 duration=1 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,336 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=TimeToSubmit start=1443116303673 end=1443116304336 duration=663 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,337 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,338 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,355 INFO  [HiveServer2-Handler-Pool: Thread-41]: ql.Driver (Driver.java:launchTask(1651)) - Starting task [Stage-0:DDL] in serial mode

2015-09-24 17:38:24,357 INFO  [HiveServer2-Handler-Pool: Thread-41]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: get_database: default

2015-09-24 17:38:24,357 INFO  [HiveServer2-Handler-Pool: Thread-41]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hive ip=unknown-ip-addr cmd=get_database: default

2015-09-24 17:38:24,386 INFO  [HiveServer2-Handler-Pool: Thread-41]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: get_database: default

2015-09-24 17:38:24,387 INFO  [HiveServer2-Handler-Pool: Thread-41]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hive ip=unknown-ip-addr cmd=get_database: default

2015-09-24 17:38:24,421 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=runTasks start=1443116304337 end=1443116304421 duration=84 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,422 INFO  [HiveServer2-Handler-Pool: Thread-41]: hooks.ATSHook (ATSHook.java:<init>(84)) - Created ATS Hook

2015-09-24 17:38:24,745 WARN  [HiveServer2-Handler-Pool: Thread-41]: conf.HiveConf (HiveConf.java:initialize(2762)) - HiveConf of name hive.server2.enable.impersonation does not exist

2015-09-24 17:38:24,750 INFO  [HiveServer2-Handler-Pool: Thread-41]: hook.HiveHook (HiveHook.java:<clinit>(110)) - Created Atlas Hook

2015-09-24 17:38:24,751 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,752 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1443116304751 end=1443116304752 duration=1 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,752 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,897 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook start=1443116304752 end=1443116304897 duration=145 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,897 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=Driver.execute start=1443116303690 end=1443116304897 duration=1207 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,901 INFO  [HiveServer2-Handler-Pool: Thread-41]: ql.Driver (SessionState.java:printInfo(951)) - OK

2015-09-24 17:38:24,901 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,902 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=releaseLocks start=1443116304901 end=1443116304901 duration=0 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,903 INFO  [HiveServer2-Handler-Pool: Thread-41]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=Driver.run start=1443116303672 end=1443116304903 duration=1231 from=org.apache.hadoop.hive.ql.Driver>

2015-09-24 17:38:24,903 INFO  [Atlas Logger 0]: hook.HiveHook (HiveHook.java:fireAndForget(170)) - Entered Atlas hook for hook type POST_EXEC_HOOK operation SWITCHDATABASE

2015-09-24 17:38:24,916 WARN  [HiveServer2-Handler-Pool: Thread-41]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error fetching results: 

java.lang.NullPointerException

at org.apache.hive.service.cli.RowBasedSet.<init>(RowBasedSet.java:40)

at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:32)

at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:336)

at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:220)

at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:685)

at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:454)

at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)

at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)

at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

I understand that this is due to problem with authorization since Hue is installed on machine different from the one where Hiveserver2 is running.

Please let me know what configuration am I missing here.

[Romain Rigaux]

Is beeline (HiveServer2 command working properly?)

You seem to have a funky hook there:

Entered Atlas hook for hook type POST_EXEC_HOOK operation SWITCHDATABASE

Looking at above logs the impersonation property seems to be not picked up too.

BTW you could give a try with the very latest Hue: http://gethue.com/hadoop-hue-3-on-hdp-installation-tutorial/

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

[vvi...@gmail.com]

Thanks for your reply Romain.

I have not tried beeline.  Will install beeline and make a test.

Does this property 

hive.server2.enable.doAs

have any impact as it is currently set to "false" in the hive-site.xml.

I will give a try with the latest version of Hue also (3.9).

Thanks again.

-Vijay

[Romain Rigaux]

Without the doAs enabled, it means that any user logged in, like bob, joe... will have their SQL queries sent as the hue user (so not good for real security, less a problem for just testing and without Sentry).