Presto and Hive: column does not exist

384 views
Skip to first unread message

matthew....@gmail.com

unread,
Jun 26, 2017, 9:50:30 AM6/26/17
to Presto
I can't return any data from Hive. I am running CDH 5.11.0 and Presto 0.178. I thought it just wanted me to include all partition columns in the where clause (which was weird) but then I tracked a combo of partitions filters that returns data. In Hive, I get the data returned without error but in Presto and HMS I get an error.

Metadata commands like show create table and describe work in both Presto and Hive. I have Kerberos enabled for both Presto and Hive, as well as impersonation. This is a Avro table with partitions columns of product, procid_, enddt, and trialgroup. I get the same error when I create a simple text table with no partitions (a difference is that Presto and Hive report the same exception; instead of Presto throwing this useless exception that the password could not be obtained). If I remove any of the partition columns from the where clause then the HMS error changes to the highest level of the directory structure instead of reporting that $path doesn't exist.

Oh, I don't get an error if I do query with all partitions columns used but with filters that do not return any data.

The Hive code is validating a list of column names against the table.getCols but I haven't traced it back to see where the colNames List is coming from.

I am stumped on this one.

select mcsbsize, packetsize from ghl2im.ccb_testsequence where product = 'apk' and procid_ = '6400' and enddt = '20170102' and trialgroup = 'N' limit 12;
OK
NULL 192
NULL 25
NULL 26
NULL 19
NULL 25
NULL 25
NULL 25
NULL 27
NULL 21
NULL 21
NULL 46
NULL 17

select mcsbsize, packetsize from ghl2im.ccb_testsequence where product = 'apk' and procid_ = '6400' and enddt = '20170102' and trialgroup = 'N' limit 12;

Query 20170626_132920_00009_gw5ti, FAILED, 1 node
http://localhost:8889/query.html?20170626_132920_00009_gw5ti
Splits: 29 total, 11 done (37.93%)
CPU Time: 0.0s total, 0 rows/s, 0B/s, 100% active
Per Node: 0.0 parallelism, 0 rows/s, 0B/s
Parallelism: 0.0
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20170626_132920_00009_gw5ti failed: javax.security.auth.login.LoginException: Unable to obtain password from user

java.lang.RuntimeException: javax.security.auth.login.LoginException: Unable to obtain password from user

at com.google.common.base.Throwables.propagate(Throwables.java:240)
at com.facebook.presto.hive.authentication.KerberosAuthentication.getSubject(KerberosAuthentication.java:67)
at com.facebook.presto.hive.authentication.KerberosHadoopAuthentication.getUserGroupInformation(KerberosHadoopAuthentication.java:51)
at com.facebook.presto.hive.authentication.CachingKerberosHadoopAuthentication.refreshUgi(CachingKerberosHadoopAuthentication.java:56)
at com.facebook.presto.hive.authentication.CachingKerberosHadoopAuthentication.getUserGroupInformation(CachingKerberosHadoopAuthentication.java:48)
at com.facebook.presto.hive.authentication.ImpersonatingHdfsAuthentication.createProxyUser(ImpersonatingHdfsAuthentication.java:44)
at com.facebook.presto.hive.authentication.ImpersonatingHdfsAuthentication.doAs(ImpersonatingHdfsAuthentication.java:39)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:66)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:60)
at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:64)
at com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:157)
at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:87)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:56)
at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:259)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:303)
at com.facebook.presto.operator.Driver.lambda$processFor$6(Driver.java:234)
at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:537)
at com.facebook.presto.operator.Driver.processFor(Driver.java:229)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:623)
at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:463)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at com.facebook.presto.hive.authentication.KerberosAuthentication.getSubject(KerberosAuthentication.java:63)
... 23 more

2017-06-26 13:28:28,399 ERROR org.apache.hadoop.hive.metastore.ObjectStore: [pool-5-thread-103]: Error retrieving statistics via jdo
MetaException(message:Column $path doesn't exist in table ccb_testsequence in database ghl2im)
at org.apache.hadoop.hive.metastore.ObjectStore.validateTableCols(ObjectStore.java:6579)
at org.apache.hadoop.hive.metastore.ObjectStore.getMPartitionColumnStatistics(ObjectStore.java:6715)
at org.apache.hadoop.hive.metastore.ObjectStore.access$1300(ObjectStore.java:171)
at org.apache.hadoop.hive.metastore.ObjectStore$10.getJdoResult(ObjectStore.java:6648)
at org.apache.hadoop.hive.metastore.ObjectStore$10.getJdoResult(ObjectStore.java:6637)
at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionColumnStatisticsInternal(ObjectStore.java:6637)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionColumnStatistics(ObjectStore.java:6630)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
at com.sun.proxy.$Proxy7.getPartitionColumnStatistics(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_statistics_req(HiveMetaStore.java:4695)
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy9.get_partitions_statistics_req(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_statistics_req.getResult(ThriftHiveMetastore.java:11054)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_statistics_req.getResult(ThriftHiveMetastore.java:11038)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:735)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:730)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:730)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-06-26 13:28:28,399 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-5-thread-129]: 123: Cleaning up thread local RawStore...
2017-06-26 13:28:28,399 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [pool-5-thread-118]: <PERFLOG method=get_partitions_statistics_req from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>

matthew....@gmail.com

unread,
Jun 27, 2017, 9:08:22 PM6/27/17
to Presto, matthew....@gmail.com
I am pretty sure that my issue is that SSL is not working. I was not forcing HTTPS or using the username/password cli switches. I was getting a cert mismatch error with the *.realm.com certs I was using. I switched to full hostname for the coordinator and copied the coordinator jks to the worker. I also switched to the TD release and enabled internal SSL communication. I am no longer getting any SSL errors on the worker, but the coordinator is still having an issue. I'll post the error in a few.

matthew....@gmail.com

unread,
Jun 27, 2017, 9:22:50 PM6/27/17
to Presto, matthew....@gmail.com
This all I get with kerberos and ssl debugging enabled.

2017-06-27T21:15:43.247-0400 INFO http-client-shared-58 stdout Java config name: /etc/krb5.conf
2017-06-27T21:15:43.247-0400 INFO http-client-shared-58 stdout Loaded from Java config
2017-06-27T21:15:43.247-0400 INFO http-client-shared-58 stdout >>> KdcAccessibility: reset
2017-06-27T21:15:43.247-0400 INFO http-client-shared-58 stdout Looking for keys for: presto/_H...@HITACHIGST.GLOBAL
2017-06-27T21:15:43.259-0400 INFO http-client-shared-58 stdout Java config name: /etc/krb5.conf
2017-06-27T21:15:43.259-0400 INFO http-client-shared-58 stdout Loaded from Java config
2017-06-27T21:15:43.259-0400 INFO http-client-shared-58 stdout >>> KdcAccessibility: reset
2017-06-27T21:15:43.259-0400 INFO http-client-shared-58 stdout Looking for keys for: presto/_H...@HITACHIGST.GLOBAL
2017-06-27T21:15:43.260-0400 INFO main com.facebook.presto.server.PrestoServer ======== SERVER STARTED ========
2017-06-27T21:15:43.261-0400 ERROR Announcer-0 io.airlift.discovery.client.Announcer Cannot connect to discovery server for announce: Announcement failed for https://abo--pmstr01.hitachigst.global:8889
2017-06-27T21:15:43.261-0400 ERROR Announcer-0 io.airlift.discovery.client.Announcer Service announcement failed after 20.14ms. Next request will happen within 0.00s
...
2017-06-27T21:15:43.982-0400 WARN http-client-shared-59 com.facebook.presto.memory.RemoteNodeMemory Error fetching memory info from https://abo-ld-pmstr01.hitachigst.glol:8889/v1/memory: Failed to establish LoginContext for request https://abo-ld-pmstr01.hitachigst.global:8889/v1/memory

Reply all
Reply to author
Forward
0 new messages