Has anyone successfully implemented a secure coordinator with presto-yarn?
rockcl...@gmail.com
The server comes up fine, however, when I connect to the coordinator via the CLI, the server throws the following error.
Guessing I'm doing something wrong, but I'm beginning to wonder. The truststore, keystore and keytabs appear correct... The configs appear correct as well (listed below).
Thanks,
Rob
java.lang.RuntimeException: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.facebook.presto.hive.authentication.KerberosAuthentication.getSubject(KerberosAuthentication.java:67)
at com.facebook.presto.hive.authentication.KerberosHadoopAuthentication.getUserGroupInformation(KerberosHadoopAuthentication.java:51)
at com.facebook.presto.hive.authentication.CachingKerberosHadoopAuthentication.refreshUgi(CachingKerberosHadoopAuthentication.java:56)
at com.facebook.presto.hive.authentication.CachingKerberosHadoopAuthentication.getUserGroupInformation(CachingKerberosHadoopAuthentication.java:48)
at com.facebook.presto.hive.authentication.KerberosHiveMetastoreAuthentication.authenticate(KerberosHiveMetastoreAuthentication.java:79)
at com.facebook.presto.hive.thrift.Transport.create(Transport.java:36)
at com.facebook.presto.hive.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:51)
at com.facebook.presto.hive.StaticHiveCluster.createMetastoreClient(StaticHiveCluster.java:74)
at com.facebook.presto.hive.metastore.ThriftHiveMetastore.lambda$getAllTables$2(ThriftHiveMetastore.java:149)
at com.facebook.presto.hive.metastore.HiveMetastoreApiStats.lambda$wrap$0(HiveMetastoreApiStats.java:42)
at com.facebook.presto.hive.metastore.ThriftHiveMetastore.lambda$getAllTables$4(ThriftHiveMetastore.java:166)
at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:137)
at com.facebook.presto.hive.metastore.ThriftHiveMetastore.getAllTables(ThriftHiveMetastore.java:165)
at com.facebook.presto.hive.metastore.BridgingHiveMetastore.getAllTables(BridgingHiveMetastore.java:80)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadAllTables(CachingHiveMetastore.java:308)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.access$200(CachingHiveMetastore.java:59)
at com.facebook.presto.hive.metastore.CachingHiveMetastore$3.load(CachingHiveMetastore.java:125)
at com.facebook.presto.hive.metastore.CachingHiveMetastore$3.load(CachingHiveMetastore.java:120)
at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:189)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.get(CachingHiveMetastore.java:246)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.getAllTables(CachingHiveMetastore.java:302)
at com.facebook.presto.hive.HiveMetadata.listTables(HiveMetadata.java:308)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.listTables(ClassLoaderSafeConnectorMetadata.java:129)
at com.facebook.presto.metadata.MetadataManager.listTables(MetadataManager.java:382)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.getTablesList(InformationSchemaPageSourceProvider.java:204)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.buildTables(InformationSchemaPageSourceProvider.java:186)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.getInformationSchemaTable(InformationSchemaPageSourceProvider.java:138)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.getInternalTable(InformationSchemaPageSourceProvider.java:129)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.createPageSource(InformationSchemaPageSourceProvider.java:87)
at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.createSourceIfNecessary(ScanFilterAndProjectOperator.java:292)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.isFinished(ScanFilterAndProjectOperator.java:180)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at com.facebook.presto.hive.authentication.KerberosAuthentication.getSubject(KerberosAuthentication.java:63)
... 46 more
BTW: here's my appConfig.json.
{
"schema": "http://example.org/specification/v2.0.0",
"metadata": {
},
"global": {
"site.global.app_user": "yarn",
"site.global.user_group": "hadoop",
"site.global.data_dir": "/home/presto/data",
"site.global.config_dir": "/home/presto/lib/presto/etc",
"site.global.app_name": "presto-server-0.152",
"site.global.app_pkg_plugin": "${AGENT_WORK_ROOT}/app/definition/package/plugins/",
"site.global.singlenode": "true",
"site.global.coordinator_host": "${COORDINATOR_HOST}",
"site.global.presto_query_max_memory": "50GB",
"site.global.presto_query_max_memory_per_node": "600MB",
"site.global.presto_server_port": "8080",
"site.global.additional_config_properties":"['http.server.authentication.enabled=true','http.server.authentication.krb5.service-name=presto','http.server.authentication.krb5.keytab=/etc/keytabs/presto.keytab','http.authentication.krb5.config=/etc/krb5.conf','http-server.https.enabled=true','http-server.https.port=7778','http-server.https.keystore.path=/home/presto/keystore.jks','http-server.https.keystore.key=password']",
"site.global.plugin": "{'hive-cdh5': ['hive-serde-1.0.jar']}",
"site.global.catalog": "{'tpch': ['connector.name=tpch'], 'hive': ['connector.name=hive-cdh5','hive.metastore.uri=thrift://cms01.hadoop.test.com:9083','hive.metastore.authentication.type=KERBEROS','hive.metastore.service.principal=hive/_HO...@HADOOP.TEST.COM','hive.metastore.client.principal=pre...@HADOOP.TEST.COM','hive.metastore.client.keytab=/etc/keytabs/presto.keytab','hive.hdfs.authentication.type=KERBEROS','hive.hdfs.presto.principal=pre...@HADOOP.TEST.COM','hive.hdfs.presto.keytab=/etc/keytabs/presto.keytab','hive.config.resources=/opt/cloudera/hadoop-conf/core-site.xml,/opt/cloudera/hadoop-conf/hdfs-site.xml,/opt/cloudera/hadoop-conf/hive-site.xml']}",
"site.global.jvm_args": "['-server', '-Xmx1024M', '-XX:+UseG1GC', '-XX:G1HeapRegionSize=32M', '-XX:+UseGCOverheadLimit', '-XX:+ExplicitGCInvokesConcurrent', '-XX:+HeapDumpOnOutOfMemoryError', '-XX:OnOutOfMemoryError=kill -9 %p']",
"application.def": ".slider/package/PRESTO/presto-yarn-package-1.4-SNAPSHOT-0.152.zip",
"java_home": "/usr/lib/jvm/java"
},
"components": {
"slider-appmaster": {
"jvm.heapsize": "128M"
}
},
"coordinator": {
"http.server.authentication.enabled" : "true"
}
}
Here's my CLI wrapper:
#!/bin/bash
./presto-cli-*-executable.jar \
--server https://data03.hadoop.test.com:7778 \
--enable-authentication \
--krb5-config-path /etc/krb5.conf \
--krb5-principal pre...@HADOOP.TEST.COM \
--krb5-keytab-path /etc/keytabs/presto.keytab \
--krb5-remote-service-name presto \
--keystore-path /home/presto/keystore.jks \
--keystore-password password \
--catalog hive \
--debug \
--schema default
Anu Sudarsan
"site.global.catalog": "{'tpch': ['connector.name=tpch'], 'hive': ['connector.name=hive-cdh5','hive.metastore.uri=thrift://cms01.hadoop.test.com:9083','hive.metastore.authentication.type=KERBEROS','hive.metastore.service.principal=hive/_HO...@HADOOP.TEST.COM','hive.metastore.client.principal=pre...@HADOOP.TEST.COM','hive.metastore.client.keytab=/etc/keytabs/presto.keytab','hive.hdfs.authentication.type=KERBEROS','hive.hdfs.presto.principal=pres...@HADOOP.TEST.COM','hive.hdfs.presto.keytab=/etc/keytabs/presto.keytab','hive.config.resources=/opt/cloudera/hadoop-conf/core-site.xml,/opt/cloudera/hadoop-conf/hdfs-site.xml,/opt/cloudera/hadoop-conf/hive-site.xml']}",
rockcl...@gmail.com
> "application.def": ".slider/package/PRESTO/presto-yarn-package-1.4-SNAPSHOT-0.152.zip",
> "java_home": "/usr/lib/jvm/java"
> },
> "components": {
> "slider-appmaster": {
> "jvm.heapsize": "128M"
> }
> },
> "coordinator": {
> "http.server.authentication.enabled" : "true"
> }
> }
>
> Here's my CLI wrapper:
> #!/bin/bash
> ./presto-cli-*-executable.jar \
> --server https://data03.hadoop.test.com:7778 \
> --enable-authentication \
> --krb5-config-path /etc/krb5.conf \
> --krb5-principal pre...@HADOOP.TEST.COM \
> --krb5-keytab-path /etc/keytabs/presto.keytab \
> --krb5-remote-service-name presto \
> --keystore-path /home/presto/keystore.jks \
> --keystore-password password \
> --catalog hive \
> --debug \
> --schema default
For clarity, I do have the metastore authentication working with presto-yarn. It's only when I also add authentication to the coordinator, that it breaks.
To me, there's no point to having all of the downstream authentication in place, if the coordinator is wide open, allowing any client to submit a query without authentication.
Note that I also have stood up the non-yarn version and am running it in production, however, we really wanted to limit resources via our yarn implementation, which is why I'm working on this migration.
Thanks,
Rob