Coordinator authentication with presto-yarn

[rockclim...@gmail.com]

Hello,

Does anyone have a sample appConfig.json file that implements a secure coordinator?  I see examples with a non-yarn implementation, which uses config.properties, but nothing for a yarn implementation which uses appConfig.json.  I've tried to port the below configuration to the appConfig.json format, with no success.

Thanks,

Rob

Non-yarn Example:

config.properties

Kerberos authentication is configured in the coordinator node’s config.properties file. The entries that need to be added are listed below.

http.server.authentication.enabled=true

http.server.authentication.krb5.service-name=presto
http.server.authentication.krb5.keytab=/etc/presto/presto.keytab
http.authentication.krb5.config=/etc/krb5.conf

http-server.https.enabled=true
http-server.https.port=7778

http-server.https.keystore.path=/etc/presto_keystore.jks
http-server.https.keystore.key=keystore_password

[Akshat Nair]

Try adding them to site.global.additional_config_properties

[rockcl...@gmail.com]

Thanks. Using site.global.additional_config_properties almost works..... The server comes up fine, however, when I connect to the coordinator via the CLI, the server throws the following error:

java.lang.RuntimeException: javax.security.auth.login.LoginException: Unable to obtain password from user

at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.facebook.presto.hive.authentication.KerberosAuthentication.getSubject(KerberosAuthentication.java:67)
at com.facebook.presto.hive.authentication.KerberosHadoopAuthentication.getUserGroupInformation(KerberosHadoopAuthentication.java:51)
at com.facebook.presto.hive.authentication.CachingKerberosHadoopAuthentication.refreshUgi(CachingKerberosHadoopAuthentication.java:56)
at com.facebook.presto.hive.authentication.CachingKerberosHadoopAuthentication.getUserGroupInformation(CachingKerberosHadoopAuthentication.java:48)
at com.facebook.presto.hive.authentication.KerberosHiveMetastoreAuthentication.authenticate(KerberosHiveMetastoreAuthentication.java:79)
at com.facebook.presto.hive.thrift.Transport.create(Transport.java:36)
at com.facebook.presto.hive.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:51)
at com.facebook.presto.hive.StaticHiveCluster.createMetastoreClient(StaticHiveCluster.java:74)
at com.facebook.presto.hive.metastore.ThriftHiveMetastore.lambda$getAllTables$2(ThriftHiveMetastore.java:149)
at com.facebook.presto.hive.metastore.HiveMetastoreApiStats.lambda$wrap$0(HiveMetastoreApiStats.java:42)
at com.facebook.presto.hive.metastore.ThriftHiveMetastore.lambda$getAllTables$4(ThriftHiveMetastore.java:166)
at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:137)
at com.facebook.presto.hive.metastore.ThriftHiveMetastore.getAllTables(ThriftHiveMetastore.java:165)
at com.facebook.presto.hive.metastore.BridgingHiveMetastore.getAllTables(BridgingHiveMetastore.java:80)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadAllTables(CachingHiveMetastore.java:308)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.access$200(CachingHiveMetastore.java:59)
at com.facebook.presto.hive.metastore.CachingHiveMetastore$3.load(CachingHiveMetastore.java:125)
at com.facebook.presto.hive.metastore.CachingHiveMetastore$3.load(CachingHiveMetastore.java:120)
at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:189)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.get(CachingHiveMetastore.java:246)
at com.facebook.presto.hive.metastore.CachingHiveMetastore.getAllTables(CachingHiveMetastore.java:302)
at com.facebook.presto.hive.HiveMetadata.listTables(HiveMetadata.java:308)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.listTables(ClassLoaderSafeConnectorMetadata.java:129)
at com.facebook.presto.metadata.MetadataManager.listTables(MetadataManager.java:382)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.getTablesList(InformationSchemaPageSourceProvider.java:204)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.buildTables(InformationSchemaPageSourceProvider.java:186)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.getInformationSchemaTable(InformationSchemaPageSourceProvider.java:138)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.getInternalTable(InformationSchemaPageSourceProvider.java:129)
at com.facebook.presto.connector.informationSchema.InformationSchemaPageSourceProvider.createPageSource(InformationSchemaPageSourceProvider.java:87)
at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.createSourceIfNecessary(ScanFilterAndProjectOperator.java:292)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.isFinished(ScanFilterAndProjectOperator.java:180)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at com.facebook.presto.hive.authentication.KerberosAuthentication.getSubject(KerberosAuthentication.java:63)
... 46 more

I've tried everything. The truststore is correct, the keystore is correct, the keytabs are correct... The configs appear to be correct. Everything checks out. Has anyone out there actually implemented a secure coordinator with presto-yarn?

BTW: here's my appConfig.json.

{
"schema": "http://example.org/specification/v2.0.0",
"metadata": {
},
"global": {
"site.global.app_user": "yarn",
"site.global.user_group": "hadoop",
"site.global.data_dir": "/home/presto/data",
"site.global.config_dir": "/home/presto/lib/presto/etc",
"site.global.app_name": "presto-server-0.152",
"site.global.app_pkg_plugin": "${AGENT_WORK_ROOT}/app/definition/package/plugins/",
"site.global.singlenode": "true",
"site.global.coordinator_host": "${COORDINATOR_HOST}",
"site.global.presto_query_max_memory": "50GB",
"site.global.presto_query_max_memory_per_node": "600MB",
"site.global.presto_server_port": "8080",
"site.global.additional_config_properties":"['http.server.authentication.enabled=true','http.server.authentication.krb5.service-name=presto','http.server.authentication.krb5.keytab=/etc/keytabs/presto.keytab','http.authentication.krb5.config=/etc/krb5.conf','http-server.https.enabled=true','http-server.https.port=7778','http-server.https.keystore.path=/home/presto/keystore.jks','http-server.https.keystore.key=password']",
"site.global.plugin": "{'hive-cdh5': ['hive-serde-1.0.jar']}",
"site.global.catalog": "{'tpch': ['connector.name=tpch'], 'hive': ['connector.name=hive-cdh5','hive.metastore.uri=thrift://cms01.hadoop.test.com:9083','hive.metastore.authentication.type=KERBEROS','hive.metastore.service.principal=hive/_HO...@HADOOP.TEST.COM','hive.metastore.client.principal=pre...@HADOOP.TEST.COM','hive.metastore.client.keytab=/etc/keytabs/presto.keytab','hive.hdfs.authentication.type=KERBEROS','hive.hdfs.presto.principal=pre...@HADOOP.TEST.COM','hive.hdfs.presto.keytab=/etc/keytabs/presto.keytab','hive.config.resources=/opt/cloudera/hadoop-conf/core-site.xml,/opt/cloudera/hadoop-conf/hdfs-site.xml,/opt/cloudera/hadoop-conf/hive-site.xml']}",

"site.global.jvm_args": "['-server', '-Xmx1024M', '-XX:+UseG1GC', '-XX:G1HeapRegionSize=32M', '-XX:+UseGCOverheadLimit', '-XX:+ExplicitGCInvokesConcurrent', '-XX:+HeapDumpOnOutOfMemoryError', '-XX:OnOutOfMemoryError=kill -9 %p']",

"application.def": ".slider/package/PRESTO/presto-yarn-package-1.4-SNAPSHOT-0.152.zip",
"java_home": "/usr/lib/jvm/java"
},
"components": {
"slider-appmaster": {
"jvm.heapsize": "128M"
}
},

"coordinator": {
"http.server.authentication.enabled" : "true"
}

}


Here's my CLI wrapper:
#!/bin/bash

./presto-cli-*-executable.jar \
--server https://data03.hadoop.test.com:7778 \
--enable-authentication \
--krb5-config-path /etc/krb5.conf \
--krb5-principal pre...@HADOOP.TEST.COM \
--krb5-keytab-path /etc/keytabs/presto.keytab \
--krb5-remote-service-name presto \
--keystore-path /home/presto/keystore.jks \
--keystore-password password \
--catalog hive \
--debug \
--schema default