Running Tephra on Kerberized environment

[adam.d...@bigdatapartnership.com]

Hi,

I'm currently trying to run the tephra server inside a HDP 2.3.1 (i.e. HBase 1.1) sandbox with Kerberos enabled. I have previously run the server successfully before Kerberos was activated. I see in the logs the following trace when starting the server; 

16:48:25.306 [ThriftRPCServer] ERROR c.c.t.distributed.TransactionService - Transaction manager aborted, stopping transaction service

Exception in thread "HDFSTransactionStateStorage STARTING" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException

: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]

        at com.google.common.base.Throwables.propagate(Throwables.java:160)

        at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:47)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)

        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2118)

        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)

        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)

        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)

        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)

        at co.cask.tephra.persist.HDFSTransactionStateStorage.startUp(HDFSTransactionStateStorage.java:108)

        at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43)

        ... 1 more

The issue appears to be clearly linked to the authentication method Tephra is using, the question is how can I change it? I have added the following lines to tephra-env.sh

export KERBEROS_OPTS="-Djava.security.krb5.realm=EXAMPLE.COM -Djava.security.krb5.kdc=<kdc domain> -Djava.security.auth.login.config=tephra-jaas.conf"

export OPTS="$OPTS $KERBEROS_OPTS -XX:+UseConcMarkSweepGC"

where the tephra-jaas.conf file contains;

Client {

  com.sun.security.auth.module.Krb5LoginModule required

  useKeyTab=true

  keyTab="/etc/security/tephra.keytab"

  storeKey=true

  useTicketCache=true

  principal="<tephra principal>"

};

where the Tephra principal has been created along with it's keytab. Prior to seeing the exception trace above, I get the following;

16:48:23.380 [main-SendThread(sandbox.hortonworks.com:2181)] DEBUG o.a.z.client.ZooKeeperSaslClient - JAAS loginContext is: Client

16:48:23.430 [main-SendThread(sandbox.hortonworks.com:2181)] WARN  o.a.z.client.ZooKeeperSaslClient - Could not login: the client is bei

ng asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the

 client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you 

still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if 

you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookee

per client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k

 -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manu

ally refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that y

our KDC host's clock is in sync with this host's clock.

I'm not sure if this causes the later Runtime/AccessControlException. I don't believe so as it appears to successfully connect to Zookeeper after this.

Any ideas would be much appreciated! 

Kind regards,

Adam

      

NOTICE AND DISCLAIMER

This email (including attachments) is confidential. If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.

Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United Kingdom
Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United Kingdom
Big Data Partnership Limited is a company registered in England & Wales with Company No 7904824

[adam.d...@bigdatapartnership.com]

Apologies, it seems the issue was with HBase configs not being refreshed after Kerberizing the sandbox.

I now have the Tephra server running and processing transactions! 

Regards,
Adam