OpenTSDB with Kerberos: Cannot renew TGT with kinit -R

85 views
Skip to first unread message

ClusterAdmin

unread,
Jun 16, 2017, 8:24:58 AM6/16/17
to OpenTSDB

I'm using OpenTSDB in a kerberized cluster. I start OpenTSDB as root using


  1. CLASSPATH=$CLASSPATH:/home/applications/opentsdb/conf/ JVMARGS="${JVMARGS} -enableassertions -enablesystemassertions -Djava.security.auth.login.config=/home/applications/opentsdb/conf/opentsdb_client_jaas.conf" /home/applications/opentsdb/opentsdb-2.3.0/build/tsdb tsd --config /home/applications/opentsdb/conf/opentsdb.conf


The jaas config file looks like this:

  1. Client {
  2. com.sun.security.auth.module.Krb5LoginModule required debug=false
  3. renewTGT=true
  4. useKeyTab=true
  5. keyTab="/etc/security/keytabs/opentsdb.service.keytab"
  6. principal="opentsdb/host.c...@XXX.YYY.COM"
  7. useTicketCache=true;
  8. };


Everything starts just fine and in the OpenTSDB log file I see:


  1. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #3async.auth.KerberosClientAuthProvider
  2. Client will use GSSAPI as SASL mechanism.
  3. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #3async.auth.KerberosClientAuthProvider
  4. Connecting to hbase/host.cluster@XXX.YYY.COM
  5. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.HBaseClient
  6. Added client for region RegionInfo(table="tsdb", region_name="tsdb,,1497401874292.983451b817366a624c42c20e7c91af67.", stop_key="\x0B\x00\t\xD7S2Q"), which was added to the regions cache.  Now we know that RegionClient@785572588(chan=null, #pending_rpcs=0, #batched=0, #rpcs_inflight=0) is hosting 1 region.
  7. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #2async.auth.KerberosClientAuthProvider
  8. Client will use GSSAPI as SASL mechanism.
  9. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #2async.auth.KerberosClientAuthProvider
  10. Connecting to hbase/host.cluster@XXX.YYY.COM
  11. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.HBaseClient
  12. Added client for region RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1482497591937.0049eec9a851bc64e12ed2a0540192eb.", stop_key=""), which was added to the regions cache.  Now we know that RegionClient@599240979(chan=null, #pending_rpcs=0, #batched=0, #rpcs_inflight=0) is hosting 1 region.
  13. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.SecureRpcHelper96
  14. SASL client context established. Negotiated QoP: auth on for: RegionClient@159145664(chan=null, #pending_rpcs=2, #batched=0, #rpcs_inflight=0)
  15. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.RegionClient
  16. Initialized security helper: org.hbase.async.SecureRpcHelper96@4ce85bd8 for region client: RegionClient@159145664(chan=null, #pending_rpcs=2, #batched=0, #rpcs_inflight=0)
  17. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.auth.KerberosClientAuthProvider
  18. Client will use GSSAPI as SASL mechanism.
  19. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.auth.KerberosClientAuthProvider
  20. Connecting to hbase/host.cluster@XXX.YYY.COM
  21. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.auth.Login
  22. Initialized kerberos login context
  23. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.auth.Login
  24. Scheduled ticket renewal in 29266667 ms
  25. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.auth.Login
  26. TGT expires:                  Fri Jun 16 09:55:12 CEST 2017
  27. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.auth.Login
  28. TGT valid starting at:        Thu Jun 15 23:55:12 CEST 2017
  29. Thu Jun 15 23:55:12 GMT+200 2017INFOAsyncHBase I/O Worker #1async.auth.Login
  30. Successfully logged in


The TGT is granted for 10 hours. OpenTSDB says that it will try and renew the TGT in a little over 8 hours. When it does try and renew the TGT I see the following:


  1. Thu Jun 15 06:26:39 GMT+200 2017ERRORAsyncHBase Timer HBaseClient #1async.auth.Login
  2. Failed to renew ticketjava.lang.RuntimeException: Could not renew TGT due to problem running shell command: '/usr/bin/kinit -R';
  3. at org.hbase.async.auth.Login.refreshTicketCache(Login.java:340) ~[asynchbase-1.7.2.jar:na]
  4. at org.hbase.async.auth.Login.access$100(Login.java:61) ~[asynchbase-1.7.2.jar:na]
  5. at org.hbase.async.auth.Login$TicketRenewalTask.run(Login.java:386) ~[asynchbase-1.7.2.jar:na]
  6. at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556) [netty-3.9.4.Final.jar:na]
  7. at org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632) [netty-3.9.4.Final.jar:na]
  8. at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369) [netty-3.9.4.Final.jar:na]
  9. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]
  10. at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
  11. Caused by: org.apache.zookeeper.Shell$ExitCodeException: kinit: No credentials cache found (filename: /tmp/krb5cc_0) while renewing credentials
  12. at org.apache.zookeeper.Shell.runCommand(Shell.java:225) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
  13. at org.apache.zookeeper.Shell.run(Shell.java:152) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
  14. at org.apache.zookeeper.Shell$ShellCommandExecutor.execute(Shell.java:345) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
  15. at org.apache.zookeeper.Shell.execCommand(Shell.java:431) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
  16. at org.apache.zookeeper.Shell.execCommand(Shell.java:414) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
  17. at org.hbase.async.auth.Login.refreshTicketCache(Login.java:338) ~[asynchbase-1.7.2.jar:na]
  18. ... 7 common frames omitted


This part:


  1. Caused by: org.apache.zookeeper.Shell$ExitCodeException: kinit: No credentials cache found (filename: /tmp/krb5cc_0) while renewing credentials

leads me to think that it's trying to renew the TGT for the root user and isn't using the OpenTSDB keytab files. How can I get this to work properly?


Can I set


  1. useTicketCache-false;


in the jaas file? I do not have an OpenTSDB user on the cluster, only the service principals exist in the AD.

Message has been deleted

IDerr

unread,
Jun 26, 2017, 7:50:16 PM6/26/17
to OpenTSDB
I think you can, 


Like you said, he is setting the useTicketCache to false to prevent this.

Regards,
IDerr

Le vendredi 16 juin 2017 14:24:58 UTC+2, ClusterAdmin a écrit :

I'm using OpenTSDB in a kerberized cluster. I start OpenTSDB as root using


  1. CLASSPATH=$CLASSPATH:/home/applications/opentsdb/conf/ JVMARGS="${JVMARGS} -enableassertions -enablesystemassertions -Djava.security.auth.login.config=/home/applications/opentsdb/conf/opentsdb_client_jaas.conf" /home/applications/opentsdb/opentsdb-2.3.0/build/tsdb tsd --config /home/applications/opentsdb/conf/opentsdb.conf


The jaas config file looks like this:

  1. Client {
  2. com.sun.security.auth.module.Krb5LoginModule required debug=false
  3. renewTGT=true
  4. useKeyTab=true
  5. keyTab="/etc/security/keytabs/opentsdb.service.keytab"
  1. principal="opentsdb/host.clu...@XXX.YYY.COM"
  2. useTicketCache=true;
  3. };
Reply all
Reply to author
Forward
0 new messages