OpenTSDB not starting up - connection to HBase .Meta failing

[karan alang]

Hi All,

I've kerberized HDP 2.4 (HBase version - 1.1.2.2.4.0.0-169, openTSDB version - 2.2.1)
On starting OpenTSDB, i'm getting error shown below -

What i understand is that this is because of connection issues between HBase & OpenTSDB

In connecting to HBase .META, the connection is failing.
Any ideas on how to fix this ?

Here is the command used to start OpenTSDB
/build/tsdb tsd --zkbasedir=/hbase-secure --port=9999 --zkquorum=sandbox.hortonworks.com:2181 --cachedir=/tmp/tsd --staticroot=build/staticroot --auto-metric

Also attaching the hbase-site.xml, (the complete) opentsdb.log, opentsdb.conf

Appreciate your inputs !

-------------------- log --------------------------------------------------------------------

2016-12-14 18:56:00,365 INFO  [AsyncHBase I/O Worker #4] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@438f3a6
2016-12-14 18:56:00,383 INFO  [AsyncHBase I/O Worker #4-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error)
2016-12-14 18:56:00,390 INFO  [AsyncHBase I/O Worker #4-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session
2016-12-14 18:56:00,391 INFO  [AsyncHBase I/O Worker #4] HBaseClient: Need to find the .META. region
2016-12-14 18:56:00,392 WARN  [AsyncHBase I/O Worker #4] HBaseClient: Failed to schedule timer.  Ignore this if we're shutting down.
java.lang.IllegalStateException: cannot be started once stopped
    at org.jboss.netty.util.HashedWheelTimer.start(HashedWheelTimer.java:279) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.util.HashedWheelTimer.newTimeout(HashedWheelTimer.java:337) ~[netty-3.9.4.Final.jar:na]
    at org.hbase.async.HBaseClient.newTimeout(HBaseClient.java:924) [asynchbase-1.7.2.jar:na]
    at org.hbase.async.HBaseClient.handleNSRE(HBaseClient.java:2952) [asynchbase-1.7.2.jar:na]
    at org.hbase.async.RegionClient.failOrRetryRpcs(RegionClient.java:1212) [asynchbase-1.7.2.jar:na]
    at org.hbase.async.RegionClient.cleanup(RegionClient.java:1174) [asynchbase-1.7.2.jar:na]
    at org.hbase.async.RegionClient.channelDisconnected(RegionClient.java:1151) [asynchbase-1.7.2.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [netty-3.9.4.Final.jar:na]
    at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223) [asynchbase-1.7.2.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelHandler.channelDisconnected(SimpleChannelHandler.java:199) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:120) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]
    at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121) [asynchbase-1.7.2.jar:na]
    at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
2016-12-14 18:56:00,414 INFO  [AsyncHBase Timer HBaseClient #1-EventThread] TSDB: Completed shutting down the TSDB
Exception in thread "main" java.lang.RuntimeException: Initialization failed
    at net.opentsdb.tools.TSDMain.main(TSDMain.java:196)
Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:
    at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)
    at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)
    at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)
    at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82)
    at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
    at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
    at com.stumbleupon.async.Deferred.access$300(Deferred.java:430)
    at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)
    at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
    at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
    at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313)
    at com.stumbleupon.async.Deferred.doCall(Deferred.java:1284)
    at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
    at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)
    at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712)
2016-12-14 18:56:00,423 INFO  [AsyncHBase I/O Worker #4-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650234, negotiated timeout = 5000
    at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2058)
    at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920)
    at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944)
    at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927)
    at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
    at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
    at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)
    at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3645)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
Caused by: org.hbase.async.NonRecoverableException: Too many attempts: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 45, 117, 105, 100, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=11, timeout=0, hasTimedout=false)
    at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056)
    ... 9 more

[karan alang]

resending .. pls. let me know if anyone has encountered & fixed the issue (with OpenTSDB connectivity to HBase on kerberized cluster)

thanks,
Karan Alang

[Lei Chen]

I encountered similar problem when installing opentsdb 2.3.0 on top of hbase 1.1.2.2.3.6.2 from HDP-2.3.6
May I ask have you got the solution?