TSDB tables lost every now and then

[Rosy Gupta]

I have inserted the data into opentsdb successfully using HTTP API. However, every now and then when I restart the tsdb, the tables are lost. It says the hbase has lost connection with the ROOT region and that it cannot find 'tsdb' table.

How should I resolve this?

This is the error I'm getting:

INFO  [main] HBaseClient: Lost connection with the -ROOT- region

Exception in thread "main" java.lang.RuntimeException: Initialization failed

at net.opentsdb.tools.TSDMain.main(TSDMain.java:180)

Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:

at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)

at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)

at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)

at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82)

at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)

at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)

at com.stumbleupon.async.Deferred.access$300(Deferred.java:430)

at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)

at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)

at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)

at com.stumbleupon.async.Deferred.access$300(Deferred.java:430)

at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)

at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)

at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)

at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313)

at com.stumbleupon.async.Deferred.doCall(Deferred.java:1284)

at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)

at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)

at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:506)

at org.hbase.async.RegionClient.decode(RegionClient.java:1351)

at org.hbase.async.RegionClient.decode(RegionClient.java:89)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1082)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2677)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.hbase.async.TableNotFoundException: "tsdb"

at org.hbase.async.HBaseClient$7.call(HBaseClient.java:1926)

at org.hbase.async.HBaseClient$7.call(HBaseClient.java:1923)

at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)

[Lin]

Do you find your tsdb table in hbase shell via a "list" ? 

在 2015年8月12日星期三 UTC+8上午9:30:52，Rosy Gupta写道：

[Rosy Gupta]

No it shows 0 tables found!

[Pradeep Chhetri]

This must be because by default hbase stores data in /tmp and whenever you restart your machine, being tempfs, it is volatile filesystem. Hence you must be change the data.dir in hbase to persist your hbase data across reboots.

--

Pradeep Chhetri

In the world of Linux, who needs Windows and Gates...

[Rosy Gupta]

I haven't installed HBase as recommended here:

http://opentsdb.net/setup-hbase.html

Just followed routine installation. Does that make a difference?

N my hbase-site.xml config is:

http://pastie.org/10346380

The root dir is pointing to the hdfs. Where should the changes be made to change from tmpfs?

I remember having mentioned the tmpfs directory while installing opentsdb. Does that require change? It is:

http://pastie.org/10346388