tsdb dying with failure to connect to -ROOT- region

[Chris]

After a reboot...

[root@XXXXXXXXXX opentsdb]./build/tsdb tsd --port=4242 --staticroot=build/staticroot --cachedir=/tmp/tsd --zkquorum=localhost

2013-07-23 10:22:30,681 INFO  [main] TSDMain: Starting.

2013-07-23 10:22:30,690 INFO  [main] TSDMain: net.opentsdb built at revision da56730 (MINT)

2013-07-23 10:22:30,690 INFO  [main] TSDMain: Built on 2012/12/19 02:52:38 +0000 by root@hadoop-ord:/usr/local/src/opentsdb/build

2013-07-23 10:22:30,893 INFO  [main] ZooKeeper: Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT

2013-07-23 10:22:30,893 INFO  [main] ZooKeeper: Client environment:host.name=hadoop-ord.dcasolutions.com

2013-07-23 10:22:30,894 INFO  [main] ZooKeeper: Client environment:java.version=1.7.0_25

2013-07-23 10:22:30,894 INFO  [main] ZooKeeper: Client environment:java.vendor=Oracle Corporation

2013-07-23 10:22:30,894 INFO  [main] ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_25/jre

2013-07-23 10:22:30,894 INFO  [main] ZooKeeper: Client environment:java.class.path=/usr/local/src/opentsdb/build/third_party/hbase/asynchbase-1.4.0.jar:/usr/local/src/opentsdb/build/third_party/guava/guava-12.0.jar:/usr/local/src/opentsdb/build/third_party/slf4j/jcl-over-slf4j-1.6.4.jar:/usr/local/src/opentsdb/build/third_party/slf4j/log4j-over-slf4j-1.6.4.jar:/usr/local/src/opentsdb/build/third_party/logback/logback-classic-1.0.0.jar:/usr/local/src/opentsdb/build/third_party/logback/logback-core-1.0.0.jar:/usr/local/src/opentsdb/build/third_party/netty/netty-3.5.9.Final.jar:/usr/local/src/opentsdb/build/third_party/slf4j/slf4j-api-1.6.4.jar:/usr/local/src/opentsdb/build/third_party/suasync/suasync-1.3.0.jar:/usr/local/src/opentsdb/build/third_party/zookeeper/zookeeper-3.3.4.jar:/usr/local/src/opentsdb/build/tsdb-1.1.0.jar:/usr/local/src/opentsdb/build/../src

2013-07-23 10:22:30,894 INFO  [main] ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2013-07-23 10:22:30,894 INFO  [main] ZooKeeper: Client environment:java.io.tmpdir=/tmp

2013-07-23 10:22:30,895 INFO  [main] ZooKeeper: Client environment:java.compiler=<NA>

2013-07-23 10:22:30,895 INFO  [main] ZooKeeper: Client environment:os.name=Linux

2013-07-23 10:22:30,895 INFO  [main] ZooKeeper: Client environment:os.arch=amd64

2013-07-23 10:22:30,895 INFO  [main] ZooKeeper: Client environment:os.version=2.6.32-358.14.1.el6.x86_64

2013-07-23 10:22:30,895 INFO  [main] ZooKeeper: Client environment:user.name=root

2013-07-23 10:22:30,895 INFO  [main] ZooKeeper: Client environment:user.home=/root

2013-07-23 10:22:30,895 INFO  [main] ZooKeeper: Client environment:user.dir=/usr/local/src/opentsdb

2013-07-23 10:22:30,897 INFO  [main] ZooKeeper: Initiating client connection, connectString=localhost sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4638296

2013-07-23 10:22:30,916 INFO  [main-SendThread()] ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181

2013-07-23 10:22:30,920 INFO  [main] HBaseClient: Need to find the -ROOT- region

2013-07-23 10:22:30,928 INFO  [main-SendThread(localhost:2181)] ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session

2013-07-23 10:22:30,939 INFO  [main-SendThread(localhost:2181)] ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1400c0f67c6000e, negotiated timeout = 5000

2013-07-23 10:22:30,960 INFO  [main-EventThread] HBaseClient: Connecting to -ROOT- region @ 127.0.0.1:50821

2013-07-23 10:22:31,056 INFO  [main-EventThread] ZooKeeper: Session: 0x1400c0f67c6000e closed

2013-07-23 10:22:31,056 INFO  [main-EventThread] ClientCnxn: EventThread shut down

2013-07-23 10:22:31,081 INFO  [New I/O  worker #5] HBaseClient: Added client for region RegionInfo(table=".META.", region_name=".META.,,1", stop_key=""), which was added to the regions cache.  Now we know that RegionClient@689224072(chan=[id: 0xa7122fd7, /127.0.0.1:38232 => /127.0.0.1:50821], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) is hosting 1 region.

2013-07-23 10:22:31,091 INFO  [main] HBaseClient: Lost connection with the -ROOT- region

Exception in thread "main" java.lang.RuntimeException: Initialization failed

at net.opentsdb.tools.TSDMain.main(TSDMain.java:148)

Caused by: org.hbase.async.TableNotFoundException: "tsdb"

at org.hbase.async.HBaseClient$7.call(HBaseClient.java:1717)

at org.hbase.async.HBaseClient$7.call(HBaseClient.java:1714)

at com.stumbleupon.async.Deferred.doCall(Deferred.java:1261)

at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1240)

at com.stumbleupon.async.Deferred.callback(Deferred.java:988)

at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:450)

at org.hbase.async.RegionClient.decode(RegionClient.java:1185)

at org.hbase.async.RegionClient.decode(RegionClient.java:82)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:502)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:437)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)

at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1008)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)

at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2430)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:472)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:333)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:724)

hbck finds no anomalies in the hbase instance.

Any pointers to how to troubleshoot this will be greatly welcomed!

Thanks!

[ManOLamancha]

On Tuesday, July 23, 2013 11:26:12 AM UTC-4, Christopher Couples wrote:

Caused by: org.hbase.async.TableNotFoundException: "tsdb"

It looks like your 'tsdb' table is missing. You may just need to re-create the table.

[Chris Couples]

It helps when you realize that the ${USER} variable referenced within hbase-site.xml is the user starting hbase; hence, when I started hbase as root, it started fine, but opentsdb couldn't find its tables within the h aar-root tablespace (they were in hbase-hbase). Once this was corrected, all was well with that little corner of the world.

<facepalm>

thanks!

Chris 

-----

Chris Couples

...sent from my mobile, please excuse any brevity

On Jul 25, 2013, at 14:59, "Chris Larsen" <clars...@gmail.com> wrote:

Hi, Your message only went to me, not the group :)

 

On restart it sounds like your HBase master server lost the location of it’s files and started anew. If you can find the old files (maybe a disk wasn’t mounted, or HDFS is down, etc) then you should be able to get your old data back. If you were using HDFS, check the logs to see if the FS was reformatted or couldn’t load some of the HBase files.

 

From: Chris Couples [mailto:ccou...@gmail.com]
Sent: Thursday, July 25, 2013 12:54 PM
To: ManOLamancha
Subject: Re: tsdb dying with failure to connect to -ROOT- region

 

And all my time-series data - is that now gone?

 

Ugh....

 

Thanks for your help, and contributions to this project!

 

Chris 

-----

Chris Couples

 

...sent from my mobile, please excuse any brevity

[Ashish Kumar]

Hi,

I am also facing the same issue, Can you please tell me how did you resolve the issue?

I checked my ${USER} variable it is showing me my username not root.

[Gursimar Singh]

Hi Ashish,

I am also facing the same issue. Did you find the solution 

Thanks

[Jonathan Creasy]

Have you created the tsdb tables?

--
You received this message because you are subscribed to the Google Groups "OpenTSDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opentsdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/opentsdb/172716d0-e1d5-48aa-8895-9b4308fcb35e%40googlegroups.com.