TSD starts up but immediately shuts down after less than 1s with no errors in log file

[Travis Keep]

When I start open tsdb via /etc/init.d/opentsdb start command, the TSD server quickly shuts down leaving no errors in the log file.

I am running OpenTSDB2.3.0 on top of HBASE 1.2.6.  The version of HBase I am using is the quick start version bundled with zookeeper.

attached are the logs that I see when starting open TSD. Although nothing remarkable shows up in the logs, TSD decides to shut down after less than 1s.  Any ideas or help as to what I can do to get OpenTSDB running would be greatly appreciated. Thanks in advance.

21:57:28.510 INFO  [TSDMain.main] - Starting.
21:57:28.515 INFO  [TSDMain.main] - net.opentsdb.tools 2.3.0 built at revision cac608a (MINT)
21:57:28.515 INFO  [TSDMain.main] - Built on 2017/01/01 00:10:03 +0000 by hobbes@clhbase:/home/hobbes/OFFICIAL/build
21:57:28.522 INFO  [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb.conf
21:57:28.591 INFO  [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb.conf
21:57:28.782 WARN  [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter
21:57:28.833 INFO  [ZooKeeper.logEnv] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
21:57:28.834 INFO  [ZooKeeper.logEnv] - Client environment:host.name=ip-100-122-3-120.us-east-1.ec2.aws.symcpe.net
21:57:28.835 INFO  [ZooKeeper.logEnv] - Client environment:java.version=1.7.0_121
21:57:28.835 INFO  [ZooKeeper.logEnv] - Client environment:java.vendor=Oracle Corporation
21:57:28.835 INFO  [ZooKeeper.logEnv] - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
21:57:28.835 INFO  [ZooKeeper.logEnv] - Client environment:java.class.path=/usr/share/opentsdb/*.jar:/usr/share/opentsdb:/usr/share/opentsdb/bin:/usr/share/opentsdb/lib/async-1.4.0.jar:/usr/share/opentsdb/lib/asynchbase-1.7.2.jar:/usr/share/opentsdb/lib/commons-jexl-2.1.1.jar:/usr/share/opentsdb/lib/commons-logging-1.1.1.jar:/usr/share/opentsdb/lib/commons-math3-3.4.1.jar:/usr/share/opentsdb/lib/guava-18.0.jar:/usr/share/opentsdb/lib/jackson-annotations-2.4.3.jar:/usr/share/opentsdb/lib/jackson-core-2.4.3.jar:/usr/share/opentsdb/lib/jackson-databind-2.4.3.jar:/usr/share/opentsdb/lib/javacc-6.1.2.jar:/usr/share/opentsdb/lib/jgrapht-core-0.9.1.jar:/usr/share/opentsdb/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/opentsdb/lib/logback-classic-1.0.13.jar:/usr/share/opentsdb/lib/logback-core-1.0.13.jar:/usr/share/opentsdb/lib/netty-3.9.4.Final.jar:/usr/share/opentsdb/lib/protobuf-java-2.5.0.jar:/usr/share/opentsdb/lib/slf4j-api-1.7.7.jar:/usr/share/opentsdb/lib/tsdb-2.3.0.jar:/usr/share/opentsdb/lib/zookeeper-3.4.6.jar:/etc/opentsdb
21:57:28.835 INFO  [ZooKeeper.logEnv] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
21:57:28.835 INFO  [ZooKeeper.logEnv] - Client environment:java.io.tmpdir=/tmp
21:57:28.836 INFO  [ZooKeeper.logEnv] - Client environment:java.compiler=<NA>
21:57:28.836 INFO  [ZooKeeper.logEnv] - Client environment:os.name=Linux
21:57:28.836 INFO  [ZooKeeper.logEnv] - Client environment:os.arch=amd64
21:57:28.836 INFO  [ZooKeeper.logEnv] - Client environment:os.version=4.4.0-57-generic
21:57:28.836 INFO  [ZooKeeper.logEnv] - Client environment:user.name=opentsdb
21:57:28.836 INFO  [ZooKeeper.logEnv] - Client environment:user.home=/usr/share/opentsdb
21:57:28.836 INFO  [ZooKeeper.logEnv] - Client environment:user.dir=/
21:57:28.838 INFO  [ZooKeeper.<init>] - Initiating client connection, connectString=localhost sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@41dc079d
21:57:28.857 INFO  [HBaseClient.getDeferredRoot] - Need to find the -ROOT- region
21:57:28.859 INFO  [ClientCnxn.logStartConnect] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
21:57:28.867 INFO  [ClientCnxn.primeConnection] - Socket connection established to localhost/127.0.0.1:2181, initiating session
21:57:28.884 INFO  [ClientCnxn.onConnected] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15c7ea140b2000b, negotiated timeout = 5000
21:57:28.924 INFO  [HBaseClient.handleMetaZnode] - Connecting to .META. region @ 100.122.3.120:34951
21:57:28.992 INFO  [ZooKeeper.close] - Session: 0x15c7ea140b2000b closed
21:57:28.993 INFO  [ClientCnxn.run] - EventThread shut down
21:57:29.051 INFO  [HBaseClient.discoverRegion] - Added client for region RegionInfo(table="tsdb", region_name="tsdb,,1496774799494.c9b87ef95478623db1c050a4330302a1.", stop_key=""), which was added to the regions cache.  Now we know that RegionClient@536150103(chan=[id: 0x5f625ace, /100.122.3.120:37652 => /100.122.3.120:34951], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) is hosting 1 region.
21:57:29.052 INFO  [HBaseClient.discoverRegion] - Added client for region RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1496774798179.343e3100ce362036faa8f5df3bd6a24f.", stop_key=""), which was added to the regions cache.  Now we know that RegionClient@536150103(chan=[id: 0x5f625ace, /100.122.3.120:37652 => /100.122.3.120:34951], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) is hosting 2 regions.
21:57:29.102 INFO  [RpcManager.initializeBuiltinRpcs] - Mode: rw, HTTP UI Enabled: true, HTTP API Enabled: true
21:57:29.149 INFO  [RpcHandler.<init>] - TSD is in rw mode
21:57:29.149 INFO  [RpcHandler.<init>] - CORS domain list was empty, CORS will not be enabled
21:57:29.150 INFO  [RpcHandler.<init>] - Loaded CORS headers (Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since)
21:57:29.155 INFO  [ConnectionManager.<init>] - TSD concurrent connection limit set to: 0
21:57:29.159 WARN  [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.tsd.HttpSerializer
21:57:29.192 INFO  [TSDB.shutdown] - Flushing compaction queue
21:57:29.195 INFO  [HBaseClient.handleDisconnect] - Channel [id: 0x5f625ace, /100.122.3.120:37652 => /100.122.3.120:34951] is disconnecting: [id: 0x5f625ace, /100.122.3.120:37652 => /100.122.3.120:34951] DISCONNECT
21:57:29.196 INFO  [HBaseClient.removeClientFromCache] - Lost connection with the .META. region
21:57:29.203 INFO  [TSDB.call] - Completed shutting down the TSDB
21:57:29.240 INFO  [TSDB.shutdown] - Flushing compaction queue
21:57:29.241 INFO  [TSDB.call] - Completed shutting down the TSDB

[ManOLamancha]

The only thing that looks suspicious to me is that immediate compaction queue flush and the following disconnect. A couple of things to try:

Catch stderr and stdout from the TSD to see if anything is being thrown instead of logged. 

Try disabling TSD compactions, see if it loads then.

Enable debugging to see if anything else is happening behind the scenes.

[Travis Keep]

I tried catching stderr and stdout -- nothing.
I tried disabling TSD compactions -- server still quits

How do I enable debugging? I don't see anything about log levels / debugging in the open tsdb configuration file at /etc/opentsdb.conf.

[Jonathan Creasy]

Log levers are adjusted in the src/logback.xml file. They can also be adjusted via API after the service is running, which of course, would not work here. 

  <!-- Per class logger levels -->

  <logger name="QueryLog" level="OFF" additivity="false">

    <appender-ref ref="QUERY_LOG"/>

  </logger>

  <logger name="org.apache.zookeeper" level="INFO"/>

  <logger name="org.hbase.async" level="INFO"/>

  <logger name="com.stumbleupon.async" level="INFO"/>

  <!-- Fallthrough root logger and router -->

  <root level="INFO">

    <appender-ref ref="STDOUT"/>

    <appender-ref ref="CYCLIC"/>

    <!-- Uncomment to log to file -->

    <!--<appender-ref ref="FILE"/>-->

  </root>