hbase-indexer auto exit when data load frequently

[Jianxin Chen]

Recently my team is using hbase-indexer on CDH for indexing hbase table column to solr . When we deploy hbase-indexer server (which is called Key-Value Store Indexer) and begin testing. We found a situation that when we put data to hbase (We are using apache phoenix, a SQL layer above hbase) frequently, the hbase-indexer process will auto exit.We have checked the log and found ERROR log of Zookeeper Session Expired like this :

ERROR com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper: ZooKeeper session expired, shutting down.

....

The software env is :

• CDH5.4
  
• HBase1.0
  
• Phoenix4.6
  
• Hbase-Indexer (hbase-solr-1.5-cdh5.4.2)
  

The java heapsize of hbase-indexer is configured to 1GB .

Is there anyone who meet this situation ?

[Murtaza Kanchwala]

I don't think that it even works, Look at this log snippet everything is well connected.

Java heap : given to the service is 2gb in the Cloudera configuration for the KV-Indexer

OOM Killer : Also disabled "Kill if Out of Memory" in Cloudera console - Don't know for KB of writes how it will go out of memory first of all?

Also running the service with root:root instead of hbase:hbase in the cloudera console.

RAM : Above 16 gb and only HBase, Solr and KV-Indexer is working with an external zookeeper. Nobody is working on its own embedded one to be clear.

Environment : Same as yours CDH 5.4.10

Service output : hbase-indexer list-indexers

DIndexer

  + Lifecycle state: ACTIVE

  + Incremental indexing state: SUBSCRIBE_AND_CONSUME

  + Batch indexing state: INACTIVE

  + SEP subscription ID: Indexer_DIndexer

  + SEP subscription timestamp: 2016-05-23T11:55:42.854+05:30

  + Connection type: solr

  + Connection params:

    + solr.collection = mySolrCore

    + solr.zk = zookeeper-instance-n001/solr

  + Indexer config:

      277 bytes, use -dump to see content

  + Indexer component factory: com.ngdata.hbaseindexer.conf.DefaultIndexerComponentFactory

  + Additional batch index CLI arguments:

      (none)

  + Default additional batch index CLI arguments:

      (none)

  + Processes

    + 1 running processes

    + 0 failed processes

2016-05-23 11:55:44,042 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: zookeeper-instance-n001,48937,1463984744010-0x154cddaff521f8b connected

2016-05-23 11:55:54,016 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:56:04,016 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:56:04,026 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:56:14,037 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:56:24,027 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:56:24,048 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:56:34,049 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:56:44,044 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:56:44,051 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:56:54,061 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:57:04,052 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:57:04,065 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:57:14,074 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:57:24,072 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:57:24,085 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:57:34,094 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 1ms

2016-05-23 11:57:44,080 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:57:44,101 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:57:54,107 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:58:04,101 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:58:04,107 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:58:14,113 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:58:24,122 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:58:24,122 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 1ms

2016-05-23 11:58:34,126 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:58:44,132 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:58:44,142 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:58:54,142 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:59:04,148 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:59:04,157 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:59:14,157 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:59:24,160 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:59:24,169 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 11:59:34,171 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:59:44,182 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 11:59:44,189 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 1ms

2016-05-23 11:59:54,184 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 12:00:04,187 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 12:00:04,197 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 0ms

2016-05-23 12:00:14,192 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 12:00:24,203 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f7a after 0ms

2016-05-23 12:00:24,204 DEBUG org.apache.zookeeper.ClientCnxn: Got ping response for sessionid: 0x154cddaff521f8b after 1ms

I've even written 10 records in the Table and should be synced up with Solr, After so many restarts, Cluster restart, Machine Restart, Even everything restart. Couldn't worked either. I doubt if it is a good option for production?

I am stucked with this for Nearly a week. Really frustated

[Jianxin Chen]

If you install solr by cdh , You can check out the solr java heapsize to make sure solr work well.

在 2016年5月23日星期一 UTC+8下午2:39:19，Murtaza Kanchwala写道：