TSDB and Hbase region splits

[V Kurien]

Did quite a lot of googling on this, but does anyone know what the problem here may be?

1.  HBaseClient: Invalidated cache for RegionInfo(table="tsdb", region_name="tsdb,\x00\x01PU\xB5\xD6\xF0\x00\x00\x01\x00\x00\x01\x00\x00\x0D\x00\x00g,1443187768636.be5dc807a6f4f2159c6e3a576a401c2c.", stop_key="") as RegionClient@1219100908(chan=[id: 0x3c33c694, /10.0.0.132:53931 => /10.0.0.132:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2. TSD is basically down while these are going on, presumably because RPCs are being dropped

What do I do to fix it? I've shutdown tsd and done a major compaction on the tsdb table by hand already.

[ManOLamancha]

On Thursday, October 1, 2015 at 1:49:13 PM UTC-7, V Kurien wrote:

Did quite a lot of googling on this, but does anyone know what the problem here may be?

1.  HBaseClient: Invalidated cache for RegionInfo(table="tsdb", region_name="tsdb,\x00\x01PU\xB5\xD6\xF0\x00\x00\x01\x00\x00\x01\x00\x00\x0D\x00\x00g,1443187768636.be5dc807a6f4f2159c6e3a576a401c2c.", stop_key="") as RegionClient@1219100908(chan=[id: 0x3c33c694, /10.0.0.132:53931 => /10.0.0.132:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2. TSD is basically down while these are going on, presumably because RPCs are being dropped

The TSD shouldn't completely stop while a region splits and RPCs won't necessarily be dropped while the region is splitting. With that exception you should also see a bunch of NSRE logs. When the region is offline or moving, the data points will queue up in memory (and could potentially OOM the JVM) until the region is back online. With 2.2 there's a plugin interface that will allow reducing that NSRE queue size and spool the values to disk (or in an external queue) for retry later. If your TSD is going down, check the heap and GC.