shortcircuit.ShortCircuitCache: ShortCircuitCache - InvalidToken exception

[GK]

Hi ,

I am using OpenTSDB kafka plugin to load data in OpenTSDB which is using Hbase storage,.

Sometimes, we get the following error in the  'hbase-hbase-regionserver***.log' logs and we see some time series data points are missing in OpenTSDB.

Any idea - Thanks in advance.

-Gk

2018-09-23 09:39:17,927 INFO  [B.fifo.QRpcServer.handler=9,queue=0,port=16020] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x62337adb): could not load 1073752123_BP-696291452-10.1.140.157-1536254818478 due to InvalidToken exception.

--

at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:662)

at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:898)

at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:955)

at java.io.DataInputStream.read(DataInputStream.java:149)

at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:679)

at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1412)

at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1625)

at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)

at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)

at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)

at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:636)

at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:586)

at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:260)

at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)

at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)

at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:217)

at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2122)

at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2112)

at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5569)

at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2627)

at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2613)

at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2595)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2282)

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

at java.lang.Thread.run(Thread.java:745)