Hello dears, I faced a problem named UnknownScannerException while doing fsck, someone now how to deal with it? thx !

[Lynch Lee]

2015-12-08 11:27:53,720 WARN  [New I/O worker #4] Scanner: RegionInfo(table="usergrid_metrics", region_name="usergrid_metrics,\x00\x00\x02V7P@\x00\x00\x01\x00\x04D\x00\x00\x02\x00\x05\x10\x00\x00\x06\x00\x005\x00\x00\x07\x00\x004,1448975244412.f8bff4758facb0761139aafdbc5f8088.", stop_key=[0, 0, 10, 85, 93, 116, 96, 0, 0, 8, 0, 4, 55, 0, 0, 9, 0, 4, 58, 0, 0, 10, 0, 4, 62, 0, 0, 11, 0, 4, 60, 0, 0, 12, 0, 4, 58]) pretends to not know Scanner(table="usergrid_metrics", start_key=[0, 0, 2, 86, 101, -126, -48, 0, 0, 1, 0, 4, 68, 0, 0, 2, 0, 4, -2, 0, 0, 6, 0, 0, 53, 0, 0, 7, 0, 0, 52], stop_key=[0, 0, 2, 86, 101, -69, 17], columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=null, scanner_id=0x00000000000003DE).  I will retry to open a scanner but this is typically because you've been holding the scanner open and idle for too long (possibly due to a long GC pause on your side or in the RegionServer)

org.hbase.async.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 25566, already closed?

at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3132)

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29925)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)

at java.lang.Thread.run(Thread.java:744)

Caused by RPC: GetNextRowsRequest(scanner_id=0x00000000000003DE, max_num_rows=128, region=null, attempt=0)

at org.hbase.async.UnknownScannerException.make(UnknownScannerException.java:60) ~[asynchbase-1.6.0.jar:na]

at org.hbase.async.UnknownScannerException.make(UnknownScannerException.java:32) ~[asynchbase-1.6.0.jar:na]

at org.hbase.async.RegionClient.makeException(RegionClient.java:1448) [asynchbase-1.6.0.jar:na]

at org.hbase.async.RegionClient.decodeException(RegionClient.java:1468) [asynchbase-1.6.0.jar:na]

at org.hbase.async.RegionClient.decode(RegionClient.java:1299) [asynchbase-1.6.0.jar:na]

at org.hbase.async.RegionClient.decode(RegionClient.java:89) [asynchbase-1.6.0.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]

at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1082) [asynchbase-1.6.0.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]

at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2677) [asynchbase-1.6.0.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_40]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]

194832 KVs (in 194822 rows) analyzed in 114140ms (~1706 KV/s)

Found 2389 errors.

2015-12-08 11:27:56,009 INFO  [main] TSDB: Flushing compaction queue

2015-12-08 11:27:56,123 INFO  [main] HBaseClient: Lost connection with the .META. region

[ManOLamancha]

On Monday, December 7, 2015 at 7:33:19 PM UTC-8, Lynch Lee wrote:

2015-12-08 11:27:53,720 WARN  [New I/O worker #4] Scanner: RegionInfo(table="usergrid_metrics", region_name="usergrid_metrics,\x00\x00\x02V7P@\x00\x00\x01\x00\x04D\x00\x00\x02\x00\x05\x10\x00\x00\x06\x00\x005\x00\x00\x07\x00\x004,1448975244412.f8bff4758facb0761139aafdbc5f8088.", stop_key=[0, 0, 10, 85, 93, 116, 96, 0, 0, 8, 0, 4, 55, 0, 0, 9, 0, 4, 58, 0, 0, 10, 0, 4, 62, 0, 0, 11, 0, 4, 60, 0, 0, 12, 0, 4, 58]) pretends to not know Scanner(table="usergrid_metrics", start_key=[0, 0, 2, 86, 101, -126, -48, 0, 0, 1, 0, 4, 68, 0, 0, 2, 0, 4, -2, 0, 0, 6, 0, 0, 53, 0, 0, 7, 0, 0, 52], stop_key=[0, 0, 2, 86, 101, -69, 17], columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=null, scanner_id=0x00000000000003DE).  I will retry to open a scanner but this is typically because you've been holding the scanner open and idle for too long (possibly due to a long GC pause on your side or in the RegionServer)

org.hbase.async.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 25566, already closed?

This will happen if the TSD is too slow due to GC or resource contention and closes the scanner. Try making sure only the TSD is running on the machine/VM where you're running FSCK and also enable/check the GC logs to see what's up.