org.hbase.async.RemoteException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit

114 views
Skip to first unread message

lovelesh chawla

unread,
Jan 27, 2017, 1:55:13 PM1/27/17
to OpenTSDB
I am importing a TSV file through CLI import command. The file has around 17 million records out of which only 20000 are imported and i get a following error after that. It looks like the issue is that openTSDB is trying to write data points to Hbase faster than it can handle. Is there a workaround for this issue ?

2017-01-27 13:35:10,584 ERROR [AsyncHBase I/O Worker #2] TextImporter: Exception caught while processing file /home/s1378/augNew.gz
org.hbase.async.RemoteException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=tsdb,,1485464358141.1a983e99db7355d026d9bdd220e60cce., server=lnx21120.csxt.csx.com,16020,1485537738135, memstoreSize=536944496, blockingMemStoreSize=536870912
        at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:3751)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2925)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2876)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)



ManOLamancha

unread,
Feb 11, 2017, 7:07:04 PM2/11/17
to OpenTSDB
On Friday, January 27, 2017 at 10:55:13 AM UTC-8, lovelesh chawla wrote:
I am importing a TSV file through CLI import command. The file has around 17 million records out of which only 20000 are imported and i get a following error after that. It looks like the issue is that openTSDB is trying to write data points to Hbase faster than it can handle. Is there a workaround for this issue ?

2017-01-27 13:35:10,584 ERROR [AsyncHBase I/O Worker #2] TextImporter: Exception caught while processing file /home/s1378/augNew.gz
org.hbase.async.RemoteException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=tsdb,,1485464358141.1a983e99db7355d026d9bdd220e60cce., server=lnx21120.csxt.csx.com,16020,1485537738135, memstoreSize=536944496, blockingMemStoreSize=53687091

We don't have a great way to limit this yet though if you or anyone wants to take a stab, it shouldn't be too difficult to throw a Guava rate limiter in the write path. We just haven't gotten around to it. Otherwise try adding salting if you're starting with a new table as that will help to spread the load a little. And you may be able to tun the flush limit and mem store sizes in the region server though I'm not sure what values would be appropriate. We have this old issue to track it: https://github.com/OpenTSDB/opentsdb/issues/173
Reply all
Reply to author
Forward
0 new messages