java.lang.OutOfMemoryError encountered during batch import of gzip files

[David Yu]

I was trying to import 64 .gz files into my freshly configured OpenTSDB using the batch importer. The import process when fairly quickly at the start of the process. But overtime, the process slowed down and eventually bailed out when the following exception was thrown:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

 

    at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)

    at com.google.protobuf.ByteString.copyFrom(ByteString.java:204)

    at org.hbase.async.RegionInfo.toProtobuf(RegionInfo.java:92)

    at org.hbase.async.GetRequest.serialize(GetRequest.java:409)

    at org.hbase.async.RegionClient.encode(RegionClient.java:1146)

    at org.hbase.async.RegionClient.sendRpc(RegionClient.java:894)

    at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1698)

    at org.hbase.async.HBaseClient.get(HBaseClient.java:995)

    at net.opentsdb.core.TSDB.get(TSDB.java:1114)

    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:185)

    at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116)

    at net.opentsdb.core.TSDB.shutdown(TSDB.java:814)

at net.opentsdb.tools.TextImporter.main(TextImporter.java:88)

I have 8 region servers serving 256 regions for this TSD setup. 

Can anyone provide me some insights on how to debug this failure?

Thanks,

David

[David Yu]

FYI, here is the heap allocation of my JVM, if that's helpful:

$ java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'

    uintx AdaptivePermSizeWeight                    = 20              {product}          

     intx CompilerThreadStackSize                   = 0               {pd product}        

    uintx ErgoHeapSizeLimit                         = 0               {product}          

    uintx HeapSizePerGCThread                       = 87241520        {product}          

    uintx InitialHeapSize                          := 245433792       {product}          

    uintx LargePageHeapSizeThreshold                = 134217728       {product}          

    uintx MaxHeapSize                              := 3927965696      {product}          

    uintx MaxPermSize                               = 85983232        {pd product}        

    uintx PermSize                                  = 21757952        {pd product}        

     intx ThreadStackSize                           = 1024            {pd product}        

     intx VMThreadStackSize                         = 1024            {pd product}        

java version "1.7.0_80"

Java(TM) SE Runtime Environment (build 1.7.0_80-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

[ManOLamancha]

On Friday, November 27, 2015 at 10:57:37 AM UTC-8, David Yu wrote:

I was trying to import 64 .gz files into my freshly configured OpenTSDB using the batch importer. The import process when fairly quickly at the start of the process. But overtime, the process slowed down and eventually bailed out when the following exception was thrown:

I have 8 region servers serving 256 regions for this TSD setup. 

Can anyone provide me some insights on how to debug this failure?

The first place to look, in this case, are the region servers to see if any of them fell over during the import. Also check their logs to see how many regions split during the run. Whenever the TSD OOMs it's usually because HBase started splitting or moving regions so that the TSD had to backup some stuff in memory. We need better back pressure there around the imports. In the mean time you can tune some AsyncHBase params (with version 2.2.0RC3), particularly the high RPC watermark: http://opentsdb.github.io/asynchbase/docs/build/html/configuration.html