Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
at com.google.protobuf.ByteString.copyFrom(ByteString.java:204)
at org.hbase.async.RegionInfo.toProtobuf(RegionInfo.java:92)
at org.hbase.async.GetRequest.serialize(GetRequest.java:409)
at org.hbase.async.RegionClient.encode(RegionClient.java:1146)
at org.hbase.async.RegionClient.sendRpc(RegionClient.java:894)
at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1698)
at org.hbase.async.HBaseClient.get(HBaseClient.java:995)
at net.opentsdb.core.TSDB.get(TSDB.java:1114)
at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:185)
at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116)
at net.opentsdb.core.TSDB.shutdown(TSDB.java:814)
at net.opentsdb.tools.TextImporter.main(TextImporter.java:88)
I have 8 region servers serving 256 regions for this TSD setup.
Can anyone provide me some insights on how to debug this failure?
Thanks,
David
$ java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
uintx AdaptivePermSizeWeight = 20 {product}
intx CompilerThreadStackSize = 0 {pd product}
uintx ErgoHeapSizeLimit = 0 {product}
uintx HeapSizePerGCThread = 87241520 {product}
uintx InitialHeapSize := 245433792 {product}
uintx LargePageHeapSizeThreshold = 134217728 {product}
uintx MaxHeapSize := 3927965696 {product}
uintx MaxPermSize = 85983232 {pd product}
uintx PermSize = 21757952 {pd product}
intx ThreadStackSize = 1024 {pd product}
intx VMThreadStackSize = 1024 {pd product}
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
I was trying to import 64 .gz files into my freshly configured OpenTSDB using the batch importer. The import process when fairly quickly at the start of the process. But overtime, the process slowed down and eventually bailed out when the following exception was thrown:
I have 8 region servers serving 256 regions for this TSD setup.
Can anyone provide me some insights on how to debug this failure?