TSDB gives OutOfMemoryError

[Sumanjeet Bhatti]

Hi team,

I am running OpenTSDB v2.0 on Single node HBase over file system. I got the OutOfMemoryError with stack trace given below while inserting Data Points into it:

 java.lang.OutOfMemoryError: Java heap space

        at java.lang.Throwable.getOurStackTrace(Throwable.java:825)

        at java.lang.Throwable.getStackTrace(Throwable.java:816)

        at org.jboss.netty.util.internal.StackTraceSimplifier.simplify(StackTraceSimplifier.java:56)

        at org.jboss.netty.logging.InternalLoggerFactory$1.warn(InternalLoggerFactory.java:127)

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:321)

        at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

java.lang.OutOfMemoryError: Java heap space

I have 4GB memory configured for TSDB. I looked into the GC logs. It is showing the call for FullGC, but not able to release the memory effectively.

The data I am adding into TSDB has 3 tag key-values along with timestamp and metric value. 

Can anybody give the direction to find the rootcause of this issue.

[ManOLamancha]

Check further up the logs for NSRE errors. These indicate your regions are moving or splitting so you may need to tune HBase a bit. Also, it could be that you're simply writing too much data to a TSD so it could be time to scale out by adding more TSDs behind a load balancer.