OutOfMemory in tasks

[ankur jain]

Hi Guys,

I am seeing this error in my task logs:

==========

Exception in thread "plumber_merge_0" java.lang.OutOfMemoryError: Java heap space

at java.util.Arrays.copyOf(Arrays.java:3181)

at java.util.ArrayList.grow(ArrayList.java:265)

at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:239)

at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:231)

at java.util.ArrayList.add(ArrayList.java:462)

at io.druid.segment.CompressedVSizeIndexedSupplier.fromIterable(CompressedVSizeIndexedSupplier.java:116)

at io.druid.segment.IndexIO$DefaultIndexIOHandler.convertV8toV9(IndexIO.java:677)

at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:898)

at io.druid.segment.IndexMerger.merge(IndexMerger.java:437)

at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:241)

at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:213)

at io.druid.segment.realtime.plumber.RealtimePlumber$2.doRun(RealtimePlumber.java:415)

at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

==============

I have attached the complete task logs.  

One thing to note is not all tasks fail for this datasource. Some are successful but most of them fail.

I have other datasources also on this cluster and all of them work without problems.

I have played around the maxRowsInMemory and reduced it to 25k, but still no success. However, I feel this is happening during the segment packaging stage so maxRowsInMemory should not have any effect.

Any help is appreciated.

Thanks

[Gian Merlino]

Hey Ankur,

Counter-intuitively, you may want to increase maxRowsInMemory. An OutOfMemoryError during the segment packaging / publishing stage can happen if there are too many spills created in the initial ingestion phase, which can happen if maxRowsInMemory is too low. Another thing you could do is get Druid to make segments that are somewhat smaller. You'd do this by increasing taskCount (w/ kafka indexing) or task.partitions (w/ tranquility).

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/a1fb5d9c-6036-4f88-9734-8cb1d8a1d208%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ankur jain]

Hey Gian,

Thanks for the reply.  Really appreciate it.

I tried multiple things but none of them seems to work.  Here are the things I tried:

1) Set the max rows in memory to really high to 200k  and then to 500k but still results in OOM. I set the number of partitions to 25.

I see 'Terminating due to java.lang.OutOfMemoryError: Java heap space'  as the last line of the log. I do not see any other error in log. The log is attached.

2) I moved the maxRowsInMemory to opposite side of the spectrum to 25k. Still the same problem.

The stream produces about 100m records per hour and has a size of ~15-20g.  I thought 25 partitions should be enough.

I also checked the task directory and each split is about 120mb and there is a total of 7 splits.  This is when maxrows in memory is set to 500k.

Thanks

Ankur

[Jihoon Son]

Hi Ankur,

what was the jvm max memory for peon?

Also, maxRowsInMemory looks too low. I think you can increase it to 1m or so.

Jihoon

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/f0ed5f2c-e610-4bf9-b0bf-a3abbfa4b984%40googlegroups.com.

[ankur jain]

Hi Jihoon,

It's 3gb.

Here is the peon jvm command -        java -server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+ExitOnOutOfMemoryError

Thanks

Ankur

[Jihoon Son]

Thanks Ankur,

Per your description, each segment can be around 1GB at the peak. If this is the physical size on disk, it can be bigger in memory. If this the case, 3GB looks too low to me.

Can you try 4 - 6GB?

Jihoon

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/cfd58a8a-df26-4cbd-8730-ca72752db238%40googlegroups.com.

[ankur jain]

Thanks, Jihoon. I will try it out.

I have other tasks(for different datasource) in the same cluster which have segment size of 1gb and they do work with the same config. 

Thanks