OutOfMemory

[Ahmet Uyar]

Hi Supun,

When I run twister2 with the local disk, it works with 10M tweetID-date pairs. It completed the membership finding job successfully with 240 workers. However, when I try to process 50M tweetID-date pairs with 240 workers. It gives an out of memory error. 

I copied the java heap dump files of some workers to the following directory at the login node of victor:

/scratch_hdd/auyar/heap-dump/

$ ls
java_pid102168.hprof  

java_pid102185.hprof  

java_pid102208.hprof  

java_pid102333.hprof  

java_pid102444.hprof 

logs
java_pid102172.hprof  

java_pid102196.hprof 

java_pid102272.hprof  

java_pid102349.hprof  

java_pid102475.hprof
java_pid102176.hprof  

java_pid102200.hprof  

java_pid102305.hprof  

java_pid102353.hprof  

java_pid102549.hprof

thanks,

Ahmet