java.lang.OutOfMemoryError

[Ahmet Uyar]

Hi Chathura, Supun and all,

Another thing that I noticed with these tests is that when I used hdfs for saving intermediate data, twsiter2 goes out of memory easily. 

I cancelled persisting delete tweetIDs and cached them instead. They are small, 1K per worker.

I reduced the number of tweetID-date pairs per worker to 10K.

I used only 4 workers.  

But I got  java.lang.OutOfMemoryError from all workers. 

I attached the logs and put the java heap dumps at the login node of victor: 

/scratch_hdd/auyar/heap-dump/java_pid141124.hprof

/scratch_hdd/auyar/heap-dump/java_pid193304.hprof

/scratch_hdd/auyar/heap-dump/java_pid194512.hprof

/scratch_hdd/auyar/heap-dump/java_pid195638.hprof

Can this be related to the other problem? 

Ahmet

[Ahmet Uyar]

Hi guys,

This problem is not there anymore after the fix. 

Ahmet

[Supun Kamburugamuve]

Nice. Did it run to the end without issues?

Best,

Supun..

--
You received this message because you are subscribed to the Google Groups "Twister2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to twister2+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/twister2/CAPBRfYeJaT187o29bo7r2MrqoMUkrw0O4bEAN10vzUYzQ8q80w%40mail.gmail.com.

--

Supun Kamburugamuve, PhD

Digital Science Center, Indiana University

Member, Apache Software Foundation; http://www.apache.org
E-mail: supun@apache.org;  Mobile: +1 812 219 2563

[Ahmet Uyar]

yes. it is completing successfully. I am testing with larger data. I am also testing with hdfs 2.10. I will let you know. 

Ahmet

[Ahmet Uyar]

It ran successfully with 240 workers each processing 10M tweetID-date pairs. 

It was running on hdfs 2.10. It also works fine with hdfs 3.2.

Ahmet

 

[Chathura Widanage]

Super!

To view this discussion on the web visit https://groups.google.com/d/msgid/twister2/CAPBRfYebqN6CqsfL2Q7qQ_Use90zV72WxxVPhNL78ODq0Y9mbw%40mail.gmail.com.