Tune the EMR JVM size too solve java.lang.OutOfMemoryError: Java heap space

[Eric Pantera]

Hello,

I would like to share with you a quick solution that we had implemented here to solve an OutOfMemoryError: Java heap space error during the EMR process.

1) Create a custom bootstrap script file hosted in S3

eg: s3://viadeo-snowplow-processing/bootstrap-actions/snowplow-emr-customize-heap-size.sh with content :

#!/bin/sh
/usr/share/aws/emr/scripts/configure-hadoop -m mapred.child.java.opts=-Xmx3G

2) Declare the custom bootstrap action script in the snowplow config file

:emr:
  ...
  :bootstrap:
    - s3://viadeo-snowplow-processing/bootstrap-actions/snowplow-emr-customize-heap-size.sh

Simple and non intrusive - works well for us.

Hope it helps.

[shilpi singh]

Hi,

I am trying to increase the Java Heap space using the method mentioned in the post  but it did not work. There was no errors in the emr cluster bootstrap steps but I could not find  my custom bootstrap action over there. Also I logged in to the Mater node of the cluster and there was no such path:

/usr/share/aws/emr/scripts/

Please let me know If I am missing some steps .

[Alex Dean]

What version of Snowplow are you running, with what EMR AMI version?

A

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Co-founder
Snowplow Analytics
The Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom
+44 (0)203 589 6116
+44 7881 622 925
@alexcrdean

[shilpi singh]

Hi Alex,

ami version: 2.4.2

 :hadoop_enrich: 0.14.1 

    :hadoop_shred: 0.4.0 

[Alex Dean]

Can you share how you are adding your bootstrap action in your config.yml?

A

--

You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[shilpi singh]

My config.yml has:

:emr:

  :ami_version: 2.4.2      # Choose as per http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-ami.html

  :region: eu-west-1

  :bootstrap:

    - s3://udmd-p-emr-heapsize/customize/emr-customize-bootsize.sh    #s3 buckets that contains customized script

Contents of emr-customize-bootsize.sh :

#!/bin/sh

/usr/share/aws/emr/scripts/configure-hadoop -m mapred.child.java.opts=-Xmx3G

 

[Alex Dean]

• Is the executable flag set on your bash script
• Is your S3 bucket in eu-west-1
• Is your bash script readable by your AWS user
  

--

You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[shilpi singh]

Thanks a lot Alex. Got the issue- script was not accessible by the AWS user. Its working fine now.