How to set YARN specific option (yarn.nodemanager.vmem-check-enabled) in EMR

[regata]

Hey guys,

I'm seeing the following problem when trying to run a job on EMR using mrjob: 

Container [pid=11274,containerID=container_1420305915489_0001_01_000003] is running beyond virtual memory limits. Current usage: 364.7 MB of 1 GB physical memory used; 38.2 GB of 5 GB virtual memory used. Killing container.

AMI version: 3.3.1

Hadoop distribution: Amazon 2.4.0

mrjob: 0.4.2

After googling it was suggested to adjust 2 configuration options:

mapreduce.map.memory.mb: 1024
yarn.nodemanager.vmem-check-enabled: false

I tried using mrjob.conf yaml file:

  emr:
    jobconf:
      yarn.nodemanager.vmem-check-enabled: false
      mapreduce.map.memory.mb: 1024

but only mapreduce.map.memory.mb is being updated.

How do I set yarn.nodemanager.vmem-check-enabled option?

Thanks!

[regata]

figured it out

  emr:
    bootstrap_actions:
     - >
      s3://elasticmapreduce/bootstrap-actions/configure-hadoop
      -y yarn.nodemanager.vmem-check-enabled=false
      -y yarn.nodemanager.vmem-pmem-ratio=50
      -y yarn.nodemanager.resource.memory-mb=12288
      -y yarn.scheduler.minimum-allocation-mb=3057
      -y yarn.app.mapreduce.am.resource.mb=6114