Memory Setup

[Chris]

Hi all,

I'm using Hadoop 2.3.0 and MR2/Yarn on a tiny 3-node test vm cluster on my laptop.

yarn.nodemanager.resource.memory-mb is set to 1024 in yarn-site.xml.

The streaming job requests 2048MB which results in the following

---

2014-06-24 22:29:59,551 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Node : hadoop3.local:60954 does not have sufficient resource for request : {Priority: 0, Capability: <memory:2048, vCores:1>, # Containers: 1, Location: *, Relax Locality: true} node total capability : <memory:1024, vCores:8>

---

in the yarn-resourcemanager-master.log.

Where can I change/reduce the respective container memory setting for mrjob / hadoop streaming?

Thanks.

Chris

[viskumar]

Hi,

You can set below property  to set container min and max allocation in yarn-site.xml

<property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>2048</value>
 </property>

<property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>8192</value>
    </property>

Regards,
Vishnu Kumar

[Chris]

Thanks Vishnu!

Not the clue though ...

My yarn container and scheduler seemed alright.

--- snip --- yarn-site.xml ---

  <property>

      <name>yarn.nodemanager.resource.memory-mb</name>

      <value>1024</value>

  </property>

  <property>

      <name>yarn.scheduler.minimum-allocation-mb</name>

      <value>512</value>

  </property>

  <property>

      <name>yarn.scheduler.maximum-allocation-mb</name>

      <value>1024</value>

  </property>

--- snap ---

--- snip --- mapred-site.xml ---

 <property>

    <name>mapreduce.map.memory.mb</name>

  <value>1024</value>

 </property>

 <property>

    <name>mapreduce.reduce.memory.mb</name>

  <value>1024</value>

 </property>

 <property>

    <name>mapreduce.map.java.opts</name>

  <value>-Xmx512m</value>

 </property>

 <property>

    <name>mapreduce.reduce.java.opts</name>

  <value>-Xmx512m</value>

 </property>

--- snap ---

What was missing is the App Master Resource Limitation in yarn-site.xml

  <property>

      <name>yarn.app.mapreduce.am.resource.mb</name>

      <value>1024</value>

  </property>

Its default is 1536. Without setting appropriately (within yarn.nodemanager.resource.memory-mb, not sure whether also within yarn.scheduler.maximum-allocation-mb) no job could run at all.

Btw: Good Yarn Resource Mngt explanation:

http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

Cheers.

Chris