Increase Yarn Memory for Apache Apex Application using DataTorrent RTS docker

[matthia...@gmail.com]

I would like to deploy a Apache Apex application to the DataTorrent RTS sandbox running in docker.

I am running the latest official docker image as described here: https://hub.docker.com/r/datatorrent/rts/

My application consists of 5 operators. 2 of these operators shall run on 3 containers in parallel using Partitioning.

When trying to launch my application using these specifications, all my containers are stuck in Status: PENDING_DEPLOY.

If I remove partitioning, my app and all its containers launch and run without problems. (the memory allocation in this case is 6GB).

I realized, that my containers always go to the state PENDING_DEPLOY if my total memory allocation would go above 8GB.

While trying to find a solution for this problem, I found that there exists a YARN property called: yarn.nodemanager.resource.memory-mb
According to this link this value defaults to 8192MB. 

Inside of my container there exists a file: /etc/hadoop/conf/yarn-site.xml
I tried increasing the memory to ~12GB (my machine has around 15GB RAM) by adding the following property to the file:

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>12000</value>
</property>

I also tried adding this property as proposed in the docs:

<property>
  <name>yarn.nodemanager.vmem-pmem-ratio</name>
  <value>10</value>
</property>

After changing the values, I launched my application again, but again without success: PENDING_DEPLOY.
In dtManage it again seems to stuck at 8GB of memory allocation.

I tried this by restarting the docker container, immediately changing the values as described and only afterwards did followed the Installation Wizard steps in dtManage. Same output.

Does someone know how I can use the docker image with more memory than 8GB?

[matthia...@gmail.com]

Finally found the solution myself.

To increase the memory, as described above, the /etc/hadoop/conf/yarn-site.xml needs to be customized.
Since this file needs to be available when the docker container starts, using a bind-mount is an option to add custom files to the container.
Using bind-mount, a directory on the host can be shared with the container.

To get this working, following steps are necessary:

• copy all files inside the container that are in this directory: /etc/hadoop/conf/
  

docker cp <CONTAINER-ID> /etc/hadoop/conf/ /some/dir/at/host

• on the host machine, change the yarn-site.xml file accordingly
• create a bind-mount and run the container (code below shows how to do this using compose)

services:

apex:
image: datatorrent/rts:latest
ports:
  - "9090:9090"
  - "50070:50070"
  - "8088:8088"
stdin_open: true # Must be added, otherwise container shuts down
tty: true # Must be added, otherwise container shuts down
volumes:
  - "/some/dir/at/host:/etc/hadoop/conf/"

 This way, I was able to start my application with many partitions.

[Sanjay Pujare]

Good to know, thanks for posting the solution.

BTW, in your original post I assumed you restarted the resource-manager and node-managers after modifying yarn-site.xml but looks like you didn't. Do you think that also could have worked?

Sanjay

 

--
You received this message because you are subscribed to the Google Groups "DataTorrent Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+unsubscribe@googlegroups.com.
To post to this group, send email to dt-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dt-users/a4e6112d-6838-466f-988c-bf5107c6b4b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[matthia...@gmail.com]

I assumed you restarted the resource-manager and node-managers after modifying yarn-site.xml but looks like you didn't. Do you think that also could have worked?

I intended to do a restart, but unfortunately I couldn't find any information how to this inside the docker container?
As described here i was trying to find any scripts inside the container that are named something like:

start-yarn.sh
stop-yarn.sh

I couldn't find any though. Do you perhaps know how i can restart hadoop/yarn inside the DataTorrent RTS sandbox docker container?
If that works, it would make it a lot easier to change settings.

To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+u...@googlegroups.com.

[Ankit Singh Rathore]

Hi,

You can restart yarn with below command:

/etc/init.d/hadoop-yarn-resourcemanager restart

You will have to use sudo with dtadmin user. I hope this helps.

Thanks,

Ankit.

[matthia...@gmail.com]

Thanks, this worked and is a much simpler solution to the one I posted before.
Only make sure to also restart the dtGateway.