FileNotFoundException: File does not exist: /user/dtadmin/datatorrent/apps/application_1434453368487_0029/events/index.txt

[Aalok Gupta]

Hi Team,

Please pardon our stupid issues, we are at beginner's level.

The issue is after scucessful installation of DT and Hadoop. We wanted to run a demo package.

We imported 'WordCount' package as per the steps mentioned at

https://www.datatorrent.com/docs/guides/GettingStartedGuide.html

We used the config file already provided my-app-conf1.xml.

Clicked on launch and observed the following view DT application overview.PNG

Also attaching the logs.

We are seeking a 'running' state application overview as depicted in user guide.

**Please note that we haven't made any changes in conf, under /opt/datatorrent/current/conf

Thanks

Aalok

[Sandeep Deshmukh]

Hi Aalok,

Can you check value of yarn.nodemanager.resource.memory-mb propery in yarn-site.xml. Yarn may not have resources to allocate for this program. 

Also please check Yarn logs.  Do a grep on jobid  in yarn logs. Do you see any logs regarding memory/resources.

Regards,

Sandeep

--
You received this message because you are subscribed to the Google Groups "Malhar" group.
To unsubscribe from this group and stop receiving emails from it, send an email to malhar-users...@googlegroups.com.
To post to this group, send email to malhar...@googlegroups.com.
Visit this group at http://groups.google.com/group/malhar-users.
For more options, visit https://groups.google.com/d/optout.

[Aalok Gupta]

Thanks for the reply Sandeep,

Yarn log excerpt

***************

2015-06-23 16:34:07,978 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 22142 for container-id container_1435047027715_0401_01_000001: 149.9 MB of 1 GB physical memory used; 1021.8 MB of 2.1 GB virtual memory used

2015-06-23 16:34:11,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 22142 for container-id container_1435047027715_0401_01_000001: 149.9 MB of 1 GB physical memory used; 1021.8 MB of 2.1 GB virtual memory used

 

added following prop @ hadoop  > yarn-site.xml

**************************

<property>

  <name>yarn.nodemanager.resource.memory-mb</name>

    <value>8192</value>

 </property>

 

My question is, what is the exact content of index.txt.

I am unable to find such file in the project code downloaded from malhar-Github

[Aalok Gupta]

One more thing.

To free some memory, we stopped few processes on the server where we hosted Data Torrent. The exception didn't come. The status of wordcount demo application moved to 'Running' :-)

However, now the operators are in 'Pending state'.

[Sandeep Deshmukh]

Hi Aalok,

The issue is still with resources not getting allocated. In the application overview | Physical details, 3 containers are planned but none of them is allocated.

Scroll below on the same page, Containers widget should give you details on state of each container. The state for each container should be ACTIVE. If not, memory is not getting allocated to the containers.

Regards,

Sandeep

[Aalok Gupta]

Thank for your reply Sandeep,

I checked the containers window. Its active.

[Sandeep Deshmukh]

If you see previous screenshot, the required containers is not getting allocated. Planned containers is 3 and allocated is 0. In the Containers widget, you should have 4 entries, 1 for STRAM (Application Master) and 3 more for application containers. 

What is the available memory in your cluster? Try with no other job running on the cluster.

Regards,

Sandeep

[Aalok Gupta]

Hi Sandeep,

What is the minimum memory requirement for running a datatorrent Demo, like wordcount.

All we got from free -m is :

[root@VDTVBUFCH02 datatorrent]# free -m                              

             total       used       free     shared    buffers     cached   

Mem:          3822       2792       1029          0         57       1107 

-/+ buffers/cache:       1627       2195                                       

Swap:         3999          0       3999                                          

[root@VDTVBUFCH02 datatorrent]#                                        

[Sandeep Deshmukh]

Around 5 GB. Have at least 6 GB available to YARN and try it out.

Regards,

Sandeep

[Aalok Gupta]

Thanks Sandeep! That was a good learning.

The issue got resolved once we did the following memory allocation:

In mapred-site.xml

mapreduce.map.memory.mb

mapreduce.reduce.memory.mb

In yarn-site.xml

yarn.nodemanager.resource.memory-mb yarn.scheduler.minimum-allocation-mb

ref : http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

[Raven]

My containers have available memory but some of them are in "Active" state and some in "Allocated state". All the operators are in pending state. Planned containers are allocated. What else might be the reason? How do I debug this scenario for the root cause?

[Gaurav Gupta]

Can you check if there are any exceptions in the Application Master logs? Can you also please share the screen shot for the Application Page? 

Thanks

- Gaurav

[Raven]

There are no errors in logs of 0001 container which I suppose is the master. Attached the status of containers.

[Ashwin Chandra Putta]

Raven,

The container ids are in the range of 3xxx. Seems like the containers are facing failures as soon as they are allocated and requests for new containers are being made (every new container allocation has id incremented by 1). Can you check the last killed container for exceptions, the logs in that container will be a good place to start looking for root cause. You can click on "retrieve killed" button in the Containers pane under physical tab to get the killed containers.

Regards,
Ashwin.

Regards,

Ashwin.

[MOOLAMREDDY SIDDA REDDY 15MCB1013]

iam getting an error how to configure the hadoop location in a data torrent can you please help me

thanks for advance