Distributed shell example is failed

[Pavel Mezentsev]

Hello all!

I learn YARN and investigate kitten project.
I started your distributed shell example with command (also I copied kitten-master-0.1.0-jar-with-dependencies.jar in current dir)
  hadoop jar ../../client/target/kitten-client-0.1.0-jar-with-dependencies.jar distshell.lua distshell
but it was failed.

Example wrote to the standard output:
12/07/08 16:36:11 INFO service.ApplicationsManagerConnectionFactory: Connecting to ResourceManager at: /0.0.0.0:8032
12/07/08 16:36:12 INFO service.YarnClientServiceImpl: Setting up application submission context for the application master
12/07/08 16:36:13 INFO service.YarnClientServiceImpl: Submitting application to the applications manager
12/07/08 16:36:16 INFO client.KittenClient: Application is running, but did not specify a tracking URL
and stopped.

Resource manager wrote in web interface that application had been failed:

Please tell me, where I make mistakes? And what this example will do in success?
I use YARN from Cloudera's CDH4. Hadoop was started in pseudo-distributed mode on single computer.
Computer is under Ubuntu 12.04

Best regards
Mezentsev Pavel

[Павел Мезенцев]

I fogot screen shot of resource manager web interface.
It is in attachment

Pavel.

2012/7/8 Pavel Mezentsev <pa...@mezentsev.org>

[Josh Wills]

Hey Pavel,

It's useful to look at the logs that the ApplicationMaster generates to stdout and stderr, which you can get to by clicking on the job links on the UI page that you sent. If you see an error code mentioning 143, it means that the application was killed for exceeding virtual memory limits.

Out of curiousity, are you running on RHEL 6? There is some funky stuff w/MALLOC_ARENA_MAX that you may need to modify in order to get the virtual memory consumption of the container down. See the very last FAQ here:

http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html

and this JIRA issue: https://issues.apache.org/jira/browse/HADOOP-7154

In Cloudera Manager, you can set this value globally to be MALLOC_ARENA_MAX=1 on the configuration panel for YARN.

Best,

Josh

--

Director of Data Science

Cloudera

Twitter: @josh_wills

[anand.prab...@gmail.com]

Hi Josh,

Old thread I know, but I'm having the same issue on the same setup (Ubuntu 12.04) as Pavel and haven't found the solution elsewhere on the web. The first failure I saw in the stdout log follows, any thoughts? 

2013-07-05 17:14:17,122 INFO  [pool-3-thread-1] service.ApplicationMasterServiceImpl (ApplicationMasterServiceImpl.java:startUp(350)) - Starting container: container_1373050319534_0015_01_000002

2013-07-05 17:14:17,663 INFO  [pool-1-thread-1] service.ApplicationMasterServiceImpl (ApplicationMasterServiceImpl.java:checkContainerStatuses(282)) - container_1373050319534_0015_01_000002 failed with exit code = -1

Also, out of curiosity, what are your/Cloudera's plans for the future of Kitten?

Many thanks in advance,

Anand

[Josh Wills]

Hey Anand,

The useful debug messages are usually in the logs for the nodes, not the app master. If I had to guess, you're probably having an issue like the one here:

http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201210.mbox/%3C5089602...@zfabrik.de%3E

where the fix is to tweak the yarn.nodemanager.vmem-pmem-ratio property for the system.

Re: Kitten's future, no decisions yet. I work on lots of things, some of which become bigger things (like Apache Crunch), some of which don't really go anywhere or are just neat demos (like https://github.com/cloudera/matching ), and some that are still just getting started but have been very useful in my work w/my customers (like Cloudera ML). Kitten was a lot of fun to write, but it hasn't really come up in my day-to-day data science work in the past year, so I haven't had many excuses to hack on it. If folks want to start sending pull requests, I'll be happy to review them and incorporate them into the project.

Best,

Josh