CDH5 with storm-yarn

[Dhanasekaran Anbalagan]

Hi Guys,

I am trying to run storm-yarn in CDH5 . I am getting  error 'NATURAL JSON notation configured, but JAXB RI 2.1.10 not found'

tech@dvlab231:~$ java -version

java version "1.7.0_25"

tech@dvlab231:~$ hadoop version

Hadoop 2.2.0-cdh5.0.0-beta-1

console log:

http://paste.ubuntu.com/6664143/

can you guide me. how to fix this.

-Dhanasekaran

Did I learn something today? If not, I wasted it.

[Bobby Evans]

I am not an expert on this, but the error says that you need to have at least JAXB 2.1.12 for YARN to work properly.

Running mvn dependency:tree on master with 2.2.0-cdh5.0.0-SNAPSHOT and 2.2.0-cdh5.0.0-beta-1 both show that JAXB 2.2.3 is being pulled in as the correct dependency.  The test that is failing is the integration test, and sadly it uses the yarn command on your PATH to set up the classpath and run.  So if you have a different version of yarn installed on your path from what you are compiling against you might see something like this.  That is just a guess.

—Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dhanasekaran Anbalagan]

Hi Bobby,

thanks for kindly help. after your suggestion. I will able to start storm-application master.

tech ~ $ storm-yarn launch /home/tech/storm-yarn-app-try/storm-0.9.0-wip21/conf/storm.yaml

my job accepted resource manager.

tech ~ $ yarn application -list

13/12/31 14:05:08 INFO client.RMProxy: Connecting to ResourceManager at dvcloudlab231/192.168.70.231:8032

Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):1

                Application-Id    Application-Name    Application-Type      User     Queue             State       Final-State       Progress                       Tracking-URL

application_1388423038661_0014       Storm-on-Yarn                YARN      tech root.tech          ACCEPTED         UNDEFINED             0%                                N/A

tech ~ $

tech ~ $ storm-yarn getStormConfig -appId application_1388423038661_0014 -output ~/.storm/storm.yaml

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.0.0-0.cdh5b1.p0.57/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/tech/storm-yarn-app-try/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

13/12/31 14:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

13/12/31 14:01:21 INFO client.RMProxy: Connecting to ResourceManager at dvcloudlab231/192.168.70.231:8032

13/12/31 14:01:31 INFO yarn.StormOnYarn: No host/port returned for Application Master application_1388423038661_0014

Exception in thread "main" java.lang.NullPointerException

at com.yahoo.storm.yarn.StormMasterCommand.downloadStormYaml(StormMasterCommand.java:180)

at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:91)

at com.yahoo.storm.yarn.Client.execute(Client.java:142)

at com.yahoo.storm.yarn.Client.main(Client.java:148)

tech ~ $

for you reference my storm.yaml

http://paste.ubuntu.com/6670167/

my understanding:

Resource manager accept the job, But not able to start application master.

can you guide me. How to fix this.

-Dhanasekaran.

Did I learn something today? If not, I wasted it.

[Bobby Evans]

I would first start off my looking at the tech queue to see if it has any free space, or if it is full.  Alternatively many of the schedulers have limits on the number of concurrent applications that can be running in a queue.  You may want to check to see if that is the case.  The other possibility is that the AM just was never able to register with the RM.  You could check in the UI for the RM to see if it has any logs/app attempts for the storm application.

[Dhanasekaran Anbalagan]

HI Bobby,

I verified cluster utilization free. we are not configured in number of concurrent application to run cluster.

I am not understanding  AM just was never able to register with the RM. 

I RM Page. I can see 4 nodes connected to RM.

http://i.imgur.com/Skh8DoM.png

Also I tested DistributedShell yarn application works well. It's possible how to debug. I can't see not log.

when i click log link. It's redirected to RM home page.

can you please guide me. How to debug.

-Dhanasekaran.

Did I learn something today? If not, I wasted it.

[Dhanasekaran Anbalagan]

HI Bobby,

One more info. When I start  storm-yarn launch <config.yaml>

console-log

http://paste.ubuntu.com/6684865/

14/01/03 09:01:12 INFO yarn.StormOnYarn: Setting up app master command:[/usr/lib/jvm/java-7-oracle-cloudera/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr]

I want to know LOG_DIR variable absolute path.

I verified container folder no logs.

:~# ls -ltr /var/log/hadoop-yarn/container

total 0

Did I learn something today? If not, I wasted it.

[Bobby Evans]

<LOG_DIR> is expanded by YARN before it launches the command, and it is a special directory created just for that container.

If I were you I would start looking in the logs for the RM to see if it tried to launch the container at all.  It could be that the resources requested for the AM are too large for it to schedule.  That is all that I can think of at this point.

—Bobby

application_1388423038661_0014       Storm-on-Yarn               YARN     techroot.tech         ACCEPTED        UNDEFINED            0%                               N/A

[Dhanasekaran Anbalagan]

Hi Bobby,

Thanks for reply, After I start storm-yarn,  I ween in to Resource manger logs. I seen this exception. 

2014-01-05 17:10:37,337 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1388959450407_0001_02_000001, NodeId: dvcloudlab233:8041, NodeHttpAddress: dvcloudlab233:8042, Resource: <memory:5013, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.70.233:8041 }, ] resource=<memory:5013, vCores:1> queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:36864, vCores:12>
2014-01-05 17:10:37,337 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:36864, vCores:12>
2014-01-05 17:10:37,337 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1388959450407_0001 failed 2 times due to AM Container for appattempt_1388959450407_0001_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
	at org.apache.hadoop.util.Shell.run(Shell.java:379)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)

In my case I have 3 nodes running node manager. Each node has 16GB memory and  each as 4 phy. core.

I configured yarn.nodemanager.resource.memory-mb=12GB

yarn.nodemanager.resource.cpu-vcores=4 cores.

As per the below log in my understanding I trying to start application master in 192.168.70.233 machine. it's failed. I went in .233 machine

root@dvcloudlab233:~# free -m total used free shared buffers cached Mem: 15889 1854 14035 0 22 227 -/+ buffers/cache: 1603 14285 Swap: 48659 0 48659

Resource manger full log.

http://paste.ubuntu.com/6699701/

please guide me.

[Bobby Evans]

What this is saying is that it tried to run the app master but it crashed with an exit code of 1.  Is there anything in the node manager logs for this app?

—Bobby

From: Dhanasekaran Anbalagan <bugc...@gmail.com>
Date: Sunday, January 5, 2014 at 4:24 PM
To: "Yahoo! Inc." <ev...@yahoo-inc.com>
Cc: "storm...@googlegroups.com" <storm...@googlegroups.com>
Subject: Re: CDH5 with storm-yarn

[Dhanasekaran Anbalagan]

HI Bobby,

finny I am able to run stom on CDH5. thanks for you support.

http://paste.ubuntu.com/6708636/

my mistakes.

1. thifit server using 9000 port number. I my case cloudera-scm-agent using 9000 port number. I change the agent port number. 

2. storm.local.dir: "/mnt/storm-local" I am not created After I fixed it. with proper permissions in all the node.

I got hit from above your email, I went into the node manager logs. all the logs in hdfs file system.

hdfs@dvcloudlab231:~$ hadoop fs -ls /tmp/logs/hdfs/logs/ 

Found 9 items

drwxrwx---   - hdfs hadoop          0 2014-01-06 14:18 /tmp/logs/hdfs/logs/application_1388959450407_0009

drwxrwx---   - hdfs hadoop          0 2014-01-06 14:23 /tmp/logs/hdfs/logs/application_1388959450407_0010

drwxrwx---   - hdfs hadoop          0 2014-01-06 14:24 /tmp/logs/hdfs/logs/application_1388959450407_0011

drwxrwx---   - hdfs hadoop          0 2014-01-06 14:27 /tmp/logs/hdfs/logs/application_1388959450407_0012

drwxrwx---   - hdfs hadoop          0 2014-01-06 14:29 /tmp/logs/hdfs/logs/application_1389036527411_0001

drwxrwx---   - hdfs hadoop          0 2014-01-06 14:34 /tmp/logs/hdfs/logs/application_1389036527411_0002

drwxrwx---   - hdfs hadoop          0 2014-01-07 06:26 /tmp/logs/hdfs/logs/application_1389037460221_0001

drwxrwx---   - hdfs hadoop          0 2014-01-07 06:41 /tmp/logs/hdfs/logs/application_1389037460221_0002

drwxrwx---   - hdfs hadoop          0 2014-01-07 07:19 /tmp/logs/hdfs/logs/application_1389037460221_0003

hdfs@dvcloudlab231:~$ 

one more info. I am running storm-yarn in hdfs user.

thanks bobby.

-Dhanasekaran

Did I learn something today? If not, I wasted it.

[Bobby Evans]

I am glad to see it working.