hadoop-2.0.0-cdh4.4.0 can use storm-yarn？

[zhangfenghua]

hi，all,

  I used the master version and the anfeng/hadoop-0.23 version,but I am not succeed till now,my yarn cluster version is hadoop-2.0.0-cdh4.4.0 

  I Realized that the master version is used by a 2.1.x version of hadoop,and the hadoop-0.23 version is used by a cdh4.3.x or 0.23.x hadoop version,how about the cdh4.4.x .

  I use the anfeng/hadoop-0.23 version and execute a command：mvn -f pom-cdh43.xml package ，it builded successful.

   then I execute storm-yarn launch,the output below:

--------------------------------

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/storm-yarn/lib/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

13/10/17 17:09:16 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.

13/10/17 17:09:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

13/10/17 17:09:17 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.

13/10/17 17:09:17 INFO yarn.StormOnYarn: Copy App Master jar from local filesystem and add to local environment

13/10/17 17:09:20 INFO yarn.StormOnYarn: Set the environment for the application master

13/10/17 17:09:20 INFO yarn.StormOnYarn: Setting up app master command:[/usr/local/jdk/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stderr, 2><LOG_DIR>/stdout]

13/10/17 17:09:20 INFO client.YarnClientImpl: Submitted application application_1381911735430_0154 to ResourceManager at datanode10/172.21.40.67:8032

application_1381911735430_0154

13/10/17 17:09:20 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.

-------------------------------------------------------

after than I want to check the stat of this submission,I run

 yarn logs -applicationId application_1381911735430_0154

the result looks like below:

----------------------------------------

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

13/10/17 17:09:22 INFO yarn.MasterServer: Starting the AM!!!!

13/10/17 17:09:24 INFO service.AbstractService: Service:com.yahoo.storm.yarn.client.AMRMClientImpl is inited.

13/10/17 17:09:25 INFO service.AbstractService: Service:com.yahoo.storm.yarn.client.AMRMClientImpl is started.

13/10/17 17:09:25 INFO yarn.StormMasterServerHandler: Storm master host:172.21.40.69

13/10/17 17:09:25 INFO yarn.MasterServer: launch nimbus

13/10/17 17:09:25 INFO yarn.StormMasterServerHandler: starting nimbus...

13/10/17 17:09:25 INFO yarn.MasterServer: launch ui

13/10/17 17:09:25 INFO yarn.StormMasterServerHandler: starting UI...

13/10/17 17:09:25 INFO yarn.MasterServer: launch 1 supervisors

13/10/17 17:09:25 INFO yarn.StormMasterServerHandler: adding 1 supervisors...

13/10/17 17:09:25 INFO yarn.StormAMRMClient: Added 1 supervisors, but not requesting containers now.

13/10/17 17:09:25 INFO yarn.StormMasterServerHandler: Running: /usr/local/jdk/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/*:./storm/storm-0.9.0-wip21/lib/* -Xmx768m -Dstorm.options=nimbus.host=localhost -Dlogfile.name=/var/log/hadoop-yarn/container/application_1381911735430_0154/container_1381911735430_0154_01_000001/ui.log backtype.storm.ui.core

13/10/17 17:09:25 INFO yarn.StormMasterServerHandler: Running: /usr/local/jdk/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/*:./storm/storm-0.9.0-wip21/lib/* -Xmx1024m -Dlogfile.name=/var/log/hadoop-yarn/container/application_1381911735430_0154/container_1381911735430_0154_01_000001/nimbus.log backtype.storm.daemon.nimbus

13/10/17 17:09:26 INFO yarn.MasterServer: Got a registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 1, }, 

13/10/17 17:09:26 INFO yarn.MasterServer: Max Capability <memory:8192, vCores:1>

13/10/17 17:09:26 INFO yarn.StormAMRMClient: Max Capability is now <memory:8192, vCores:1>

13/10/17 17:09:26 INFO yarn.MasterServer: Starting HB thread

13/10/17 17:09:26 INFO yarn.MasterServer: Starting launcher

13/10/17 17:09:26 INFO yarn.StormAMRMClient: Creating new ContainerRequest with <memory:8192, vCores:1> and 1

13/10/17 17:09:26 INFO yarn.StormAMRMClient: Requested container ask: Capability[<memory:8192, vCores:1>]Priority[0]ContainerCount[1]

13/10/17 17:09:26 INFO yarn.MasterServer: Starting Master Thrift Server

13/10/17 17:09:27 ERROR yarn.MasterServer: Unhandled error in AM: 

java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse.getAMResponse()Lorg/apache/hadoop/yarn/api/records/AMResponse;

        at com.yahoo.storm.yarn.client.AMRMClientImpl.allocate(AMRMClientImpl.java:179)

        at com.yahoo.storm.yarn.MasterServer$1.run(MasterServer.java:70)

13/10/17 17:09:27 INFO yarn.StormMasterServerHandler: stopping supervisors...

13/10/17 17:09:27 INFO yarn.StormMasterServerHandler: stopping UI...

13/10/17 17:09:27 INFO yarn.StormMasterServerHandler: stopping nimbus...

-----------------------------------------------------------------------------------

An Error :

java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse.getAMResponse

How about the Error be caused ?the version of my yarn cluster is not cdh4.3.x but the cdh4.4.x?

I wish somebody could give me a replay，thx

[Bobby Evans]

The pom on that branch (pom-cdh43.xml) is compiled against 2.0.0-cdh4.3.0. I am not sure what the differences are between that and cdh-4.4.0.  It looks like they may have upgraded to something closer to the 2.2 hadoop release that is about to go GA.  If so you may want to try the master branch.  We have not tested with cdh-4.4.0 so I don't know more then that.  But if you figure it out, please let us know so we can fix it accordingly.

--Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Andy Feng]

Folks:

There should not be meaningful difference b/w storm-yarn/hadoop-0.23 vs. anfeng/hadoop-0.23.

It looks like that CDH 4.4.0 has some API changes from CDH-4.3.0. AMResponse is not available at CDH –4.4.0.

We need to look into the internals of CDH –4.4.0.

Is there anyone in community want to give it a try?

Andy

From: zhangfenghua <zhang...@gmail.com>
Date: Thursday, October 17, 2013 2:13 AM
To: <storm...@googlegroups.com>
Subject: hadoop-2.0.0-cdh4.4.0 can use storm-yarn？

[杨晓青]

when  i deploy storm yarn on haddop Hadoop 2.0.0-cdh4.3.1，I  Encountered similar problems but not entirely sure，seems when  start storm process  some resource not ready, I hack StormMasterServerHandler.java add  "Thread.sleep(30000)".  you can add the sleep code and have a try. i not sure is a bug?

        private void startStormProcess() {

132             try {

133                 LOG.info("Running: " + Joiner.on(" ").join(buildCommands()));

134                 Thread.sleep(30000);

135                 ProcessBuilder builder =

136                         new ProcessBuilder(buildCommands());

137                 builder.redirectError(Redirect.INHERIT);

138                 builder.redirectOutput(Redirect.INHERIT);

139 

140                 _process = builder.start();

141             } catch (IOException e) {

142                 LOG.warn("Error starting nimbus process ", e);

143             } catch (Exception e1){

144                 LOG.warn("interrupt exception");

145                 }

在 2013年10月17日星期四UTC+8下午5时13分56秒，zhangfenghua写道：

[zhangfenghua]

HI，Andy,

     we use the Cloudra Manager to manage the hadoop cluster  ,So,now it is difficult to reduce the hadoop version using the manager web UI.The 2.0.0-cdh4.4.0 has different Api from the chd4.3.0,I think it is not easy to upgrade storm-yarn can be use by cdh4.4.0, But  also I hope your group would fix it .We really want to use the project on the cluster.

在 2013年10月19日星期六UTC+8上午7时04分38秒，Andy Feng写道：

[zhangfenghua]

hi ,Bobby

     I have tried all the verson the master，the hadoop-0.23 with two different pom,all of them failed,When I use the hadoop-0.23 branch with cdh4.3.pom ,I can build successful,But when I run the the lanch command,it said like 'java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse.getAMResponse()',the 2.0.0-cdh4.4.0 has different api from the 2.0.0-cdh4.3.0.I expect storm-yarn cancompatible the 2.0.0-cdh4.4.0.

在 2013年10月17日星期四UTC+8下午10时15分31秒，evans写道：

[Andy Feng]

Hi,

 

As stated earlier, we need to find out the exact changes by chd 4.4. You will not be able to use storm-yarn as it is.

Andy Feng

Sent from my iPhone