Re: Storm on Yarn

[Andy Feng]

Worthy:

Storm.yaml should be downloaded via getStormConfig. You should NOT modify
nimbus host etc.
Please put the downloaded storm.yaml at the file path for storm client to
pick up (ex. ~/.storm/storm.yaml).

If you have any questions, please reach out to
<storm...@googlegroups.com>

Andy


On 12/5/13 12:18 PM, "Derek Dagit" <der...@yahoo-inc.com> wrote:

>This came across storm apache list. Should we redirect?
>
>
>-------- Original Message --------
>Subject: Storm on Yarn
>Date: Thu, 5 Dec 2013 14:13:43 -0600
>From: Worthy LaFollette <wor...@gmail.com>
>Reply-To: <d...@storm.incubator.apache.org>
>To: <d...@storm.incubator.apache.org>
>
>Question about running storm on YARN.
>
>Caveat: I'm still learning both YARN and Storm at this time.
>
>When running on a multi-node YARN installation, I am having to manually
>modify the storm.yaml on the YARN master node to point Nimbus to the host
>where the Storm AM is running. This is even after performing the
>getConfig
>to create it.
>
>How can I get this to be picked up dynamically? How will this ultimately
>work in the final release when the node may go down? Hopefully I am
>missing something somewhere as this seems fairly "static"-y at the moment.
>
>Regards,
>
>Worthy LaFollette
>
>
>

[Worthy LaFollette]

Resending now  I'm a member of Storm Yarn

On Thu, Dec 5, 2013 at 5:05 PM, Worthy LaFollette <wor...@gmail.com> wrote:

Hi Andy..

I should have given more information... 

I am following this plan: http://hortonworks.com/kb/storm-on-yarn-install-on-hdp2-beta-cluster/

I have two nodes running hosts "k" and "m" - the original YARN RM - runs on host "k"... 

When I execute the following on host "k:

storm-yarn launch

and then

yarn application -list 

I see storm application (nimbus) is running on host "m" (the other node).  

However, when I execute the following on "k" to get the storm.yaml I have an issue. 

storm-yarn getConfig -appId <application ID from list above> -output /.storm/storm.yaml

The nimbus host is shown as host "k", not "m" where Nimbus is actually running.

If I were to try and run any of the Storm applications shown on the page above, they fail because they can not find the nimbus host. 

I can manually edit storm.yaml to point to "m" and then these storm sample applications work.  

In line with your comment about not needing to edit storm.yaml - my question is why do I have to edit storm.yaml at all?  Why did it not pick up that the nimbus host was on "m" and not "k" ?  And as a followup, how does it work should host "m" fail?  There should be some resiliency here in the future.  Or does a failure of the single nimbus host represent a failed Storm application that would have to be restarted to pickup a new nimbus host?

Regards,

Worthy LaFollette

[Andy Feng]

Worthy:

Something is wrong in your environments, or execution steps.

(1) let's make sure that you don't have some exiting storm.yaml file in the environment:

   * ~/.storm/storm.yaml

   *<storm home>/conf/storm.yaml

(2) start a zookeeper server on some host, say "zk0.mycompany.com", with some listening port, say 5000.

(3) let's create a simple master.yaml file with zk servers and port defined:
storm.zookeeper.servers:
      - zk0.mycompany.com
storm.zookeeper.port: 5000

(4) launching storm-yarn command as described at README (https://github.com/yahoo/storm-yarn/blob/master/README.md).

 storm-yarn launch <master.yaml path>



(5) Download storm.yaml 


storm-yarn getStormConfig <master.yaml path> --appId <Application-ID> --output <storm.yaml path>



If you still have problems, please share the log files from yarn.


FYI, my recent storm-yarn talk has some detailed instructions as well. https://github.com/anfeng/share/blob/master/StormYARN_Groupon.pdf

Andy

Andy