Storm 0.9.4 on Yarn

[Satish Mittal]

Hi All,

We are trying to use storm-yarn from yahoo github (that implements the AM for Storm) for launching storm. However we realized that this project is last updated in Nov 2004. It only supports an archived storm version (0.9.0-wip21), which seems to be an outdated version (around incubation time). This version doesn't support storm-kafka (within core), and we want to use storm netty transport with Kafka based Trident topology.

Since the last tagged release is 0.9.4, we tried to build storm-yarn with storm-0.9.4 and launch storm cluster on Yarn. The storm AM, nimbus server, supervisor and web UI daemons are able to come up. However the UI is not opening up. On accessing http://host:7070, it simply redirects to http://host:7070/index.html which says:

"Page not found"

The ui.log contents are:

2015-04-21 01:25:25 o.m.log [INFO] Logging to Logger[org.mortbay.log] via org.mortbay.log.Slf4jLog

2015-04-21 01:25:25 o.m.log [INFO] jetty-6.1.26

2015-04-21 01:25:25 o.m.log [INFO] Started SocketC...@0.0.0.0:7070

Has anyone been able to successfully bring storm-0.9.4 on Yarn?

Regards,

Satish Mittal

_____________________________________________________________

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.

[Bobby Evans]

We originally wrote this as a proof of concept and may move back to updating it in the future, but currently are concentrating on supporting multi-tenant storm instead.  If you want to run newer versions of storm on YARN.  http://slider.incubator.apache.org/ is probably the simplest approach.  If you look at http://slider.incubator.apache.org/docs/getting_started.html there are some instructions on how to set it up for storm.

 

- Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Satish Mittal]

Hi Bobby,

Thanks for providing the direction. Will explore slider for the same.

Just curious, by multi-tenant storm, are we planning to enable tasks of different topologies within same supervisor (since running multiple topologies is already supported in a storm cluster)?

Regards,

Satish

--
You received this message because you are subscribed to a topic in the Google Groups "storm-yarn" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/storm-yarn/mfXIXtp94Mw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to storm-yarn+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

_____________________________________________________________

[Bobby Evans]

By multi-tenant we mean multiple different users and topologies all running on shared hardware, but kept separate from one another through some basic security.  Most of our work on this will be released as a part storm 0.10.0.  We added in Authorization and Authentication, and allow topologies to run as the users that launched them.

 

- Bobby