Workers not working (backtype.storm.daemon.worker class not found)

[Gufran Pathan]

Hi,

I’m a newbie with Storm and I have setup a Storm-on-Yarn on an HDP cluster using the instructions at http://hortonworks.com/kb/storm-on-yarn-install-on-hdp2-beta-cluster/ and the storm-yarn-master from https://github.com/anfeng/storm-yarn.

I’m able to get Nimbus running and even submit topologies and see them on Storm UI. However, the spouts and the bolts don’t seem to be “working” (0 counts of tuples emitted).

I did some digging around and realized that my worker daemons are not starting. The supervisor log spits out these:

2014-03-13 11:22:03 b.s.d.supervisor [INFO] 18bf93a1-1cea-4e99-93da-8f36a4e9c056 still hasn't started

 I tried launching the worker command from the “Launching worker with command” line in the supverviser log and I got this error:

 

Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/daemon/worker

Caused by: java.lang.ClassNotFoundException: backtype.storm.daemon.worker

        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Could not find the main class: backtype.storm.daemon.worker.  Program will exit.

 

It looks like it can’t find the worker class although it’s present in the storm-core jar.

Any ideas on how I can proceed with troubleshooting this? I’ve attached the nimbus and the supervisor logs.

Thanks in advance! 

-

Gufran Pathan

[Gufran Pathan]

Hi guys,

Update: Turns out there was a mistake in the way I was debugging. The path in the commands that are executed are relative and I wasn’t launching them from the right location hence the class not found error. 

When I launched it from the correct location, I got a Major.minor 51 error. The command that the supervisor is using to launch the workers is /usr/jdk64/…/bin/java which is a JDK 1.6. However, my JAVA_HOME points to JDK 1.7. I don’t know why (and from where) it’s taking the JDK 1.6 path. My update-alternatives –config java is also set to 1.7.

I did a very dirty hack, replacing the JDK 1.6 folder contents with that of JDK 1.7 and my workers are launching just fine now. Clearly, I can’t do this on all nodes. Any idea why it’s taking the wrong path?

The supervisor logs show that the Java version is set to the correct version. Log below:

This:

2014-03-13 11:05:40 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_51

and this:

2014-03-13 11:05:40 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/java/jdk1.7.0_51/jre 

Any idea where it’s picking up 1.6 from?

Thanks in advance!

--

Gufran Pathan

[Bobby Evans]

I’m not positive on this.  What version of storm-yarn are you using?  There have been some fixes recently on https://github.com/yahoo/storm-yarn/ that deal with JAVA_HOME, and I want to be sure you have the correct version of the software before I tell you what to do.

—Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gufran Pathan]

I took the master branch from https://github.com/anfeng/storm-yarn because I have Hadoop 2.2 installed and that looked like the only storm-yarn version compatible with Hadoop 2.2.

[Bobby Evans]

That is what I was scared of.  Andy’s branch is rather old compared to the official release, I am a bit disappointed that Hortonworks pointed to it instead of https://github.com/yahoo/storm-yarn/

I think the master branch is what you want.  Also the code on the master branch has been updated to not need JDK7 any more, it only needs JDK6.

—Bobby 

[Gufran Pathan]

Aah, no I wouldn't blame Hortonworks. Their page (http://hortonworks.com/kb/storm-on-yarn-install-on-hdp2-beta-cluster/) pointed to the Yahoo Storm-Yarn master branch at git. 

I tried installing it, but I think I had some issue with compatibility. Is it compatible with Hadoop 2.2? If not, which is the most updated version compatible with Hadoop 2.2?

[Bobby Evans]

Trunk is compiled against 2.1 and should be compatible with 2.2 as well.  I have compiled it against 2.2 and 2.3 and it works for both of them.