HDFS Plugin Issue

[barno...@gmail.com]

Hey,

I am having an issue when I try to get the hdfs plugin working.  When I try to start azkaban-web i get the following error:

2013/07/17 17:24:36.038 -0500 ERROR [AzkabanWebServer] [Azkaban] Plugin class azkaban.viewer.hdfs.HdfsBrowserServlet

2013/07/17 17:24:36.040 -0500 INFO [AzkabanWebServer] [Azkaban] Source jar /usr/lib/azkaban/azkaban-2.1-web/plugins/viewer/hdfs/lib/azkaban-hdfs-viewer-2.1.jar

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/security/AccessControlException

at java.lang.Class.getDeclaredConstructors0(Native Method)

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2416)

at java.lang.Class.getConstructor0(Class.java:2726)

at java.lang.Class.getConstructor(Class.java:1676)

at azkaban.webapp.AzkabanWebServer.loadViewerPlugins(AzkabanWebServer.java:585)

at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:446)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.AccessControlException

at java.net.URLClassLoader$1.run(URLClassLoader.java:217)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:205)

at java.lang.ClassLoader.loadClass(ClassLoader.java:321)

at java.lang.ClassLoader.loadClass(ClassLoader.java:266)

... 6 more

Any ideas what may be causing this? Do I need to have Azkaban proxy to another user?

Thanks!

Brian A.

[Richard Park]

That looks like a bug in the plugin code.
Luckily there's an easy work around. In the conf directory and the plugin.properties file, there's the viewer.external.classpaths parameter. Point that directly to the hadoop core jar you're using.

Hope that fixes things.
-Richard

---

From: azkab...@googlegroups.com [azkab...@googlegroups.com] on behalf of barno...@gmail.com [barno...@gmail.com]
Sent: Wednesday, July 17, 2013 3:32 PM
To: azkab...@googlegroups.com
Subject: [azkaban] HDFS Plugin Issue

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[yu.ch...@gmail.com]

The error is saying it needs hadoop-core-xxx.jar, in your $HADOOP_HOME/

This is the jar that comes from your hadoop installation. It should be on the plugin classpath. Specifying the jar in plugin conf should solve that.

[barno...@gmail.com]

Hmm that doesn't seem to have worked.  My hdfs plugin plugin.properties file looks like this now:

viewer.name=HDFS

viewer.path=hdfs

viewer.order=1

viewer.hidden=false

viewer.external.classpaths=/usr/lib/hadoop/hadoop-core-1.2.0.1.3.0.0-107.jar

viewer.servlet.class=azkaban.viewer.hdfs.HdfsBrowserServlet

hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_1_0

azkaban.should.proxy=false

proxy.user=azkaban

proxy.keytab.location=

allow.group.proxy=false

[yu.ch...@gmail.com]

If it is still saying class not found of the same class, then we are still missing the jar.

azkaban uses a child classloader to load plugin. 

in this case, you can put the jar on the main process classpath so that the child classloader gets it regardless of its own config settings.

[barno...@gmail.com]

So would that go in the azaban.properties file then? Which property would I use toset that?

On Wednesday, July 17, 2013 5:32:14 PM UTC-5, barno...@gmail.com wrote:

[yu.ch...@gmail.com]

Simply put the jar on azkaban web server classpath.

[barno...@gmail.com]

Ok, got this working.  I just created a symlink in the hdfs plugin's lib directory to the hadoop-core jar.  Now that I have got everything running I am seeing another issue.  When I go into the HDFS viewer in the UI, it is actually displaying the structure of my linux file system, and not hdfs.  Do you know what would be causing that? Also,  I was able to successfully run the example wc job, and the pig job writes the wordcountin data to hdfs, but when the java program tries to read the file for some reason it is looking at the same on my linux filesystem instead of hdfs.  I was able to get the job to run successfully by just dropping the expected input file in the correct location on my linux filesystem.  One other thing I noticed is that even though the jobs actually run successfully, and I verified the output (which was written to my linux filesystem, not hdfs), for some reason when I look at the JobTracker UI it doesn't even show that anything has been run. 

Thanks so much for your help!

Brian 

On Wednesday, July 17, 2013 5:32:14 PM UTC-5, barno...@gmail.com wrote:

[Felix GV]

Usually, Hadoop runs locally by default, when it's not configured.

You probably need to have /etc/hadoop/conf/ (or whatever your correct config directory is) in the Azkaban classpath, so that the Hadoop jars pick it up and connect to the correct JobTracker and NameNode, instead of defaulting to running locally.

--
Felix

--

[yu.ch...@gmail.com]

Glad to hear that solved your previous problem.

Now the new one:

you need to put your $HADOOP_HOME/conf on the classpath as well. Otherwise, the program has no idea where your namenode/jobtrack/... are, and will default to local file system.

The example word count program does indeed load data from local to HDFS first, as it doesn't expect the data on HDFS magically by itself.

But it should be on HDFS from there. Your problem does seem like it is not run on HDFS at all. Did you install hadoop correctly? Everything is tested on real cluster and pseudo distributed mode. The local mode is way too stripped down and we never test with it.

[Richard Park]

Hmm, looks like the docs need to be beefed up.

--