CDAP router cannot be started failing with java.lang.NoClassDefFoundError - org/apache/hadoop/conf/

[Elkhan]

Hi all,

I'm trying to install CDAP on RHEL, and run with different username. I installed CDAP using Manual installation using Packages.

CDAP router cannot be started, failing with this error:

2016-06-23 12:59:54,105 - INFO  [main:c.c.c.g.r.RouterMain@63] - Initializing Router...

2016-06-23 12:59:54,178 - ERROR [main:c.c.c.g.r.RouterMain@88] - org/apache/hadoop/conf/Configuration

java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

        at co.cask.cdap.common.guice.ConfigModule.<init>(ConfigModule.java:43) ~[co.cask.cdap.cdap-common-3.4.2.jar:na]

        at co.cask.cdap.gateway.router.RouterMain.createGuiceInjector(RouterMain.java:113) [co.cask.cdap.cdap-gateway-3.4.2.jar:na]

        at co.cask.cdap.gateway.router.RouterMain.init(RouterMain.java:80) [co.cask.cdap.cdap-gateway-3.4.2.jar:na]

        at co.cask.cdap.common.runtime.DaemonMain.doMain(DaemonMain.java:36) [co.cask.cdap.cdap-common-3.4.2.jar:na]

        at co.cask.cdap.gateway.router.RouterMain.main(RouterMain.java:50) [co.cask.cdap.cdap-gateway-3.4.2.jar:na]

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration

        at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_75]

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_75]

        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_75]

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_75]

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_75]

Any hints for the cause of failure of starting CDAP service ?

The UI component can be accessed at my_ip_address:9999, but it just flashes "Unable to connect to CDAP Router Attempting to connect... "

My Hadoop version is 2.6.0-cdh5.4.2-xyz-1.3.1 which i set in cdap-env.sh file:

export OPTS="${OPTS} -Dhdp.version=2.6.0-cdh5.4.2-xyz-1.3.1"

I attached cdap-site.xml to this post too.

Thanks.

----------

These are the steps i followed after installing CDAP to run CDAP under different user name xyz (I followed this post https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/cdap-user/SuO5CtpLrk8/PP_2ovy-AQAJ):

sudo chown -R xyz:xyz /var/cdap/run

sudo chown -R xyz:xyz /var/log/cdap

sudo chown -R xyz:xyz /var/run/cdap

sudo chown -R xyz:xyz /var/tmp/cdap

sudo chown -R xyz:xyz /etc/cdap

sudo chown -R xyz:xyz /opt/cdap

./hdfs dfs -mkdir -p /user/xyz && ./hadoop fs -chown xyz /user/xyz && ./hadoop fs -chgrp xyz /user/xyz

./hadoop fs -mkdir -p /cdap && ./hadoop fs -chown xyz /cdap

./hadoop fs -mkdir -p /cdap/tx.snapshot && ./hadoop fs -chown xyz /cdap/tx.snapshot

I also created my custom configuration folder under /etc/cdap/conf.mycdap :

ln -sfn /etc/cdap/conf.mycdap conf

I also modified all scripts where i replaced this line

su cdap -c "$SVC_COMMAND"

with 

su xyz -c "$SVC_COMMAND"

Then for starting CDAP services I run this command:

for i in `ls /etc/init.d/ | grep cdap` ; do service $i start ; done

[Elkhan]

Full exception from router log (/var/log/cdap/router--<my_host>.log) is :

WARN: could not find Hadoop and HBase libraries

Thu Jun 23 14:23:26 EDT 2016 Starting Java auth-server service on <my_host>

ERROR: Failed to start Java auth-server service

WARN: could not find Hadoop and HBase libraries

Thu Jun 23 14:23:28 EDT 2016 Starting Java kafka-server service on <my_host>

ERROR: Failed to start Java kafka-server service

WARN: could not find Hadoop and HBase libraries

ERROR: Unknown/unsupported version of HBase found: 2016-06-23 14:23:31,242 - ERROR [main:c.c.c.d.u.h.HBaseVersion@109] - Unable to determine HBase version from string 'null', are HBase classes available?

2016-06-23 14:23:31,248 - ERROR [main:c.c.c.d.u.h.HBaseVersion@110] - Exception was: 

java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.VersionInfo

at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_75]

at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_75]

at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]

at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_75]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_75]

at java.lang.Class.forName0(Native Method) ~[na:1.7.0_75]

at java.lang.Class.forName(Class.java:191) ~[na:1.7.0_75]

at co.cask.cdap.data2.util.hbase.HBaseVersion.<clinit>(HBaseVersion.java:75) ~[co.cask.cdap.cdap-data-fabric-3.4.2.jar:na]

unknown

WARN: could not find Hadoop and HBase libraries

Thu Jun 23 14:23:31 EDT 2016 Starting Java router service on <my_host>

ERROR: Failed to start Java router service

Thu Jun 23 14:23:33 EDT 2016 Starting ui service on <my_host>

[Sreevatsan Raman]

Hey Eikhan,

CDAP needs HBase and Hadoop client libraries installed as a pre-requisite. Please do install the client libraries that is compatible with your Hadoop/Hbase versions.

Relevant section from the docs:

http://docs.cask.co/cdap/3.4.2/en/admin-manual/installation/packages.html#admin-installation-packages

• You can download the Hadoop client and HBase client libraries, and then install them on the hosts running CDAP services. No Hadoop or HBase services need be running.
  

Let us know if you have any additional questions.

Thanks,

Sree

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/b66dd74b-7957-4fd6-9ea9-a86b3663fee9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Elkhan]

Hi Sreevatsan,

Thank you.

Actually Hadoop is installed and running (also Hbase, Zookeper and other services).

What are the environment variables CDAP expects to be exported ? (JAVA_HOME, HADOOP_HOME ?)

Is there any link to documentation on CDAP site describing how to install Hadoop and Hbase clients using CDAP services ?

Thanks.

[Derek Wood]

Hi Eikhan,

What do you get when you run "hbase classpath" as user "xyz"?

The CDAP init scripts run this command to determine the hbase/hadoop classpath and append to its own.  (The hbase command itself similarly runs "hadoop classpath").  CDAP will use HBASE_HOME and JAVA_HOME if set, otherwise it'll expect "hbase" and "java" to be in the $PATH.

Thanks,

-Derek

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/0f492784-f964-4eeb-9285-1bdebb712dbc%40googlegroups.com.

[Elkhan]

HI Derek,

I added HBASE_HOME and JAVA_HOME to environment variables. But still it fails.

When i run hadoop classpath and hbase classpath, they both give me correct classpath.

But when i try to start CDAP services, it still fails with same reason, also with warning that could not find Hadoop and Hbase libraries:

for i in `ls /etc/init.d/ | grep cdap` ; do service $i start ; done

[Derek Wood]

Hi Elkhan,

You can try running "sudo service cdap-master classpath", and it will print out the classpath it detects.  It should contain both hbase and hadoop client library directories/jars.

The presence of that warning during startup indicates it cannot run, or is not getting any results from, "hbase classpath".  See https://github.com/caskdata/cdap/blob/develop/cdap-common/bin/common.sh#L215, which is triggered when it cannot set HBASE_CP here: https://github.com/caskdata/cdap/blob/develop/cdap-common/bin/common.sh#L198-L202

Thanks,

-Derek

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/449c7598-7ffd-4ad4-9d61-021b907e1b4d%40googlegroups.com.

[Elkhan]

Thanks, Derek.

It helped me to progress little bit.

The issue was, when i was runnign this command for starting all services:

for i in `ls /etc/init.d/ | grep cdap` ; do service $i start ; done

As I do not run it as sudo root user, but xyz user, and as it is not mentioned, even though inside init scripts i replaced all 'su cdap -c "$SVC_COMMAND"' line with 'su xyz -c "$SVC_COMMAND"' line, it was not running it as xyz user.

So instead of that i run all these 5 CDAP services from /opt/cdap directory:

cdap-auth-server

cdap-kafka-server

cdap-master

cdap-router

cdap-ui

From /opt/cdap/<service>/bin/ directory, for example for master service:

cd /opt/cdap/master/bin

./svc-master start

Then cdap-auth-server and cdap-kafka-server succeeds without any exception.

But master fails with this error:

Running startup checks -- this may take a few minutes

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cdap/master/lib/ch.qos.logback.logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/.../hbase/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/.../hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]

...

2016-06-23 20:59:35,130 - INFO  [main:c.c.c.c.s.ConfigurationLogger@46] - Important config settings:

Exception in thread "main" java.lang.ClassCastException: com.sun.org.apache.xerces.internal.dom.DeferredElementNSImpl cannot be cast to org.w3c.dom.Text

        at co.cask.cdap.common.conf.Configuration.loadResource(Configuration.java:1704)

        at co.cask.cdap.common.conf.Configuration.loadResources(Configuration.java:1612)

        at co.cask.cdap.common.conf.Configuration.getProps(Configuration.java:1582)

        at co.cask.cdap.common.conf.Configuration.get(Configuration.java:495)

        at co.cask.cdap.common.startup.ConfigurationLogger.logImportantConfig(ConfigurationLogger.java:48)

        at co.cask.cdap.master.startup.MasterStartupTool.main(MasterStartupTool.java:57)

[Sreevatsan Raman]

Hey Elkhan, 

Is your cdap-site.xml valid? Perhaps you can try to run xmllint on your config file. Here is an online XMLLint you can use: http://www.freeformatter.com/xml-formatter.html

Thanks,
Sree

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/b882fd96-0c3c-47c0-b747-c7f3db1d331e%40googlegroups.com.

[Elkhan]

This was related to wrong Zookeeper quorum property in cdap-site.xml.

Maybe exception message could have been little more detailed.

Now it is fixed.

[Sreevatsan Raman]

You are right, opened this JIRA based on your feedback: https://issues.cask.co/browse/CDAP-6285

Please feel free to reach out incase you need any additional help. 

Thanks,

Sree

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/c2de85ed-cb6f-4e46-8b31-b3c4445feb3e%40googlegroups.com.