Help please starting the Web Server

[chris...@derbysoft.net]

Hi all,

I have a fresh install of the Web Server (setting up a multiple executor model), and I'm having issues getting it to start.

I've followed the steps here to set up the Jetty SSL stuff:  http://www.eclipse.org/jetty/documentation/current/configuring-ssl.html#tls-and-ssl-versions

I have the jetty.crt, jetty.csr and jetty.key files.  I've imported the .crt into the default Java keystore here:

/usr/java/latest/jre/lib/security/cacerts

which worked fine.  The property files are updated in the azkaban.properties file as well:

jetty.maxThreads=25

jetty.ssl.port=8443

jetty.port=8081

jetty.keystore=/usr/java/latest/jre/lib/security/cacerts

jetty.password=thekeystorepassword

jetty.keypassword=azkaban

jetty.truststore=/usr/java/latest/jre/lib/security/cacerts

jetty.trustpassword=azkaban

My first question is: Which cacerts files need that certificate imported? I have a multiple node Hadoop cluster, a server for the Web Server, and a server for the Executor Server. Do all of the servers in the cluster need it, or just a subset?

When I start the Web Server, this is what I get:

root@bdprodm14:[84]:/opt/azkaban-web-server> bin/azkaban-web-start.sh 

Using Hadoop from 

Using Hive from 

bin/..

:bin/../lib/azkaban-2.5.0.jar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-configuration-1.8.jar:bin/../lib/commons-dbcp-1.4.jar:bin/../lib/commons-dbutils-1.5.jar:bin/../lib/commons-email-1.2.jar:bin/../lib/commons-fileupload-1.2.1.jar:bin/../lib/commons-io-2.4.jar:bin/../lib/commons-jexl-2.1.1.jar:bin/../lib/commons-lang-2.6.jar:bin/../lib/commons-logging-1.1.1.jar:bin/../lib/commons-pool-1.6.jar:bin/../lib/guava-13.0.1.jar:bin/../lib/h2-1.3.170.jar:bin/../lib/httpclient-4.2.1.jar:bin/../lib/httpcore-4.2.1.jar:bin/../lib/jackson-core-asl-1.9.5.jar:bin/../lib/jackson-mapper-asl-1.9.5.jar:bin/../lib/jetty-6.1.26.jar:bin/../lib/jetty-util-6.1.26.jar:bin/../lib/joda-time-2.0.jar:bin/../lib/jopt-simple-4.3.jar:bin/../lib/junit-4.8.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/mail-1.4.5.jar:bin/../lib/mysql-connector-java-5.1.28.jar:bin/../lib/servlet-api-2.5.jar:bin/../lib/slf4j-api-1.6.1.jar:bin/../lib/slf4j-log4j12-1.6.4.jar:bin/../lib/velocity-1.7.jar:bin/../lib/velocity-tools-2.0.jar:bin/../extlib/mysql-connector-java-5.1.38-bin.jar:bin/../plugins/*/*.jar:/conf:/*:/conf:/lib/*

2016/05/26 16:41:09.878 +0000 ERROR [AzkabanWebServer] [Azkaban] Starting Jetty Azkaban Executor...

2016/05/26 16:41:09.910 +0000 INFO [AzkabanServer] [Azkaban] Loading azkaban settings file from bin/../conf

2016/05/26 16:41:09.910 +0000 INFO [AzkabanServer] [Azkaban] Loading azkaban properties file

2016/05/26 16:41:09.927 +0000 INFO [AzkabanWebServer] [Azkaban] Setting up Jetty Https Server with port:8443 and numThreads:25

2016/05/26 16:41:09.965 +0000 INFO [log] [Azkaban] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

2016/05/26 16:41:10.009 +0000 INFO [AzkabanWebServer] [Azkaban] Loading user manager class azkaban.user.XmlUserManager

2016/05/26 16:41:10.055 +0000 INFO [XmlUserManager] [Azkaban] Loading user azkaban

2016/05/26 16:41:10.058 +0000 INFO [XmlUserManager] [Azkaban] Loading user gobblin

2016/05/26 16:41:10.058 +0000 INFO [XmlUserManager] [Azkaban] Loading user metrics

2016/05/26 16:41:10.642 +0000 INFO [ExecutorManager] [Azkaban] Cleaning old logs from execution_logs

2016/05/26 16:41:10.642 +0000 INFO [AzkabanWebServer] [Azkaban] Loading JDBC for project management

2016/05/26 16:41:10.649 +0000 INFO [ProjectManager] [Azkaban] Project version retention is set to 3

2016/05/26 16:41:10.717 +0000 INFO [ExecutorManager] [Azkaban] Cleaning old log files before 2016-03-03T16:41:10.653Z

2016/05/26 16:41:10.719 +0000 INFO [ExecutorManager] [Azkaban] Cleaned up 0 log entries.

2016/05/26 16:41:10.770 +0000 INFO [TriggerManager] [Azkaban] TriggerManager loaded.

2016/05/26 16:41:10.770 +0000 INFO [AzkabanWebServer] [Azkaban] Loading built-in checker and action types

2016/05/26 16:41:10.777 +0000 INFO [CheckerTypeLoader] [Azkaban] Registering checker BasicTimeChecker

2016/05/26 16:41:10.778 +0000 INFO [CheckerTypeLoader] [Azkaban] Registering checker SlaChecker

2016/05/26 16:41:10.778 +0000 INFO [CheckerTypeLoader] [Azkaban] Registering checker ExecutionChecker

2016/05/26 16:41:10.778 +0000 INFO [ActionTypeLoader] [Azkaban] Registering action ExecuteFlowAction

2016/05/26 16:41:10.778 +0000 INFO [ActionTypeLoader] [Azkaban] Registering action KillExecutionAction

2016/05/26 16:41:10.778 +0000 INFO [ActionTypeLoader] [Azkaban] Registering action AlertAction

2016/05/26 16:41:10.778 +0000 INFO [ActionTypeLoader] [Azkaban] Registering action CreateTriggerAction

2016/05/26 16:41:10.779 +0000 INFO [AzkabanWebServer] [Azkaban] Loading trigger based scheduler

2016/05/26 16:41:10.783 +0000 INFO [AzkabanWebServer] [Azkaban] Loading plug-in checker and action types

2016/05/26 16:41:10.783 +0000 ERROR [AzkabanWebServer] [Azkaban] plugin path plugins/triggers doesn't exist!

2016/05/26 16:41:10.784 +0000 INFO [AzkabanWebServer] [Azkaban] Setting timezone to UTC

2016/05/26 16:41:10.784 +0000 INFO [AzkabanWebServer] [Azkaban] Registering MBeans...

2016/05/26 16:41:10.801 +0000 INFO [AzkabanWebServer] [Azkaban] Bean azkaban.jmx.JmxJettyServer registered.

2016/05/26 16:41:10.805 +0000 INFO [AzkabanWebServer] [Azkaban] Bean azkaban.jmx.JmxTriggerManager registered.

2016/05/26 16:41:10.808 +0000 INFO [AzkabanWebServer] [Azkaban] Bean azkaban.jmx.JmxExecutorManager registered.

2016/05/26 16:41:10.811 +0000 INFO [AzkabanWebServer] [Azkaban] Setting up web resource dir web/

2016/05/26 16:41:10.900 +0000 ERROR [AzkabanWebServer] [Azkaban] Plugin class azkaban.viewer.hdfs.HdfsBrowserServlet

2016/05/26 16:41:10.902 +0000 ERROR [AzkabanWebServer] [Azkaban] External library path /opt/azkaban-web-2.5.0/plugins/viewer/hdfs/extlib/* not found.

2016/05/26 16:41:10.907 +0000 INFO [AzkabanWebServer] [Azkaban] Source jar /opt/azkaban-web-2.5.0/plugins/viewer/hdfs/lib/azkaban-hdfs-viewer-2.5.0.jar

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/security/AccessControlException

at java.lang.Class.getDeclaredConstructors0(Native Method)

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)

at java.lang.Class.getConstructor0(Class.java:3075)

at java.lang.Class.getConstructor(Class.java:1825)

at azkaban.webapp.AzkabanWebServer.loadViewerPlugins(AzkabanWebServer.java:1089)

at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:771)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.AccessControlException

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 6 more

The first error doesn't really give me much to go on.  Some JETTY error :)  There's nothing listening on either port 8081 or 8443.

The second error really is stumping me.  That plugin extlib directory most definitely is there, and open to reading/executing, so I have no idea why it seems to not find it!

root@bdprodm14:[91]:/opt/azkaban-web-server> ls -l

total 44

-rw-r--r-- 1 root root    0 May 26 16:30 azkaban-access.log

-rw-r--r-- 1 root root  105 Apr 22  2014 azkaban.version

-rw-r--r-- 1 root root 6252 May 26 16:41 azkaban-webserver.log

drwxr-xr-x 2 root root 4096 May 26 16:42 bin

drwxr-xr-x 2 root root 4096 May 26 16:27 conf

-rw-r--r-- 1 root root    6 May 26 16:41 currentpid

drwxr-xr-x 2 root root 4096 May 26 16:27 extlib

drwxr-xr-x 2 root root 4096 May 26 16:27 lib

drwxr-xr-x 3 root root 4096 May 26 16:27 plugins

drwxr-xr-x 2 root root 4096 May 26 16:30 temp

drwxr-xr-x 6 root root 4096 May 26 16:27 web

root@bdprodm14:[92]:/opt/azkaban-web-server> cd plugins/

root@bdprodm14:[93]:/opt/azkaban-web-server/plugins> ls -l

total 4

drwxr-xr-x 3 root root 4096 May 26 16:27 viewer

root@bdprodm14:[94]:/opt/azkaban-web-server/plugins> cd viewer/

root@bdprodm14:[95]:/opt/azkaban-web-server/plugins/viewer> ls -l

total 8

drwxr-xr-x 5 root root 4096 May 26 16:27 azkaban-hdfs-viewer-2.5.0

lrwxrwxrwx 1 root root   64 May 26 16:27 hdfs -> /opt/azkaban-web-server/plugins/viewer/azkaban-hdfs-viewer-2.5.0

root@bdprodm14:[96]:/opt/azkaban-web-server/plugins/viewer> cd hdfs

root@bdprodm14:[97]:/opt/azkaban-web-server/plugins/viewer/hdfs> ls -l

total 16

drwxr-xr-x 2 root root 4096 May 26 16:27 conf

drwxr-xr-x 2 root root 4096 May 26 16:27 extlib

drwxr-xr-x 2 root root 4096 May 26 16:27 lib

-rwxr-xr-x 1 root root  112 Apr 22  2014 package.version

root@bdprodm14:[98]:/opt/azkaban-web-server/plugins/viewer/hdfs> cd extlib/

root@bdprodm14:[99]:/opt/azkaban-web-server/plugins/viewer/hdfs/extlib> ls -l

total 3840

-rwxr-xr-x 1 root root 3928530 Apr 22  2014 hadoop-core-1.0.4.jar

root@bdprodm14:[100]:/opt/azkaban-web-server/plugins/viewer/hdfs/extlib> jar tvf hadoop-core-1.0.4.jar | grep AccessControlException

   806 Wed Oct 03 05:05:14 UTC 2012 org/apache/hadoop/fs/permission/AccessControlException.class

   743 Wed Oct 03 05:05:16 UTC 2012 org/apache/hadoop/security/AccessControlException.class

Can anyone shed some light on these two issues for me please? :)

Thank you very much for your time.

Chris

[chris...@derbysoft.net]

Hi all.

Still working on this.  No luck yet. :(  Hoping someone might happen by this group soon!

Thanks for the help.

[yzsl...@gmail.com]

I have the same problem with you and have no idea.So,have you solved it? If so,can you tell me how to get it?thank you!

在 2016年5月27日星期五 UTC+8上午12:59:20，chris...@derbysoft.net写道：

[xiezh...@gmail.com]

I also hava the same problem!when u solved ,please tell me !think u!

在 2016年5月27日星期五 UTC+8上午12:59:20，chris...@derbysoft.net写道：

Hi all,

[Juho Autio]

Chris, your error message says:

External library path /opt/azkaban-web-2.5.0/plugins/viewer/hdfs/extlib/* not found.

So the fix is simple: remove the additional asterisk and set your path to /opt/azkaban-web-2.5.0/plugins/viewer/hdfs/extlib/ instead.

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Juho Autio

Senior Analytics Developer

Data Ops, Games

Rovio Entertainment Ltd

Mobile: + 358 (0)45 313 0122
juho....@rovio.com 
www.rovio.com

This message and its attachments may contain confidential information and is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient and / or you have received this message in error, please contact the sender immediately and delete all material you have received in this message. You are hereby notified that any use of the information, which you have received in error in whatsoever form, is strictly prohibited. Thank you for your co-operation.