Azkaban Hadoop Impersonation

[Brad Ruderman]

Hi All-

I hope all is well. I was wondering about setting up Azkaban so that it wouldn't impersonate. I am running a hadoop cluster with kerberos authentication. I am running the azkaban process under the "userproxy". When I launch I successfully see this user being logged into kerb (within the hadoop security module). However I do have an issue while browsing the HDFS viewer plugin:

viewer.name=HDFS File Browser

viewer.path=hdfs

viewer.order=1

viewer.hidden=false

viewer.external.classpaths=extlib/*

viewer.servlet.class=azkaban.viewer.hdfs.HdfsBrowserServlet

hadoop.home=/etc/hadoop

hadoop.conf.dir=/etc/hadoop/conf

hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_1_0

azkaban.should.proxy=false

hdfs.browser.proxy.user=userproxy

proxy.user=userproxy

proxy.keytab.location=/tmp/userproxy.keytab

allow.group.proxy=false

For some reason the hdfs browser viewer is still trying to proxy as the loggedIn user even though I gave the permissions (azkaban.should.proxy=false). Any insight would be great!

Thanks,

Brad

[Brad Ruderman]

Stack Trace:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: userproxy is not allowed to impersonate bruderman

at org.apache.hadoop.ipc.Client.call(Client.java:1237)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)

at $Proxy13.getFileInfo(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)

at $Proxy13.getFileInfo(Unknown Source)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)

at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)

at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)

at azkaban.viewer.hdfs.HdfsBrowserServlet.handleFSDisplay(HdfsBrowserServlet.java:222)

at azkaban.viewer.hdfs.HdfsBrowserServlet.handleGet(HdfsBrowserServlet.java:146)

at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:102)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)

at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[Chenjie Yu]

There may be a bug in hdfs browser or hadoopsecuritymanager that does proxying when security is enabled.

Just curious though, if your hadoop has security, what is the use case your hdfs browser won't proxy as the user but as the one who started the web server?

On Monday, December 16, 2013 6:13:04 PM UTC-8, Brad Ruderman wrote:

[Brad Ruderman]

We have exploratory querying and production querying. Exploratory is used by analysts and we have a specific server they can log into when they want to do this type of work. This gives them CLI access to the cluster. Within the hive cluster they have access to their own database (read +write) and write to all other databases.

We have production jobs, which write to production databases (for example aggregates). These production jobs will be scheduled within azkaban, therefore should run under a user with elevated permissions. I would rather not give users the ability to write to the production databases so that they stay small and not with a ton of "staging" or "temporary" tables. 

Thanks!

[Brad Ruderman]

I just walked through the code. It appears that this property:

azkaban.should.proxy

Doesn't really have any affect except to hide the ability to switch proxy users. In my case the current user is the logged-in user, however when I access my hadoopsecuritymanger it uses the userproxy thus it tries to proxy the currentuser (bruderman). Estentially my use case is strange because I actually do not want any proxying to be done. 

Will work on a code fix later.

Thanks,

Brad

[Brad Ruderman]

Hi All-

After more consideration, I think this is working as intended. When the cluster security is turned on, the hadoop security class automatically tries to impersonate the logged in user (See line 146 - shouldProxy = True, https://github.com/azkaban/azkaban-plugins/blob/master/plugins/hadoopsecuritymanager/src/azkaban/security/HadoopSecurityManager_H_1_0.java). However you can override this impersonation but specifying a user in the viewer plugin. I have modified the actual viewer plugin to impersonate the proxy user from the config.

Thanks,

Brad