Multi-user support for Spark standalone mode

[Jerry Shao]

Hi all,

Currently we face two problems when submitting Spark apps into Spark cluster with different users:

1. StandaloneExecutorBackend do not distinguish the user who submits app and will run as cluster user, this will cause hdfs file permission issue, also hdfs user isolation mechanism may not work on Spark.
2. For security hdfs, Spark can only use cluster credentials to get hdfs authentication, not the user's credentials who submit apps.

Our current method is to pass the user to StandaloneExecutorBackend, StandaloneExecutorBackend start itself using Hadoop UserGroupInformation's doAs to run  as specific user. For security hdfs, a proxy user will be created to get kerberos authentication. 

I think it's not the elegant way to solve this problem, I post here to see any suggestion about this problem.

BTW, I heard some Yahoo guys are focusing on Spark multi-user and security things, is there anything you can share :-)

Thanks

Jerry

[Mridul Muralidharan]

I will let Tom comment more, but most of the initial work has been
primarily on the yarn mode - not standalone mode.
In yarn mode, you launch the job as the specific user, and if the
hadoop cluster is security aware, passing/validation of delegation
token is done, etc.


Regards,
Mridul

> --
> You received this message because you are subscribed to the Google Groups
> "Spark Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-develope...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Reynold Xin]

+Tom in this he is not subscribed. 

[moji...@gmail.com]

Hi all

   

     I  deploy  a  spark cluster with  standalone mode (no use  Mesos), and  start the cluster by the defualt user --- Spark. Then i run the shark , suppose i am the user -- Shark。Please pay attention to the user i use

     Now i executed some sql sentence in Shark , for example " select * from table" , but it throw the exception：

 

     rg.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.security.AccessControlException: Permission denied: user=shark, access=ALL, inode="/tmp/hive_2013-07-     16_849_1224166558688754033/_task_tmp.-ext-10001" park:upergroup  rwxr-xr-x

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:205)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkSubAccess(FSPermissionChecker.java:174)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:144)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4547)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:2679)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:2643)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:2626)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:612)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:406)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44094)

    

    and then i found the directory in HDFS: (the current hive.exec.scratchdir  is set to /tmp )

   drwxrwxrwx  shark  supergroup   2013-07-16 15:45 /tmp/hive_2013-07- 

   16_849_1224166558688754033

   obviously，the owner of the directory is Shark (the user i start shark), but this directory contain:

    drwxr-xr-x   - spark supergroup          0 2013-07-16 15:54 /tmp/hive_2013-07- 

    16_849_1224166558688754033/-ext-10001

    drwxr-xr-x   - spark supergroup          0 2013-07-16 15:54 /tmp/hive_2013-07- 

   16_849_1224166558688754033/_task_tmp.-ext-10001

     the owner is Spark( the user i start Spark） !!!!

     In my option, when i execute "select xxxx"  it would generate some temp file , such as above /tmp/hive-xxxxxx. And the owner of this directory is shark, that is reasonable，because i start shark with user--shark,  but  why the owner of the subdirectory of this directory ( e.g. _task_tmp.-ext-10001) is spark ( maybe i start the spark with user --Spark).

      Now the problem comes, looking at the exception, it said that  Shark must has the ALL permission  of directory  _task_tmp.-ext-10001，but now the access control of _task_tmp.-ext-10001 is (Spark:upergroup  rwxr-xr-x)

       My question is: i have to start shark and spark using the same user? But if so , it is so

Strange. If several users want to use shark, how could i do?

[Jerry Shao]

Hi,

This is the problem what I mentioned previously. Spark StandaloneExecutorBackend do not use the user (shark) who submits app to communicate with hdfs, instead it uses the user (spark) who start Spark cluster, so it will cause file permission exception. Subdirectory and file is created in the task run on ExecutorBackend with user (spark) which cannot be deleted by user (shark). 

Currently the only way is to run all the stuff in the same user. we are focusing on this issue and will solve it ASAP.

Thanks

Jerry

在 2013年7月25日星期四UTC+8下午5时41分11秒，moji...@gmail.com写道：

[Thomas graves]

Yeah currently I have been working on the yarn mode so its passing the delegation token from the master to workers and running things as the appropriate user.   But it just runs one job and then exits also right now. Once we have that working we do want to start looking at having the yarn mode be longer lived similar to standalone and allow for multi-tenancy.  But unfortunately I haven't looked at it yet.

I'm also working on security for securing the internal SPARK connections, the first cut is a simple shared secret exchange to authenticate.

For the secure hdfs you need to either ship the UGI (with credentials) around or a key tab file.  Something that lets the individual user authenticate.

I don't think you need to run the actual process as the user if its just the hdfs security that is the issue, but for a true multi tenant secure environment you would want to. It looks like the ApplicationDescription is already pulling the user from the environment  via the user.name property. Are you using using that user for the doAs?

I'm not sure I completely understand how you are using the proxy user? Are you having the spark user be the proxy user that then access hdfs on behalf of the user?

Tom

--

[Jerry Shao]

Hi Tom, 

Thanks for your replying, actually I use the user in the ApplicationDsecription for doAs like the below code. if security is enabled I will create the proxy user to pass the Kerberos authentication using spark user, and actually communicate with hdfs using app user; if security is not enabled, I simply create the remote user for doAs

val ugi = if (isSecurityOn) {

      UserGroupInformation.createProxyUser(user, UserGroupInformation.getLoginUser())

    } else {

      UserGroupInformation.createRemoteUser(user)

    }

    ugi.doAs(new PrivilegedExceptionAction[Unit] {

      def run: Unit = {

        func(args)

      }

    })

This can be worked with or without security enabled, one problem when security is enabled is that each task in Spark should get Kerberos authentication to access hdfs, it will delay the execution time, also KDC will be the SPOF. I think the good solution is that client driver get hdfs delegation token and broadcast to all tasks, task use this delegation token to access the hdfs, do you think it's reasonable and applicable ?

BTW, do you have any plan to support multi-user in Spark standalone mode? If you have a whole plan then I can currently use this method until you submit a patch, otherwise I will focus on it.

BR

Jerry

在 2013年7月26日星期五UTC+8上午1时22分00秒，Thomas graves写道：

[Thomas graves]

Hey Jerry,

Ok, the doAs logic makes sense now.

Yes I agree, having the client authenticate and then pass it around is a good idea - this is what I meant by ship the UGI with the credentials.  This is how Hadoop YARN/MapReduce does it too.  

There are a few other issues that might need to be addressed with this though too. Hadoop tokens only live for a certain amount of time - there is actually 2 timeouts - a soft one (renewal interval) and then a hard one (max lifetime). The soft one requires you to renew it within that interval if you are going to keep using it (default 24 hours), but it can only be renewed up to the hard limit.  So for instance the default hard limit is 1 week so after that you have to get a new token.  Those are configurable so as long as you can set them pretty high it might not be an issue or if your jobs don't run for that long its not an issue either.   I haven't tried out the spark streaming stuff but it might be more of an issue there.  It is definitely an issue on YARN though as there are tokens for talking to the ResourceManager to request more resources too.  So long lived spark processes would somehow need to renew the tokens and then possibly after a week get a new one.  Its actually an issue for other long lived processes using Hadoop and there are a few people thinking about that issue.  

I don't have plans for supporting it in standalone mode, but we essentially want to go that way on YARN to allow multi-tenancy on a single Spark instance running on top of YARN. The issues there are the same so I think you solve it for one it works on the other. But that timeline for me is out quite a ways so if you want something soon you might want to work on it.

Tom

[Jerry Shao]

Hi Tom, 

Thanks for your replying.

I agree with you that renew a token for long-run apps in Spark is necessary. Hadoop has a mechanism to nominate a token renew service to renew a soft expired token, and it assumes that jobs in Hadoop cannot run longer than 7 days. I'm not sure Hadoop exposes some APIs to use to update token, otherwise a token renew service should be created in Spark.

Furthermore, after a token is hard expired, a new token should be created, distributed to Spark cluster, a better way to handle all of this is need.

Thanks

Jerry 

在 2013年7月26日星期五UTC+8下午9时50分04秒，Thomas graves写道：