Multi Tenancy in Alluxio

[mikes...@gmail.com]

Hi,

We have a rather large spark cluster that runs spark apps (not jobs) from different tenants of our corp inside their own docker containers. The spark apps do share many input files which are sitting in multiple different s3 accounts (not just buckets). As the ops team, we have no idea which files of which s3 bucket of which s3 account is going to be accessed by a given spark app until run-time (We get jars only). 

The apps on this large cluster use spark built-in cache and often cache the same thing into their memory which is just waste. Also, on job failure, they need to re-cache their world. This is alluxio sweet spot and we are very interested to add alluxio to our cluster but still not clear on how to manage multi-tenancy with alluxio. In other words, it is HIGHLY critical to isolate each spark app from each other (coming from different departments with strict isolation requirements) such that the apps would NOT be able to access or affect other spark apps data through their shared alluxio cluster. 

Is this possible with alluxio? how can we implement multi-tenancy on top of alluxio among running spark apps? 

Thanks,

Mike

[Calvin Jia]

Hi Mike,

Alluxio is a good fit for the scenario you described.

With regard to multi-tenancy, Alluxio enforces files level permissions, so you could set the permissions to be the same as what you have in S3. Alluxio also supports automatically matching these permissions from the underlying storage (in this case S3), but because S3's permission model is not easily translated, you may not get the exact same semantics.

From a performance / resource isolation point of view, Alluxio supports ensuring certain files have higher priority than others through the pin feature. However, ensuring user A only uses a certain amount of total Alluxio storage and cannot cause user B's cached files from being evicted is not supported.

Hope this helps,

Calvin