Storage management and delegation to the end users

[Jean-Baptiste Denis]

Hello,

this won't be a post specific to Isilon/EMC, but rather some general thoughts
about storage management and delegation. I hope it's OK for everybody. If you
think of better place for this kind discussion, I'm all ears !

I won't talk about interactions with the outside world (federation of
identities, sharing among sites...) although this is an absolute critical
question.

My technical and human context have nothing exotic :

- about 2500 daily users
- mixed access to the same data between cifs and nfs
- multiple research teams managing multiple projects
- 1 PB isilon cluster
- A compute cluster accessing data through NFS
- A vast majority of program of the compute cluster need POSIX access to the data

Each team has its own share with an associated quota. When multiple teams need
to work together on some projects, we (the sysadmins) create a specific
share/quota for those projects. To manage access/permissions, we also create
a specific POSIX group for each project. The project is accessible through CIFS
and NFS for the compute cluster.

From my point of view, the difficulties begin here. How do you track storage
allocation for each team or/and for multiple team projects. This sounds quite
basic, but I'm not sure the spreadsheet is a scale-out solution :D

From the user/team/manager point of view, the needs are quite simple : "just
give me some space and let me choose the people I want to give access to."

I think that those tasks (in my use case) could be both delegated. A team
manager could create a new project and choose to allocate some space to it (ie
: share + quota creation) from his own storage pool (initially allocated by the
sysadmin).

The permissions side of the problem could be solved using extended ACLs, but
I don't know any common GUI/CLI interfaces between the CIFS and NFS world. And
the world of extended ACLs in a mixed (CIFS/NFS) environnement looks like (to
me) a bunch of fragile compromises than can be broken at any time (even with the
kind of control Isilon provides on this matter). In my case, that the reason why
we're only using POSIX permissions. I'm also very interested in real use case of
ACLs usage in a mixed environnement. (I'm aware of the Isilon whitepaper about
permissions management)

In the POSIX permissions world, it's just a matter of creating/modifying posix
groups. A task that could be delegated to the manager of a project through
a CLI/GUI interface (not trivial to design, but not very difficult either). Of
course, you don't have the expressiveness of extended ACLs, but I'm already
using this way of accessing data.

In the case of shared projects, you can use the same mechanisms. A shared
project also need a quota and access permissions. The final quota can be build
from contributions of different team (from their own storage allocation) and the
associated POSIX group could be managed by each manager.

Maybe the problem is more political than technical. But I'm convinced that
political stiffness could be mitigated if you give users adequate tools allowing
themselves to be more autonomous, even in a constrained environnement. Also,
even with just a bunch of political storage rules, you still have to apply them
in a way that can scale.

Thinking on the problem by focusing on delegation (nothing fancy, just quota and
group membership management) seems to me like the obvious solution. The fact
that I didn't find any software solution with this spirit makes me think that
I'm missing something =)

I'm really very curious about how you manage those basic tasks on a day to day
basis and your thought about the general picture !

Jean-Baptiste

[Peter Serocka]

On Mon 28 Apr '14 md, at 01:29 st, Jean-Baptiste Denis <jbd...@pasteur.fr> wrote:
> From the user/team/manager point of view, the needs are quite simple : "just
> give me some space and let me choose the people I want to give access to."

Simple - really? If free of charge… guess what happens.
Decide to charge fees… per provisioned or per used capacity? (yeap, politics roll in)
Fees covering actual costs or as “symbolic penalties”? (more politics)
Allow for a competition between paying for central NAS and
purchasing “home-made” storage? (leave politics, enter religion)

Tracking usage by users/groups/projects really is the smaller problem IMHO.

Once the fee policies have been established, it becomes apparent
who should be authorized to request (or self-provision)
storage space: those who spend the money ;-)

Cheers

— Peter

> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Jean-Baptiste Denis]

> Simple - really? If free of charge... guess what happens.
> ...

Did I struck a nerve here ? =)

I agree with all you've said but it doesn't change the fact that when
you've got policies in place, you have to apply them and give feedback to your
users in an efficient way.

I just wanted to point out that once you've determined the amount of space you
can give to a team after bribes and political debate, you could give total
control on this space. And from my point of view, although it won't change a bit
regarding political stuff, it would simplifies a lot of things on the technical
and production side.

Today I'm manually managing shares and quotas creation. While it's not
difficult, it is not the most interesting things to do.

How are you concretely managing those day to day aspects ?

Jean-Baptiste

[Peter Serocka]

> How are you concretely managing those day to day aspects ?

Nothing really special, a few zsh scripts
and built-in quota reporting (daily, to XML).

“daily” stuff is usually only monitoring, while
active “managing” is more on a weekly basis.

Any integrated push-button solution would involve
so many site-specific aspects (creation/updating of
POSIX groups, automount maps, snapshots and external
backups, notifications/reporting,...) that it will end
it a home-made system anyway I think.

— P.

[Daniel Pritts]

On Sunday, April 27, 2014 1:29:40 PM UTC-4, Jean-Baptiste Denis wrote:

The permissions side of the problem could be solved using extended ACLs, but
I don't know any common GUI/CLI interfaces between the CIFS and NFS world. And
the world of extended ACLs in a mixed (CIFS/NFS) environnement looks like (to
me) a bunch of fragile compromises than can be broken at any time (even with the
kind of control Isilon provides on this matter). In my case, that the reason why
we're only using POSIX permissions. I'm also very interested in real use case of
ACLs usage in a mixed environnement. (I'm aware of the Isilon whitepaper about
permissions management)

Sorry for the slow response, for some reason i'm not getting this list as e-mail.  No doubt an error on my part.

Anyway, we have significant cross-platform usage (windows & RHEL).  In practice it has been somewhat difficult to make this work well for our users.

Here are our current site best practices.

 

*) it is critical that you get the mapping of UIDs and GIDs to windows SIDs sorted out at the beginning.    This seems obvious, of course.  We had an odd situation.  Our campus AD has a custom schema (I think) with UID and GID information and other *nix attributes available.   However, it does not run services for unix.  

We didn't really understand that, and we and the EMC support folks saw the relevant attributes installed in AD, and assumed that the services for unix were running.  Turns out that when SFU is not running, the Isilon will not look in AD to see if these attributes are there.   Once we figured this out life was easier, but we ended up with a lot of clean-up since UIDs and GIDs got out of sync.  

As an aside, you can store the unix attributes in AD, and tell the Isilon to access AD as an LDAP server - that works fine.  So we have two "providers" that are both really AD in the back end, but the isilon considers one "AD" and the other "LDAP".  

*) you will find this buried deeply in the cross-platform access white paper, so i can't say it's undocumented.  However, it is such an obvious thing I feel like it should be written in orange tape on the outside of the box when you get your isilon nodes.  Assuming you are using a mixed environment like this, with both AD and LDAP, install a user mapping rule that says "*" is equivalent to "ADDOMAINNAMEHERE\*".    Otherwise, the identities that the Isilon gets from AD and LDAP will NOT be considered identical.  

*) for anything that has cross-platform usage, make sure that full windows-style ACLs are set from the beginning.  When we start with mode bits, bad things happen.  I'm not entirely sure why, but it was clear in our environment that this was the case.  

 

*) similarly, on the NFS shares of these directories, have the isilon do permissions checking locally, rather than trusting the group information it gets from the client.  You do this via a setting on the NFS export called "Map lookup UID".  

*) Once files have ACLs set, the mode bits and user/group ownership displayed via ls on the cluster are mostly irrelevant.  Learn to use and love the "-le" flags to ls, which will show you the ACLs.

*) NFSv3 clients see mode bits that are a best approximation of the ACL.  THis is described well in the white paper.  It is suboptimal in practice. 

*) NFSv4 clients can access the full ACLs.  On Linux, use nfs4_getfacl and nfs4_setfacl and nfs4_editfacl.  This is from the nfs4-acl-tools RPM on RHEL.    Remember that NFSv4 access is stateful, so you lose automatic failover by using NFSv4.  We are experimenting with mounting both ways - normal access via NFSv3 but a parallel set of NFSv4 mounts.

*) if you are using "access zones", be aware that they are irrelevant for NFS.  Whatever IP you connect to, all NFS access takes place in the System access zone.  So...make sure your user mapping is set up in the System access zone.  

*) if you are used to mode bits, you may find that it takes a while to get used to the ins and outs of windows style ACLs.  It took me forever to figure out how to emulate the sticky bit on directories, and I'm not sure I ever got it quite right.  

I hope this helps. Feel free to ask if you have questions.  

[Peter Serocka]

Daniel, fantastic post!

What is used for user authentification on the NFS *clients*?

Also the LDAP right from AD, which runs without SFU? 

Or a plain UNIX/Linux LDAP or NIS, but somehow "synchronized" with AD?

Cheers

--Peter

On 2014 Jun 12. md, at 04:26 st, Daniel Pritts wrote:

.

Here are our current site best practices.

 

*) it is critical that you get the mapping of UIDs and GIDs to windows SIDs sorted out at the beginning.    This seems obvious, of course.  We had an odd situation.  Our campus AD has a custom schema (I think) with UID and GID information and other *nix attributes available.   However, it does not run services for unix.  

We didn't really understand that, and we and the EMC support folks saw the relevant attributes installed in AD, and assumed that the services for unix were running.  Turns out that when SFU is not running, the Isilon will not look in AD to see if these attributes are there.   Once we figured this out life was easier, but we ended up with a lot of clean-up since UIDs and GIDs got out of sync.  

As an aside, you can store the unix attributes in AD, and tell the Isilon to access AD as an LDAP server - that works fine.  So we have two "providers" that are both really AD in the back end, but the isilon considers one "AD" and the other "LDAP".  

*) you will find this buried deeply in the cross-platform access white paper, so i can't say it's undocumented.  However, it is such an obvious thing I feel like it should be written in orange tape on the outside of the box when you get your isilon nodes.  Assuming you are using a mixed environment like this, with both AD and LDAP, install a user mapping rule that says "*" is equivalent to "ADDOMAINNAMEHERE\*".    Otherwise, the identities that the Isilon gets from AD and LDAP will NOT be considered identical.  

*) for anything that has cross-platform usage, make sure that full windows-style ACLs are set from the beginning.  When we start with mode bits, bad things happen.  I'm not entirely sure why, but it was clear in our environment that this was the case.  

 

*) similarly, on the NFS shares of these directories, have the isilon do permissions checking locally, rather than trusting the group information it gets from the client.  You do this via a setting on the NFS export called "Map lookup UID".  

*) Once files have ACLs set, the mode bits and user/group ownership displayed via ls on the cluster are mostly irrelevant.  Learn to use and love the "-le" flags to ls, which will show you the ACLs.

*) NFSv3 clients see mode bits that are a best approximation of the ACL.  THis is described well in the white paper.  It is suboptimal in practice. 

*) NFSv4 clients can access the full ACLs.  On Linux, use nfs4_getfacl and nfs4_setfacl and nfs4_editfacl.  This is from the nfs4-acl-tools RPM on RHEL.    Remember that NFSv4 access is stateful, so you lose automatic failover by using NFSv4.  We are experimenting with mounting both ways - normal access via NFSv3 but a parallel set of NFSv4 mounts.

*) if you are using "access zones", be aware that they are irrelevant for NFS.  Whatever IP you connect to, all NFS access takes place in the System access zone.  So...make sure your user mapping is set up in the System access zone.  

*) if you are used to mode bits, you may find that it takes a while to get used to the ins and outs of windows style ACLs.  It took me forever to figure out how to emulate the sticky bit on directories, and I'm not sure I ever got it quite right.  

I hope this helps. Feel free to ask if you have questions.  

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Peter Serocka

CAS-MPG Partner Institute for Computational Biology (PICB)

Shanghai Institutes for Biological Sciences (SIBS)

Chinese Academy of Sciences (CAS)

320 Yue Yang Rd, Shanghai 200031, China

pser...@picb.ac.cn

[Dan Pritts]

On Wednesday, June 11, 2014, Peter Serocka <pser...@picb.ac.cn> wrote:

Daniel, fantastic post!

What is used for user authentification on the NFS *clients*? 

Also the LDAP right from AD, which runs without SFU? 

Or a plain UNIX/Linux LDAP or NIS, but somehow "synchronized" with AD?

Plain Linux password files and Kerberos auth. Kerberos is from an MIT krb5 realm that is also run by campus. Uids and gids are synchronized "by hand" to the pAsswd and group files. 

In theory we are moving toward getting everything out of AD but it is a low priority.

danno

--
Sent from Phone, apologies for typos and/or brevity

[Jean-Baptiste Denis]

> On Wednesday, June 11, 2014, Peter Serocka <pser...@picb.ac.cn

> <mailto:pser...@picb.ac.cn>> wrote:
>
> Daniel, fantastic post!

I concur. Thank you very much for sharing your experience. So much to digest =)

At the moment, I've got some questions.

If I understand correctly, on a NFSv3 client side (in a cross platform access
scenario with windows ACLs, both cifs and nfs) :

- the output of ls won't mean anything useful. If, as an end user, you want
meaningful information, you have to use a windows client, since the "-e" option
is only available on the isilon cluster.

- you also must combine it with the local mapping on the Isilon side.

- also, to prevent a user messing up with ACLs on the nfsv3 client side, you
have to configure the isilon cluster to ignore chmod/chown operation.

Do you confirm or did I miss something here ?

Thank you again.

Jean-Baptiste

[Peter Serocka]

I see. Having the Isilon looking up the gids is probably

a good safety (not security :) measure in that situation.

Thanks again

-- Peter

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Pritts]

- the output of ls won't mean anything useful. If, as an end user, you want
meaningful information, you have to use a windows client, since the "-e" option
is only available on the isilon cluster.

I wouldn't go so far as to say it's not useful.  But the complexity of windows style
ACLs just cannot be well represented with the standard mode bits.  OneFS does its
best; in our situation that turns out to be not good enough.

- you also must combine it with the local mapping on the Isilon side.

That has worked out best for us.  If your group membership is tightly integrated
I suspect it won't matter much either way. 

- also, to prevent a user messing up with ACLs on the nfsv3 client side, you
have to configure the isilon cluster to ignore chmod/chown operation.

There are various options you can tweak regarding how the cluster treats NFS chmods
on files with windows acls.  Depending on your use case you might be fine with chmod.
See the cross-platform white paper for details.  In our case, by the time someone is
poking around looking at permissions, it's usually because something just isn't translating
well and going to windows or the cluster CLI is best.

regarding chown/chgrp operations -  group and user ownership are mostly irrelevant, so
we don't configure the cluster to ignore them, but we tell users not to depend on chgrp.

I don't envy the OneFS designers trying to deal with this, but I wish that chgrp in particular
were handled more gracefully.  In the common case where you have mode-bit equivalent
permissions in an ACL (e.g., owner full control, group read/maybe execute, others none),
you sure would hope that a simple chgrp on the file would change the group listed in the
ACL.  It doesn't; instead, you get an "operation not permitted" message.  I think this behavior
can be tweaked, but the tweak seemed dangerous.  IIRC it seemed like the required tweak
would allow users to give away files to other users, including possibly root. 

hope this helps.

danno

--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan
+1 (734)615-7362

[Peter Serocka]

On Thu 12 Jun '14 md, at 22:13 st, Dan Pritts <da...@umich.edu> wrote:

>
>> I see. Having the Isilon looking up the gids is probably
>> a good safety (not security :) measure in that situation.
>

> The isilon isn't just looking up GIDs, it is looking up group membership.

Correct, my wording was sloppy.

> So I think it is actually a security measure, albeit a weak one.
>
> Without that setting, the Isilon trusts the group membership and the uid in the NFS request. With the setting, the Isilon trusts only the uid.
>
> I agree though that in practice it's mostly a safety/ease of administration measure.
>
> thanks

[Dan Pritts]

I see. Having the Isilon looking up the gids is probably

a good safety (not security :) measure in that situation.

The isilon isn't just looking up GIDs, it is looking up group membership.  So I think it is actually a security measure, albeit a weak one.

[Jean-Baptiste Denis]

> hope this helps.

Yes. Thank you very much.

Jean-Baptiste