[Lustre-discuss] Secondary Groups permissions issue on files / directories

[Mike Hanby]

Howdy,

Our Lustre file system isn't properly supporting secondary groups any longer. This used to work back when we had 1.8.3 installed, when we upgraded to 1.8.5 we also installed a new MDS and backed up / restored our MTD.

All servers and clients are CentOS 5 x86_64 with kernel 2.6.18-194.17.1.el5

Here's what's happening:

1. Create a file as user root and assign group to my users primary group
$ sudo touch /lustre/testfile
$ sudo chgrp myprimary /lustre/testfile
$ sudo chmod 770 /lustre/testfile

2. Writing to the file with 770 and primary group works
$ echo test > /lustre/testfile
$ cat /lustre/testfile
test

3. Change groups to wheel (the first of the secondary groups I am assigned)
$ sudo chgrp wheel /lustre/testfile

4. I can still read and write the file
$ echo test2 > /lustre/testfile
$ cat /lustre/testfile
test2

5. Without changing anything do an ls and then a cat on the file results in permission denied
$ ls -l /lustre/testfile
-rwxrwx--- 1 root wheel 6 Mar 24 15:07 /lustre/testfile

$ cat /lustre/testfile
cat: /lustre/testfile: Permission denied

6. Change the group back to my primary and I can read and write the file many times without issue

7. Change it back to wheel and I can cat the file many times, but once I do an 'ls -l ' on it, I can no longer cat it

8. If I set it to any of my other secondary groups (after changing back to my primary), I can't cat it at all.

This happens on both Infiniband and GigE clients.

The mdt is being mounted with acl support
/dev/sdb /mnt/mdt-mtpt lustre acl 0 0

Any suggestions on where look?
_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Mike Hanby]

I forgot to point out, both the clients and servers are using the Lustre official RPMs for EL5.

Also, on the clients, the "l_getgroups -d <my-UID>" reports the correct GIDs for my user.

[Johann Lombardi]

On Thu, Mar 24, 2011 at 04:22:25PM -0500, Mike Hanby wrote:
> I forgot to point out, both the clients and servers are using the Lustre official RPMs for EL5.
>
> Also, on the clients, the "l_getgroups -d <my-UID>" reports the correct GIDs for my user.

Could you please run lctl get_param mds.*.group_upcall on the MDS node?

Johann

[Mike Hanby]

# lctl get_param mds.*.group_upcall
mds.lustre-MDT0000.group_upcall=NONE

[Johann Lombardi]

On Thu, Mar 24, 2011 at 04:46:00PM -0500, Mike Hanby wrote:
> # lctl get_param mds.*.group_upcall
> mds.lustre-MDT0000.group_upcall=NONE

That's the problem then. You should be using the l_getgroups group upcall.
You can fix this by running the following command on the MGS node:
lctl conf_param $FSNAME-MDT0000.mdt.group_upcall=/usr/sbin/l_getgroups
($FSNAME must be replaced with the name of your filesystem)

Johann

--
Johann Lombardi
Whamcloud, Inc.
www.whamcloud.com

[Mike Hanby]

Thanks Johann, that fixed it. I see where it was set to NONE during the tunefs.lustre when adding a new nid. Now I just have to figure out why.

-----Original Message-----
From: Johann Lombardi [mailto:joh...@whamcloud.com]
Sent: Thursday, March 24, 2011 5:02 PM
To: Mike Hanby
Cc: lustre-...@lists.lustre.org
Subject: Re: [Lustre-discuss] Secondary Groups permissions issue on files / directories