Isilon Find Command to list directories with specific ACLs

[kvkr]

Friends - I need some help on the find command. I have these huge directories (sizing approx 400 TB) with probably a million+ folders . I am trying to use the find command to identify all sub folders under these directories that do not have permissions for the local group "Isilon Users".

Find command doesn't seem to have any option to list something by an ACL other than a owner or group.The only way seem to be to use

find . -type d -print -exec ls -led {} \;  Capture the o/p then write a shell script to identify which directories don't list group "Isilon Users" . 

This method is too long to execute and also resource intensive and no idea how long it is going to take especially walking thru a million directories. Any better thoughts on this...Goal is to find directories that do not have ACL's for "Isilon Users" Group .

I did a man find and I see an option -acl for extended ACL's but not much information on how to use this. 

Isilon OneFS v7.0.1.6

# isi quota list

Type      AppliesTo Path                              Snap Hard Soft     Adv      Used

-------------------------------------------------------------------------------------------

directory DEFAULT   /ifs/iws/edsu/prod/vmax13          No   -    -        -        471.7250T

directory DEFAULT   /ifs/iws/edsu/prod/vmax14          No   -    -        -        191.1307T

 

# cd /ifs/iws/edsu/prod/vmax13/pr/cases/CS0003/db/FCC_Broadband

Example 

# find . -type d -print -exec ls -led {} \;

.

drwxrwxr-x 5 1001  sshd  71 Apr 17  2011 .

 OWNER: user:1001

 GROUP: group:sshd

 CONTROL:dacl_auto_inherited,sacl_auto_inherited

 0: group:AMER\et-dlg-nj-eds-prod-vmax13 allow dir_gen_read,dir_gen_write,dir_gen_execute,std_delete,object_inherit,container_inherit

 3: group:Isilon Users allow dir_gen_all,object_inherit,container_inherit

 ./xml

drwxrwxr-x 2 1001  sshd  1079 Dec 14  2011 ./xml

 OWNER: user:1001

 GROUP: group:sshd

 CONTROL:dacl_auto_inherited,sacl_auto_inherited

 0: group:AMER\et-dlg-nj-eds-prod-vmax13 allow dir_gen_read,dir_gen_write,dir_gen_execute,std_delete,object_inherit,container_inherit

 3: group:Isilon Users allow dir_gen_all,object_inherit,container_inherit

./documents

drwxrwxr-x 2 1001  sshd  2294 Dec 14  2011 ./documents

 OWNER: user:1001

 GROUP: group:sshd

 CONTROL:dacl_auto_inherited,sacl_auto_inherited

 2: group:AMER\et-dlg-nj-eds-prod allow dir_gen_read,dir_gen_write,dir_gen_execute,std_delete,object_inherit,container_inherit

 3: group:Isilon Users allow dir_gen_all,object_inherit,container_inherit

 

[David Hasson]

If you're just looking to apply a certain ACE to everyone, you might look into the PermissionRepair job engine job.  Definitely try it on a test area of your filesystem first, but you can do things like apply certain permissions to an entire tree.  It'll be more efficient than find and may get you what you want.

Look into 'isi job start PermissionRepair --help'.

Definitely experiment with this first though!

-David

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Mark May]

On many unix variants the command getfacl has a recursive option,  BSD does not. It would be possible to use find and -exec.. but, just this once,  I think i'd turn to windows powershell...

Check out get-childitem and get-acl functions.  Using it something like "get-childitem \\isilon\share\path | get-acl" will give you a start to what you want... I'm sure you can find better examples on google..

[Alexandra Harker]

I have been fighting this same issue. I wrote a shell script to target a path and apply our admin group to the ACL, if it doesn't exist. But you're right, it is not a fast process, but does not seem resource intensive. If you would like the script, let me know.

[Dan Pritts]

Alexandra Harker wrote:

I have been fighting this same issue. I wrote a shell script to target a path and apply our admin group to the ACL, if it doesn't exist. But you're right, it is not a fast process, but does not seem resource intensive. If you would like the script, let me know.

Please post.

On Tuesday, February 11, 2014 1:11:54 PM UTC-5, kvkr wrote:

Friends - I need some help on the find command. I have these huge directories (sizing approx 400 TB) with probably a million+ folders . I am trying to use the find command to identify all sub folders under these directories that do not have permissions for the local group "Isilon Users".

Find command doesn't seem to have any option to list something by an ACL other than a owner or group.The only way seem to be to use

find . -type d -print -exec ls -led {} \;  Capture the o/p then write a shell script to identify which directories don't list group "Isilon Users" .

This method is too long to execute and also resource intensive and no idea how long it is going to take especially walking thru a million directories. Any better thoughts on this...Goal is to find directories that do not have ACL's for "Isilon Users" Group .

1) parallelize by working on subdirectories.

2) change it to use xargs instead of individually exec'ing ls on each directory.  The fork/exec overhead really adds up on big find jobs.  You may have to experiment to see

find /ifs/foo -type d -print0 | xargs -0 ls -led 

the 0's tell find and xargs to use nul as the delimiter between fields - that way names with spaces don't mess things up.

Another approach I've considered is a perl hybrid of File::Find and queries to the OneFS "RAN" web api.  If I held open the connection to the RAN it would be faster than the required forks and execs that you'll have even with xargs.  

I haven't messed with the API enough to know how fast it is or isn't compared to the CLI commands - Anyone from Isilon care to comment?

I'm sure you could construct up a filesystem walker that just used the RAN but File::Find is there already. 

I did a man find and I see an option -acl for extended ACL's but not much information on how to use this.

I am pretty sure that the -acl option to find is a standard freebsd thing that works with BSD UFS acl's.  I messed with it a bit but was unable to make it do anything with onefs acls. 

danno

--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan

[Rob Peglar]

Yes, someone from Isilon would care to comment :-)  But it seems nearly every time someone from Isilon does comment in this group, we hear from others active and lurking this group that such replies are 'inappropriate for a user group, you Isilon people should get lost'.  

I personally believe if that attitude persists, those of us from Isilon who do contribute to the community will quickly exit.  

OK.  Onward.

The PAPI (Platform API) is a highly useful thing.  It is quite efficient.  If you are interested in the 'speed' difference between it and the CLI, you must first understand that the CLI invokes the PAPI to perform its commands as of 7.0.  So yes, by definition the PAPI is 'faster' than the CLI.

As for this particular directory enumeration problem I would say, take a step back.  What is the actual purpose behind trying to find all directories that do not incorporate a certain ACL?  

If it is to identify them such that the certain ACL can be added, then PermissionRepair is a useful tool.  

Lastly, I would say that directory enumeration is extremely quick if you take advantage of the capability of holding metadata in SSD persistent store, especially with the RW policy.  In that way, all metadata are held in SSD rather than just a single mirror.  Otherwise you may end up enumerating files using HDD metadata.  I personally find many Isilon clusters in the field with their SSD relatively underutilized (say, 20% or less) whereby there is zero cost to enable the RW policy and take advantage. Just enable it and let SmartPools do its thing.

Cheers

Rob

[Dan Pritts]

Rob Peglar wrote:
> Yes, someone from Isilon would care to comment :-) But it seems
> nearly every time someone from Isilon does comment in this group, we
> hear from others active and lurking this group that such replies are
> 'inappropriate for a user group, you Isilon people should get lost'.
>
> I personally believe if that attitude persists, those of us from
> Isilon who do contribute to the community will quickly exit.

I don't know who told you that but I personally appreciate your (those
of you from isilon) contributions.

I certainly wouldn't want to hear from a vendor who is badmouthing the
competition, or arguing with people about the right or wrong way to do
things. But I have not seen that here.

> The PAPI (Platform API) is a highly useful thing. It is quite
> efficient. If you are interested in the 'speed' difference between it
> and the CLI, you must first understand that the CLI invokes the PAPI
> to perform its commands as of 7.0. So yes, by definition the PAPI is
> 'faster' than the CLI

Really, including the ls and chmod commands? (in particular, your
mention of "PAPI" makes me wonder, I thought that file operations were
done through "RAN" which is somehow separate...)

thanks

[Rob Peglar]

Thanks for your reply, Dan.  I appreciate your contributions to the Isilon community at large.

There are commands in the OneFS CLI which do not invoke the PAPI, no.  Those are the commands typically found in /bin, the 'traditional' FBSD command set.  What I was referring to is the 'isi' command set, of which there are many (as you no doubt know).  

Your reference to 'the web API' confused me, since we at Isilon do not refer to RAN as an API, although strictly speaking, it is of course.  (it is in /namespace instead of /platform)

-

When you say API, we think PAPI.  RAN - the RESTful Access to Namespace - is just called RAN, not RAN API or R-API or anything like that.  It's just plain old RAN :-)

So now that I understand, you can certainly use the RAN to do things like enumerate files and directories.  Compared to native FBSD commands, it is not as 'fast', since the API calls are caught by daemon processes over a network, and doing an ls or chmod is 'native', you are invoking routines written in C.  The advantage of RAN is the flexibility and the ability to manipulate the namespace without having to establish a protocol session or login to the console.

Once again, if you want super-fast directory and file enumeration, you want metadata in SSD with RW policy.  Full stop.

Cheers

Rob

[Saker Klippsten]

Rob sure you are not mistaking this group for the Studio Sysadmin group?I know that is the sentiment on there sometimes pertaining to re-sellers mostly trying to sell product.  ( I completely disagree though  ) . This is a specific Isilon-Group as the name implies.  I would hope that Isilon listens and replies on this group. 

my 2cents.

-Saker

[Peter Serocka]

Rob and Chris, and Erik, and...

Just Thought I should mention that I do subscribe to 

what Saker and Dan already have expressed so well!

-- Peter

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Peter Serocka

CAS-MPG Partner Institute for Computational Biology (PICB)

Shanghai Institutes for Biological Sciences (SIBS)

Chinese Academy of Sciences (CAS)

320 Yue Yang Rd, Shanghai 200031, China

pser...@picb.ac.cn

[Rob Peglar]

Hi Saker,

I am not mistaking it - but no worries.  I consider it a privilege to listen and reply inside this community.

Cheers

Rob

[Rob Peglar]

Thanks Peter - the good news is most of this community subscribes similarly.  All of your contributions are more than welcome, I for one enjoy reading them (and occasionally replying) :-)

cheers

Rob

[Alexandra Harker]

So, I'm sure there are some redundant/nested variables that could be cleaned up. And I'm also sure that it doesn't need to output the recursive path listing first, but maybe could read that into buffer space. That said, this is my first shell script. 

This outputs the full paths of each item (file and directory) in a path specified. It then reads that file line by line and runs ls -led and greps checking for "storage admins" which is my team's AD group. If nothing is returned, it runs the chmod command to add the group and outputs the path to a file (to keep track).

If you have any questions or any improvements, let me know.

#!/usr/bin/bash

# Output all paths to file.

input=$1

find "$1" -print | grep -v .DS_Store |grep -v vdiredirect | grep -v $RECYCLE.BIN | grep -v grep > /ifs/data/filelist.txt

output=/ifs/data/filelist.txt

# Read file.

filename="$output"

while read -r line

do

    name=$line

ls -led "$name" | grep "storage admins" &> /dev/null

if [ $? == 0 ]; then

echo "Storage Admins group has permission to $line." 

else

echo "Storage Admins group DOES NOT have permission to $line." >> /ifs/data/testfile.txt

chmod +a group "caresource\\storage admins" allow generic_all,object_inherit,container_inherit "$line"

fi

done < "$filename"