Can't rm some stuff as root on Isilon node -- 'Operation not permitted'

[Jeff Blaine]

I'm stumped. Does anyone have any ideas what is going on here? I un-packaged (unnecessarily) a OneFS installation tarball and am merely trying to clean up the files I extracted.

isibed-1# pwd
/ifs/data
isibed-1# ls -led .
drwxr-xr-x    9 root  wheel  203 Dec 24 15:21 .
 OWNER: user:root
 GROUP: group:wheel
 SYNTHETIC ACL
 0: user:root allow dir_gen_read,dir_gen_write,dir_gen_execute,std_write_dac,delete_child
 1: group:wheel allow dir_gen_read,dir_gen_execute
 2: everyone allow dir_gen_read,dir_gen_execute
isibed-1# ls -led bin
drwxrwxrwx    2 root  wheel  21 Dec 24 15:20 bin
 OWNER: user:root
 GROUP: group:wheel
 SYNTHETIC ACL
 0: user:root allow dir_gen_read,dir_gen_write,dir_gen_execute,std_write_dac,delete_child
 1: group:wheel allow dir_gen_read,dir_gen_write,dir_gen_execute,delete_child
 2: everyone allow dir_gen_read,dir_gen_write,dir_gen_execute,delete_child
isibed-1# ls -led bin/rcp
-rwxrwxrwx    1 root  wheel  44256 Jun 10  2014 bin/rcp
 OWNER: user:root
 GROUP: group:wheel
 SYNTHETIC ACL
 0: user:root allow
file_gen_read,file_gen_write,file_gen_execute,std_write_dac
 1: group:wheel allow file_gen_read,file_gen_write,file_gen_execute
 2: everyone allow file_gen_read,file_gen_write,file_gen_execute
isibed-1# rm bin/rcp
remove bin/rcp? y
rm: bin/rcp: Operation not permitted
isibed-1#

[Jerry Uanino]

Do you have a quota on that dir?

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Blaine]

On Friday, December 26, 2014 9:29:53 AM UTC-5, jerry wrote:

Do you have a quota on that dir?

On /ifs/data/bin ? No. It was created 5 seconds before I tried to remove it. It was a "tar xzf file.tar.gz ... oh wait I didn't need to unpackage that, let me delete it all now..." situation.

This is only 1 example shown of the problem. There are various other exact same errors from other files that were extracted (in error) and now I want to clean up.

'ifs quota list' shows no quota for /ifs/data or anything beneath it.
 

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

[Rob Peglar]

Usually wouldn't step in here...but this scenario is interesting.

Being pedantic, I would ask if you are really logged in as root, or more correctly operating with a root credential.  Assuming you are...I would ask if your cluster is in SmartLock Compliance mode with autocommit enabled?  If so, you're not going to be able to delete files, even as root.  But I'm betting your cluster is not in that mode, so...

The most common scenarios for this is that you are not executing the 'rm' you think you are.  Have you tried /bin/rm, stating the absolute path?

It's really easy to have $PATHs incorrect ( I have done that, and I bet everyone else here as well at least once in the past).  /usr/bin/which is your friend here.

Worse yet, perhaps your tarball has a bin/rm itself, and by unrolling that ball your $PATH is now trying to execute bin/rm instead of the 'real' /bin/rm, which I am guessing is what you intended.

The worst scenario is that your tarball mangled the real /bin, since you are logged in as root...

Good luck

Rob

[Rob Peglar]

Oh, and for the entire group, as a reminder, remember that logged into the console you cannot directly execute files anywhere in /ifs or underneath...ACLs regardless

Just saying

cheers

Rob

On Friday, December 26, 2014 7:42:34 AM UTC-6, Jeff Blaine wrote:

[Jeff Blaine]

Hi Rob,

On Friday, December 26, 2014 12:41:36 PM UTC-5, Rob Peglar wrote:

Usually wouldn't step in here...but this scenario is interesting.

Being pedantic, I would ask if you are really logged in as root, or more correctly operating with a root credential.  Assuming you are...I would ask if your cluster is in SmartLock Compliance mode with autocommit enabled?  If so, you're not going to be able to delete files, even as root.  But I'm betting your cluster is not in that mode, so...

I am and it's not.
 

The most common scenarios for this is that you are not executing the 'rm' you think you are.  Have you tried /bin/rm, stating the absolute path?

It's really easy to have $PATHs incorrect ( I have done that, and I bet everyone else here as well at least once in the past).  /usr/bin/which is your friend here.

Worse yet, perhaps your tarball has a bin/rm itself, and by unrolling that ball your $PATH is now trying to execute bin/rm instead of the 'real' /bin/rm, which I am guessing is what you intended.

The worst scenario is that your tarball mangled the real /bin, since you are logged in as root...

It was the OneFS 6.5.5.29 installation tarball, FWIW.

isibed-1# which rm
rm: aliased to rm -i
isibed-1# unalias rm
isibed-1# which rm
/bin/rm
isibed-1# ls -ld /bin/rm
-r-xr-xr-x    2 root  wheel  14019 Mar 28  2012 /bin/rm
isibed-1# id
uid=0(root) gid=0(wheel) groups=0(wheel),5(operator),20(orphan),70(src),10(staff)
isibed-1#
isibed-1# find /ifs/data -name rm
isibed-1#

 

[Erik Weiman]

I'd try using tree delete job engine operation or try from another node. It's possible that since it was an install tar that maybe it modified that local node...

--

Erik Weiman 

Sent from my iPhone 6+

--

You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

[Cory Snavely]

I saw something similar after making a tar copy of part of /usr/bin, and was also pretty befuddled... when root can't do something, I start thinking rootkits.

Turns out it was the FreeBSD immutability bit. Google that and the chflags binary. Turn off the bit recursively on the affected directories and you should be able to clear everything out. Whew!

[Jeff Blaine]

On Friday, December 26, 2014 9:12:20 PM UTC-5, Cory Snavely wrote:

I saw something similar after making a tar copy of part of /usr/bin, and was also pretty befuddled... when root can't do something, I start thinking rootkits.

Turns out it was the FreeBSD immutability bit. Google that and the chflags binary. Turn off the bit recursively on the affected directories and you should be able to clear everything out. Whew!

WINNER!

# cd /ifs/data
# chflags -R 0 bin lib libexec sbin usr var
# rm -r  bin lib libexec sbin usr var
#

Thanks!

On Dec 26, 2014 2:32 PM, "Erik Weiman" <erik.j...@gmail.com> wrote:

I'd try using tree delete job engine operation or try from another node. It's possible that since it was an install tar that maybe it modified that local node...

This didn't work. The job failed.

[Jerry Uanino]

Then I would mark this as very weird.  It's been a few days, did you find a solution to this?

Curiously, have you tried to write to it? echo blah > bin/rcp ?

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

[Jeff Blaine]

On Monday, December 29, 2014 1:09:18 PM UTC-5, jerry wrote:

Then I would mark this as very weird.  It's been a few days, did you find a solution to this?

Hi Jerry,

Yes, it's been posted. I assume you're subscribed via email -- Check https://groups.google.com/forum/#!topic/isilon-user-group/

Thanks!

[Steven Kreuzer]

Hi Jeff-

On Fri, Dec 26, 2014 at 8:42 AM, Jeff Blaine <cjbl...@gmail.com> wrote:
> I'm stumped. Does anyone have any ideas what is going on here? I un-packaged
> (unnecessarily) a OneFS installation tarball and am merely trying to clean
> up the files I extracted.

I just extracted bin/rcp from a OneFS tarball and it has the system
immutable flag set

If you do a ls -lo on the file you will see 'schg' is set

$ ls -lo rcp
-r-sr-xr-x 1 root wheel schg 51965 Jul 23 23:12 rcp

To remove that flag, use the chflags command

$ chflags noschg rcp
$ ls -lo rcp
-r-sr-xr-x 1 root wheel - 51965 Jul 23 23:12 rcp